Advanced Plugin Testing¶

Why Write Tests?¶

Automated testing is a hallmark of modern software development. Adding a test suite to your plugin makes it easier to update and redesign your code, update its dependencies, etc. without having to manually check that every feature still works.

Plugin Tests¶

First, we assume that you have already gone through the Plugin Development Tutorial and created the plugin boilerplate with supybot-plugin-create.

Plugin Test Case Classes¶

Limnoria comes with two plugin test case classes, supybot.test.PluginTestCase and supybot.test.ChannelPluginTestCase. Use the latter when the plugin’s commands need to be run in a channel, and the former otherwise.

Both of these classes inherit from Python’s built-in unittest.TestCase; as such, most of the documentation there applies to plugin tests too.

Note that instead of python -m unittest, Limnoria plugin tests are run using the limnoria-test command: e.g. limnoria-test /path/to/your/Plugin

Plugin Test Case Example¶

A basic plugin test case requires:

The class declaration (subclassing one of the TestCase classes above)
A list of plugins to be loaded for these tests. Usually this is just the name of the plugin you’re testing. Note that the Owner, Misc, and Config plugins are always automatically loaded
Some test methods

class MyPluginTestCase(PluginTestCase):
    # List of plugins to load
    plugins = ('MyPlugin',)

    def testEcho(self):
        # Replace the command and expected response with your own,
        # add other assertions, etc.
        self.assertResponse('echo Hello world', 'Hello world')

    def testSomethingElse(self):
        # Add another test case here

As with Python’s unittest module in general, test methods must begin with test. When adding helper methods in this class, they should start with something else.

When writing your actual test methods, we recommend keeping each one short and targeted towards a specific feature. This will allow you to easily find which checks failed just by looking at a failed test’s name, instead of having to sort through line numbers inside test.py. To keep test functions unambiguous, it is fine to give them longer, more specific names to compensate (e.g. testOnlyRespondToRegisteredUsers instead of testRegistered).

Including Extra Setup¶

Plugin tests may define extra setup commands by overriding setUp() from Python’s unittest module:

    def setUp(self):
        # Important! This sets up the bot's simulated IRC network for testing
        super().setUp()

        # Define the identity of the user who we send messages as
        self.prefix = 'foo!bar@baz'

        # Send a message to the simulated IRC network, in this case to register
        # an account with the bot. self.nick refers to the bot's nick.
        self.feedMsg('register tester moo', to=self.nick, frm=self.prefix)
        m = self.getMsg(' ')  # Get the response for the last command

If your setUp function does work that should be cleaned up after, add a tearDown method as well. As with setUp, you should also call the parent class’ tearDown method after running your own cleanup.

Setting Config Variables for Testing¶

Config variables can be set at the test case level. For example, to disable nested commands for this test, you can add a config dict:

class MyPluginTestCase(PluginTestCase):
    config = {'supybot.commands.nested': False}

    def testThisThing(self):
        # stuff

Temporarily setting a configuration variable¶

To temporarily set a config variable inside a test method, use the conf.supybot.<variable name>.context(<new value>) context manager:

import supybot.conf as conf

class MyPluginTestCase(PluginTestCase):
    def testThisThing(self):
        with conf.supybot.commands.nested.context(False):
            # stuff
        # when leaving the context manager, the config value is reverted to default

Plugin Test Methods¶

In addition to Python’s built-in assertions, here are all the test methods defined in Limnoria. These are instance methods, so they should be accessed as self.assertResponse(...), etc.

assertResponse(query, expectedResponse): Feeds query to the bot as a message and checks to make sure the response is expectedResponse. The test fails if they do not match (note that prefixed nicks in the response do not need to be included in the expectedResponse).
assertError(query): Feeds query to the bot and expects an error in return. Fails if the bot doesn’t return an error.
assertNotError(query): The opposite of assertError. It doesn’t matter what the response to query is, as long as it isn’t an error. If it is not an error, this test passes, otherwise it fails.
assertRegexp(query, regexp, flags=re.I): Feeds query to the bot and expects something matching the regexp (no m// required) in regexp with the supplied flags. Fails if the regexp does not match the bot’s response.

Note

This assertRegexp() function is not the same as assertRegex() from Python’s unittest library. assertRegex() compares a regexp against a bare string, while assertRegexp() compares it to the output of a bot command. (For historical reasons, we have this confusing name.)

assertNotRegexp(query, regexp, flags=re.I): The opposite of assertRegexp. Fails if the bot’s output matches regexp with the supplied flags.
assertHelp(query): Expects query to return the help for that command. Fails if the command help is not triggered.
assertAction(query, expectedResponse=None): Feeds query to the bot and expects an action in response, specifically expectedResponse if it is supplied. Otherwise, the test passes for any action response.
assertActionRegexp(query, regexp, flags=re.I): Basically like assertRegexp but carries the extra requirement that the response must be an action or the test will fail.

Utilities¶

feedMsg(query, to=None, frm=None): Simply feeds query to whoever is specified in to or to the bot itself if no one is specified. Can also optionally specify the hostmask of the sender with the frm keyword. Does not actually perform any assertions.
getMsg(query): Feeds query to the bot and gets the response.

Tests for Helper Code¶

If you want to test plugin helpers individually without running commands from your commands, you can add additional test classes inheriting from supybot.test.SupyTestCase. This is a light wrapper around unittest.TestCase that provides some additional logging.

The MoobotFactoids plugin has an example of this (OptionListTestCase).

The same rules for using setUp and tearDown apply: be sure to call the parent class implementations in your overridden functions.