Using Supybot’s utils module

Supybot provides a wealth of utilities for plugin writers in the supybot.utils module, this tutorial describes these utilities and shows you how to use them.

str.py

The supybot.utils.str module provides a bunch of utility functions for handling string values.

The Format Function

Note

For historical reasons, Limnoria provides a format() function that overrides (and predates) the Python builtin of the same name.

We would like to get rid of this patching, but it is hard to do so without breaking existing plugins.

The Supybot version of format() formats strings using percent substitutions, similar to Python’s % operator but with different format codes. Note you have to call format() explicitly to use it; f-strings and the % operator will not run this code.

• % - literal %

• i - integer

• s - string

• f - float

• r - repr

• b - form of the verb to be (takes an int)

• h - form of the verb to have (takes an int)

• L - commaAndify (takes a list of strings or a tuple of ([strings], and))

• p - pluralize (takes a string)

• q - quoted (takes a string)

• n - n items (takes a 2-tuple of (n, item) or a 3-tuple of (n, between, item))

• S - a human-readable size (takes an int)

• t - time, formatted according to config reply.format.time (takes an int)

• T - time delta, formatted (takes an int)

• u - url, formatted according to config reply.format.url

• v - void, takes one or many arguments, but doesn’t display it (useful for translation)

Here are a few examples to help elaborate on the above descriptions:

>>> format("Error %q has been reported %n.  For more information, see %u.",
           "AttributeError", (5, "time"), "https://limnoria.net")

'Error "AttributeError" has been reported 5 times.  For more information,
 see <https://limnoria.net>.'

>>> i = 4
>>> format("There %b %n at this time.  You are only allowed %n at any given
            time", i, (i, "active", "thread"), (5, "active", "thread"))
'There are 4 active threads at this time.  You are only allowed 5 active
 threads at any given time'

>>> i = 1
>>> format("There %b %n at this time.  You are only allowed %n at any given
            time", i, (i, "active", "thread"), (5, "active", "thread"))
 'There is 1 active thread at this time.  You are only allowed 5 active
  threads at any given time'

>>> ops = ["foo", "bar", "baz"]
>>> format("The following %n %h the %s capability: %L", (len(ops), "user"),
            len(ops), "op", ops)
'The following 3 users have the op capability: foo, bar, and baz'

The Other Functions

These functions are not covered by format().

• ellipsisify(s, n) - Returns a shortened version of a string. Produces up to the first n chars at the nearest word boundary.

  • s: the string to be shortened

  • n: the number of characters to shorten it to

• perlReToPythonRe(s) - Converts a Perl-style regexp (e.g., “/abcd/i” or “m/abcd/i”) to an actual Python regexp (an re object)

  • s: the regexp string

• perlReToReplacer(s) - converts a perl-style replacement regexp (eg, “s/foo/bar/g”) to a Python function that performs such a replacement

  • s: the regexp string

• dqrepr(s) - Returns a repr() of s guaranteed to be in double quotes. (Double Quote Repr)

  • s: the string to be double-quote repr()’ed

• toBool(s) - Determines whether or not a string means True or False and returns the appropriate boolean value. True is any of “true”, “on”, “enable”, “enabled”, or “1”. False is any of “false”, “off”, “disable”, “disabled”, or “0”.

  • s: the string to determine the boolean value for

• rsplit(s, sep=None, maxsplit=-1) - historical alias to str.rsplit()

  • s: the string to be split

  • sep: the separator to split on, defaults to whitespace

  • maxsplit: the maximum number of splits to perform, -1 splits all possible splits.

• normalizeWhitespace(s) - reduces all multi-spaces in a string to a single space

  • s: the string to normalize

• depluralize(s) - the opposite of pluralize

  • s: the string to depluralize

• unCommaThe(s) - Takes a string of the form “foo, the” and turns it into “the foo”

  • s: string, the

• distance(s, t) - computes the levenshtein distance (or “edit distance”) between two strings

  • s: the first string

  • t: the second string

• soundex(s, length=4) - computes the soundex for a given string

  • s: the string to compute the soundex for

  • length: the length of the soundex to generate

• matchCase(s1, s2) - Matches the case of the first string in the second string.

  • s1: the first string

  • s2: the string which will be made to match the case of the first

The Commands Format Already Covers

These commands are also accessible via the format function, but they exist if you want to use them separately as well.

• commaAndify(seq, comma=”,”, And=”and”) - transforms a list of items into a comma separated list with an “and” preceding the last element. For example, [“foo”, “bar”, “baz”] becomes “foo, bar, and baz”. Is smart enough to convert two-element lists to just “item1 and item2” as well.

  • seq: the sequence of items (don’t have to be strings, but need to be ‘str()’-able)

  • comma: the character to use to separate the list

  • And: the word to use before the last element

• pluralize(s) - Returns the plural of a string. Put any exceptions to the general English rules of pluralization in the plurals dictionary in supybot.utils.str.

  • s: the string to pluralize

• nItems(n, item, between=None) - returns a string that describes a given number of an item (with any string between the actual number and the item itself), handles pluralization with the pluralize function above. Note that the arguments here are in a different order since between is optional.

  • n: the number of items

  • item: the type of item

  • between: the optional string that goes between the number and the type of item

• quoted(s) - Returns the string surrounded by double-quotes.

  • s: the string to quote

• be(i) - Returns the proper form of the verb “to be” based on the number provided (be(1) is “is”, be(anything else) is “are”)

  • i: the number of things that “be”

• has(i) - Returns the proper form of the verb “to have” based on the number provided (has(1) is “has”, has(anything else) is “have”)

  • i: the number of things that “has”

structures.py

This module provides some custom data structures used by plugins and the bot itself.

Simple queue classes

Warning

The queue and smallqueue classes are historical and predate many of the standard library queues in Python. We recommend using the collections.deque class or queue module instead.

The structures module provides two general-purpose queue classes. The “queue” class is a full-featured queue that scales up to larger sizes. The “smallqueue” class is for queues that will contain fewer (less than 1000 or so) items. These offer a common interface:

• a constructor which will optionally accept a sequence to start the queue off with

• enqueue(item) - adds an item to the back of the queue

• dequeue() - removes (and returns) the item from the front of the queue

• peek() - returns the item from the front of the queue without removing it

• reset() - empties the queue entirely

Special queues

TimeoutQueue holds a queue of items until they reach a certain age and then they are removed from the queue:

• TimeoutQueue(timeout, queue=None) - you must specify the timeout (in seconds) in the constructor. Note that you can also optionally pass it a queue which uses any implementation you wish to use whether it be one of the above (queue or smallqueue) or if it’s some custom queue you create that implements the same interface. If you don’t pass it a queue instance to use, it will build its own using smallqueue.

  • reset(), enqueue(item), dequeue() - all same as above queue classes

  • setTimeout(secs) - allows you to change the timeout value

MaxLengthQueue is a queue that is capped at a certain specified length:

• MaxLengthQueue(length, seq=()) - the constructor naturally requires that you set the max length and it allows you to optionally pass in a sequence to be used as the starting queue. The underlying implementation is actually the queue from before.

  • enqueue(item) - adds an item onto the back of the queue and if it would push it over the max length, it dequeues the item on the front (it does not return this item to you)

  • all the standard methods from the queue class are inherited for this class

RingBuffer

RingBuffer is essentially a MaxLengthQueue which fills up to its maximum size and then circularly replaces the old contents as new entries are added instead of dequeuing.

• RingBuffer(size, seq=()) - as with the MaxLengthQueue you specify the size of the RingBuffer and optionally give it a sequence.

  • append(item) - adds item to the end of the buffer, pushing out an item from the front if necessary

  • reset() - empties out the buffer entirely

  • resize(i) - shrinks/expands the RingBuffer to the size provided

  • extend(seq) - append the items from the provided sequence onto the end of the RingBuffer

Dictionary classes

TwoWayDictionary is a dictionary in which key-value pairs have mappings going both directions.

• TwoWayDictionary(seq=(), **kwargs) - Takes an optional sequence of (key, value) pairs as well as any key=value pairs specified in the constructor as initial values for the two-way dict.

  • other than that, no extra features that a normal Python dict doesn’t already offer with the exception that any (key, val) pair added to the dict is also added as (val, key) as well, so the mapping goes both ways. Elements are still accessed the same way you always do with Python ‘dict’s.

• ExpiringDict: TODO

• TimeoutDict: TODO

Set classes

• MultiSet: TODO

• TruncatableSet: TODO

web.py

The web portion of Supybot’s utils module is mainly used for retrieving data from websites but it also has some utility functions pertaining to HTML and email text as well.

• getUrl(url, size=None, headers=None) - gets the data at the URL provided and returns it as one large string

  • url: the location of the data to be retrieved or a urllib2.Request object to be used in the retrieval

  • size: the maximum number of bytes to retrieve, defaults to None, meaning that it is to try to retrieve all data

  • headers: a dictionary mapping header types to header data

• getUrlFd(url, headers=None) - returns a file-like object for a url

  • url: the location of the data to be retrieved or a urllib2.Request object to be used in the retrieval

  • headers: a dictionary mapping header types to header data

• htmlToText(s, tagReplace=” “) - strips out all tags in a string of HTML, replacing them with the specified character

  • s: the HTML text to strip the tags out of

  • tagReplace: the string to replace tags with

• strError(e) - pretty-printer for web exceptions, returns a descriptive string given a web-related exception

  • e: the exception to pretty-print

• mungeEmail(s) - a naive e-mail obfuscation function, replaces “@” with “AT” and “.” with “DOT”

  • s: the e-mail address to obfuscate

• getDomain(url) - returns the domain of a URL - url: the URL in question

The Best of the Rest

supybot.utils.file - file utilities

• touch(filename) - updates the access time of a file by opening it for writing and immediately closing it

• mktemp(suffix=””) - creates a decent random string, suitable for a temporary filename with the given suffix, if provided

• the AtomicFile class - used for files that need to be atomically written, i.e., if there’s a failure the original file remains unmodified. For more info consult file.py in src/utils

supybot.utils.gen - general utilities

• timeElapsed(elapsed, [lots of optional args]) - given the number of seconds elapsed, returns a string with the English description of the amount of time passed, consult gen.py in src/utils for the exact argument list and documentation if you feel you could use this function.

• exnToString(e) - improved exception-to-string function. Provides nicer output than a simple str(e).

• InsensitivePreservingDict class - a dict class that is case-insensitive when accessing keys

supybot.utils.iter - iterable utilities

• len(iterable) - returns the length of a given iterable

• groupby(key, iterable) - equivalent to the itertools.groupby function available as of Python 2.4. Provided for backwards compatibility.

• any(p, iterable) - Returns true if any element in the iterable satisfies the predicate p

• all(p, iterable) - Returns true if all elements in the iterable satisfy the predicate p

• choice(iterable) - Returns a random element from the iterable

supybot.dynamicScope / dynamic - accessing variables in the stack

Warning

Using dynamicScope without an import is deprecated as of 2025-05.

Please import it explicitly with from supybot.dynamicScope import dynamic.

This feature is not in supybot.utils but still deserves to be documented as a utility.

Althrough you should avoid using this feature as long as you can, it is sometimes necessary to access variables the Supybot API does not provide you.

For instance, the Aka plugin provides per-channel aliases by overriding getCommandMethod. However, the channel where the command is called is not passed to this functions, so when writing Aka I could either add this parameter (and thus break all plugins all plugins already overriding this method) or use this hack. I choosed this hack.

How does it work? This is quite simple: dynamic.channel is a shortcut for supybot.dynamicScope.DynamicScope.__getattr__('channel'), which browse the call stack backwards, looking for a variable named channel, and then returns is as far as it finds it (and returns None if there is no such variable).

Note that you don’t have to import dynamicScope, the dynamic object is automatically set as a global variable when Supybot starts.