Basics¶

Factoids start out pretty basic, as a way of the bot remembering things, such as:

<Alice> ibid: Ibid is an awesome bot framework
<ibid> Alice: One learns a new thing every day

Now I can ask him about that:

<Alice> ibid: what is ibid?
<ibid> Alice: Ibid is an awesome bot framework

Or even:

<Alice> ibid: Ibid
<ibid> Alice: Ibid is an awesome bot framework

They can also store multiple values. On retrieval, a value is picked at random:

<Alice> ibid: Ibid is also your father
<ibid> Alice: I'll remember that
<Alice> ibid: Ibid
<ibid> Alice: ibid is your father
<Alice> ibid: Ibid
<ibid> Alice: ibid is an awesome bot framework

You can request a specific one:

<Alice> ibid: Ibid /frame/
<ibid> Alice: ibid is an awesome bot framework
<Alice> ibid: literal ibid
<ibid> Alice: 1: is an awesome bot framework, 2: is your father
<Alice> ibid: Ibid #2
<ibid> Alice: ibid is your father

Fancy factoids¶

Factoids don’t just have to be like a dictionary, they can have some basic behaviour attached. This example also shows how you can trivially replace factoids (assuming you have permission to do so):

<Alice> ibid: hi is <reply>Hi there
<ibid> Alice: One learns a new thing every day
<Alice> ibid: hi
<ibid> Hi there
<Alice> ibid: no, hi is <action>waves
<ibid> Alice: Got it
<Alice> ibid: hi
 * ibid waves
<Alice> ibid: no, hi is <reply>Hello $who
<ibid> Alice: If you say so
<Alice> ibid: hi
<ibid> Hello Alice

The $who substitutes the speaker’s name. Other substitutions possible are:

• $channel, $source
• $year, $month, $mon, $month1, $month2, $day, $day2, $hour, $minute, $second, $date, $time, $weekday, $unixtime

$month, $mon, $month1 and $month2 substitute, respectively, the month name, abbreviated month name, month number and zero-padded month number; $day2 substitutes the zero-padded day number.

And factoids can have multiple names. Our greeting only responds to “hi”, it should respond to more greetings:

<Alice> ibid: hello is the same as hi
<ibid> Alice: Yessir
<Alice> ibid: good day is the same as hi
<ibid> Alice: Alrighty
<Alice> ibid: good day
<ibid> Hello Alice

Finally, factoids can take basic arguments:

<Alice> ibid: slap $arg is <action>slaps $1
<ibid> Alice: Got it
<Alice> ibid: slap Alice
 * ibid slaps Alice
<Alice> ibid: run from $arg to $arg is <action>starts running at $1 and stops at $2
<ibid> Alice: One learns a new thing every day
<Alice> ibid: run from here to there
 * ibid starts running at here and stops at there

Searching¶

Factoids aren’t very useful if we can’t remember what they are called:

<Alice> ibid: search slap
<ibid> Alice: slap $arg [1]
<Alice> ibid: search father
<ibid> Alice: Ibid [2]

The first example tells us about a factoid mentioning slap. The second shows what happens when a multi-valued factoid is found. The result we are after may be one of two values, so we can’t simply say “ibid: ibid”, we should rather use say:

<Alice> ibid: ibid /father/
<ibid> Alice: Ibid is your father

You can specify which part of the factoid you are searching in:

<Alice> ibid: search for values containing awesome
<ibid> Alice: Ibid [2]
<Alice> ibid: search for names containing awesome
<ibid> Alice: I couldn't find anything with that name

Modification¶

Factoids aren’t set in stone. While they can be replaced with “no, name is newvalue”, sometimes you just want to make a small change:

<Alice> ibid: ibid
<ibid> Alice: Ibid is an awesome bot framework
<Alice> ibid: ibid += , written in python
<ibid> Alice: Pattern matches multiple factoids, please be more specific
<Alice> ibid: ibid /awesome/ += , written in python
<ibid> Alice: Done
<Alice> ibid: ibid /awesome/
<ibid> Alice: Ibid is an awesome bot framework, written in python
<Alice> ibid: ibid /python/ ~= s/awesome //
<ibid> Alice: Yessir
<Alice> ibid: ibid /python/
<ibid> Alice: Ibid is an bot framework, written in python

And finally, removal:

<Alice> ibid: forget ibid
<ibid> Alice: Sure
<Alice> ibid: ibid
<ibid> Alice: What?

Or only one of the values:

<Alice> ibid: literal ibid
<ibid> Alice: 0: is your father, 1: is an awesome bot framework
<Alice> ibid: forget ibid /framework/
<ibid> Alice: Sure
<Alice> ibid: literal ibid
<ibid> Alice: 0: is your father

SYNOPSIS¶

ibid-db command [options...]

DESCRIPTION¶

This utility is for offline management of your Ibid bot’s database. Used for import, export, and upgrades.

The export format is DBMS-agnostic and can be used to migrate between different databases.

COMMANDS¶

OPTIONS¶

FILES¶

ibid.ini

Locates the database to act upon by looking for the [databases].ibid value in the bot configuration file in the current directory.

SEE ALSO¶

ibid(1), ibid.ini(5), ibid-setup(1), http://ibid.omnia.za.net/

SYNOPSIS¶

DESCRIPTION¶

This utility is for adding and removing sets of packaged factoids, known as factpacks, from your Ibid’s factoid database.

The default mode is factpack loading. The factpack-file specified is loaded into the bot’s database. Should the pack contain any facts with the same name as an existing fact in the bot’s database, loading will be aborted, unless the -s option is supplied.

Factpacks can be gzipped if the filename ends with .gz.

When invoked with the -r option, the named factpack (original import filename minus the extension) will be removed from the bot. If any of the facts contained in that pack were modified while loaded in the bot, unloading will be aborted, unless the -f option is supplied.

OPTIONS¶

FACTPACKS¶

Factpacks are JSON-encoded text files containing a list of facts. Each fact is a tuple of two lists: fact-names, fact-values. The same substitutions are available as in normal online Factoids.

[
 [["Hello", "Hi"], ["<reply> Hi There", "<action> waves"]],
 [["Bye"], ["<reply> kbye $who", "<reply> Cheers"]]
]

FILES¶

ibid.ini

Locates the database to act upon by looking for the [databases].ibid value in the bot configuration file in the current directory.

SEE ALSO¶

ibid(1), ibid.ini(5), http://ibid.omnia.za.net/

SYNOPSIS¶

ibid-knab-import knab-sa-url source [config-file]

DESCRIPTION¶

This utility imports users, last seen information, factoids, and URLs from a Knab bot’s database into an Ibid.

For best results, import directly into a brand new, clean Ibid install.

On import, strings are converted to Unicode, guessing UTF-8 and falling back to detection.

OPTIONS¶

knab-sa-url

The SQLAlchemy URI for the Knab’s database. The format is mysql://user:pass@hostname/dbname

source

The name in the Ibid bot for the source that the Knab was previously connected to.

config-file

If supplied, this is configuration file is used for locating the Ibid’s database rather than ibid.ini.

FILES¶

ibid.ini

Locates the database to act upon by looking for the [databases].ibid value in the bot configuration file in the current directory.

SEE ALSO¶

ibid(1), ibid.ini(5), ibid-setup(1), http://ibid.omnia.za.net/, http://knab.omnia.za.net/

SYNOPSIS¶

DESCRIPTION¶

This utility is for graphing memory usage from an Ibid bot configured to log such usage.

Matplotlib is required for graphing.

OPTIONS¶

FILES¶

logfile

A log file generated by loading the memory plugin into Ibid, which will periodically log memory usage. It can be gzip compressed, if the filename ends in .gz.

SEE ALSO¶

ibid(1), ibid-objgraph(1), http://ibid.omnia.za.net/

SYNOPSIS¶

DESCRIPTION¶

This utility is for graphing object-type usage from an Ibid bot configured to log such usage.

Matplotlib is required for graphing.

OPTIONS¶

FILES¶

logfile

A log file generated by loading the memory plugin into Ibid, which will periodically log object usage. It can be gzip compressed, if the filename ends in .gz.

SEE ALSO¶

ibid(1), ibid-memgraph(1), http://ibid.omnia.za.net/

SYNOPSIS¶

DESCRIPTION¶

This utility is for passing events to a running Ibid bot, or executing RPC-exposed functions remotely.

It communicates with the pb source on the Ibid.

message is a text message as could be sent to the bot by an IM source. The message is processed normally by the bot.

feature is the name of the feature to invoke exposed method method on, directly. parameters are passed directly to the method. They can be specified positionally or by key, using the same syntax as Python: key=value. They may be encoded in JSON, if not valid JSON they will be treated as strings.

The output is a JSON-encoded response.

OPTIONS¶

SEE ALSO¶

ibid(1), http://ibid.omnia.za.net/

SYNOPSIS¶

ibid-plugin [options...] [plugin[-]|plugin.Processor[-]...]

DESCRIPTION¶

This utility is for testing Ibid plugins without the full bot environment. This means testing can be performed offline and without loading all the available plugins.

This should be run in a configured Ibid bot directory.

All the listed plugins and Processors will be loaded on start-up. Naming a plugin loads the complete plugin. Suffixing a - to the name, ignores that plugin or Processor instead of loading it.

OPTIONS¶

FILES¶

ibid.ini

Locates the database to act upon by looking for the [databases].ibid value in the bot configuration file in the current directory.

BUGS¶

ibid-plugin doesn’t emulate a complete Ibid environment, and will ignore all of the following:

• Delayed and periodically executed functions.
• Messages to alternate sources.
• Messages directly dispatched, rather than added to responses.
• Permissions. All permissions are granted to the user.

SEE ALSO¶

ibid(1), ibid.ini(5), ibid-setup(1), http://ibid.omnia.za.net/

SYNOPSIS¶

ibid-setup

DESCRIPTION¶

This program sets up everything that a new Ibid bot needs before it can run. It asks a series of questions about the new bot, and writes out a basic configuration file - ibid.ini(5) - to the current directory. It also creates a database for the bot, by default a SQLite database in the current directory.

This should be run in the directory which will become the new Ibid bot’s base.

Should there be an existing ibid.ini in the current directory, it will be used, and the only questions asked will be for adding an administrative user. These can safely be skipped with a ^C.

FILES¶

ibid.ini

The Ibid bot’s configuration file, will be created if it doesn’t exist.

SEE ALSO¶

ibid(1), ibid.ini(5), http://ibid.omnia.za.net/

SYNOPSIS¶

ibid [config-file]

DESCRIPTION¶

This runs an Ibid bot in the foreground.

There should there be an existing ibid.ini (created by ibid-setup(1)) in the current directory or one should be provided on the command line.

Where possible, you should run twistd -n ibid instead of this program, as otherwise some classes of errors go unreported. See BUGS.

BUGS¶

Exceptions in twisted callbacks can go unnoticed in this program. That has no harmful effects, but the developers may miss out on some good bug reports.

FILES¶

ibid.ini

The Ibid bot’s configuration file, will be created if it doesn’t exist.

SEE ALSO¶

ibid.ini(5), ibid-setup(1), twistd(1), http://ibid.omnia.za.net/

NAME¶

DESCRIPTION¶

ibid.ini contains all the configuration for an Ibid bot.

A complete description of the contents of this file is out of the scope of this manpage. For more details see the Ibid HTML documentation: http://ibid.omnia.za.net/docs/

Lines beginning with # are considered to be comments and ignored. To use a # symbol in an option (e.g. an IRC channel name), quote the option with double-quotes, e.g. channels = "#ibid",

This file will be written to by the bot when configuration settings are altered online. It can also be edited manually and a running bot told to reload config. Manual edits and comments will be preserved when the bot modifies its own configuration, provided that they have not been edited since bot start-up or the last config reload.

SECTIONS¶

EXAMPLE¶

botname = joebot
logging = logging.ini

[auth]
    methods = password,
    timeout = 300
    permissions = +factoid, +karma, +sendmemo, +recvmemo, +feeds, +publicresponse

[sources]
    [[telnet]]
    [[timer]]
    [[http]]
        url = http://joebot.example.com
    [[smtp]]
    [[pb]]
    [[atrum]]
        channels = "#ibid",
        nick = $botname
        type = irc
        auth = hostmask, nickserv
        server = irc.atrum.org

[plugins]
    cachedir = /tmp/ibid
    [[core]]
        names = $botname, bot, ant
        ignore = ,

[databases]
    ibid = sqlite:///ibid.db

FILES¶

logging.ini

A standard Python logging.config configuration file describing loggers, handlers, and formatters for log messages. See http://docs.python.org/library/logging.html

SEE ALSO¶

ibid(1), http://ibid.omnia.za.net/

Modules¶

email_utils

Standard Python email.utils.

Functions for parsing and formatting e-Mail headers.

hashlib

Standard Python hashlib.

Cryptographic hash functions.

On Python 2.4 it won’t support the SHA-2 functions: hashlib.sha224(), hashlib.sha384() and hashlib.sha512() – these will all return 'Not Supported'.

json

Standard Python json, using SimpleJSON on older versions.

JSON serialisation and parsing library.

ElementTree

Standard Python xml.etree.cElementTree, using ElementTree on older versions.

Classes¶

class ibid.compat.defaultdict([default_factory[, ...]])¶

Standard Python collections.defaultdict.

Returns a dict-like-object where all unset values contain the value returned by default_factory().

Functions¶

ibid.compat.all(iterable)¶

Standard Python all().

Return True if every item in iterable is True.

ibid.compat.any(iterable)¶

Standard Python any().

Return True if any single item in iterable is True.

ibid.compat.strptime(date_string, format)¶

Standard Python datetime.datetime.strptime().

Return a datetime corrosponding to date_string, according to format.

ibid.compat.factorial(x)¶

Standard Python math.factorial().

Return the factorial of x.

Helper Functions¶

class ibid.config.Option(name, description, default=None)¶

A unicode string configuration item.

Parameters:	name – The configuration key description – A human-readable description for the configuration item. default – The default value, if not specified in the configuration file.

If created as an attribute to a ibid.plugins.Processor or ibid.source.IbidSourceFactory, it will retrieve the value of plugins.plugin.name or sources.source.name.

This is also the base class for other Options.

Example:

class Secret(Processor):
   password = Option('password', 'Secret Password', 's3cr3t')

Assuming that processor is in the secret plugin, the configuration item could be provided as:

[plugins]
   [[secret]]
      password = blue

class ibid.config.BoolOption(name, description, default=None)¶

A boolean configuration item.

Usage is identical to Option.

class ibid.config.IntOption(name, description, default=None)¶

A integer configuration item.

Usage is identical to Option.

class ibid.config.FloatOption(name, description, default=None)¶

A floating-point configuration item.

Usage is identical to Option.

class ibid.config.ListOption(name, description, default=None)¶

A list configuration item. Values will be unicode strings.

Usage is identical to Option.

class ibid.config.DictOption(name, description, default=None)¶

A dictionary configuration item. Keys and values will be unicode strings.

Usage is identical to Option.

Core Functions¶

ibid.config.FileConfig(filename)¶

Parses filename and returns a configuration tree.

Dispatcher¶

ibid.core.process(event, log)¶

This function takes event and passes it to the process() function in each processor, in order of increasing priority. Messages are logged on the logger log.

After each Processor, any unclean SQLAlchemy sessions are committed and exceptions logged.

class ibid.core.Dispatcher¶

The Ibid Event dispatcher.

call_later(delay, callable, oldevent, *args, **kwargs)¶

Run callable after delay seconds, passing it oldevent and *args and *kwargs.

Returns a twisted.internet.base.DelayedCall.

Can be used in plugins instead of blocking in sleep.

Reloader¶

class ibid.core.Reloader¶

The center of Ibid’s bootstrap process, the reloader loads plugins and processors. They can be reloaded at any time.

run()¶

Boostrap Ibid and run the reactor.

reload_dispatcher()¶

Reload the Ibid dispatcher.

load_source(name[, service])¶

Load source of name name, setting the service parent to service.

load_sources([service])¶

Load all enabled sources, setting the service parents to service.

Sources can be disabled by setting the configuration key service.``disabled = True``.

unload_source(name)¶

Unload source of name name.

reload_source(name)¶

Re-load source of name name.

load_processors([load, noload, autoload])¶

Load all enabled processors, according to the rules in load_processor().

load specifies the plugins to force loading, noload plugins to skip loading, and autoload whether to load everything by default. If these parameters are not supplied or are None, they will be looked up as configuration keys in the plugins block.

load_processor(name, [noload, load, load_all=False,

noload_all=False])

Load the plugin of name name. Individual Processors can be disabled by listing them in noload. If they are marked with autoload = False, then they are skipped unless listed in load or load_all is True.

unload_processor(name).

Unload plugin of name name.

reload_databases()¶

Reload the Databases.

reload_auth()¶

Reload the ibid.auth.

reload_config()¶

Notify all processors of a configuration reload, by calling setup().

Databases¶

ibid.core.regexp(pattern, item)¶

Regular Expression function for SQLite.

ibid.core.sqlite_creator(database)¶

Connect to a SQLite database, with regular expression support, thanks to regexp().

class ibid.core.DatabaseManager(check_schema_versions=True)¶

The DatabaseManager is responsible for loading databases (usually only one, 'ibid'), and is a dict of database to sqlalchemy.orm.scoping.ScopedSessions.

load(name)¶

Load the database of name name.

Echoing is configured by debugging.sqlalchemy_echo.

Databases are configured as sanely as possible:

• All databases are brought up in a UTF-8 mode, with UTC timezone.
• MySQL has the default engine set to InnoDB and ANSI mode enabled.

Decorators¶

ibid.plugins.handler()¶

Decorator that makes a method receive all events.

First parameter to the wrapped method will be the event object:

@handler
def handle(self, event):
   event.addresponse(u'Did you see that? I handled an event')

ibid.plugins.match(regex, version='clean', simple=True)¶

Decorator that makes a method receive message events matching regular expression string regex.

The regex will be matched, with re.I, re.UNICODE and re.DOTALL modes. You should anchor both sides of it. If simple is true (default), the regex will be modified to match the whole string (^ and $ are added), a space in the regex will match any sequence of whitespace, and the following shortcuts are available for common regex fragments (which are captured as arguments):

Selector	expands to
{alpha}	[a-zA-Z]+
{any}	.+
{chunk}	\S+
{digits}	\d+
{number}	\d*\.?\d+
{url}	url_regex()
{word}	\w+

Using simple regexes where they are applicable can make them much more readable.

Any match groups or selectors in the regex will be passed as parameters to the decorated method, after the event object:

@match(r'(?:foo|bar) {chunk}')
def foo(self, event, parameter):
   event.addresponse(u'Foo: %s', parameter)

The above match is equivalent to this non-simple version:

@match(r'^(?:foo|bar)\s+(\S+)$', simple=False)

Optionally you can specify keyword arguments, which will get passed on to the function. The syntax is {keyword:selector}, where the keywords will get passed onto the decorated method:

@match(r'I am {nick:chunk} on {network:chunk}')
def register(self, event, network, nick):
   pass

The above match is equivalent to this non-simple version:

@match(r'^I\s+am\s+(?P<nick>\S+)\s+on\s+(?P<network>\S+)$', simple=False)

Note that you cannot mix positional and keyword arguments. All your selectors must either be named, or all be unnamed.

version can be set to one of:

'clean'

The default, and almost always what you want. The bot name and intervening punctuation are removed from the front of the message, if the bot was addressed. Trailing punctuation and surrounding whitespace is stripped.

'raw'

The message as the bot saw it.

'deaddressed'

The bot name and intervening punctuation are removed from the front of the message, if the bot was addressed.

'stripped'

Trailing punctuation and surrounding whitespace is stripped.

	De-address	Don’t de-address
Strip	'clean'	'stripped'
Don’t strip	'deaddressed'	'raw'

ibid.plugins.authorise(fallthrough=True)¶

Decorator that requires Processor.permission for the user that would trigger this method.

fallthrough sets the failure mode. If True, the next Procesor will be called in the hope of finding another one that’ll handle it. If one is never found or fallthrough is False, an error message will be returned by ibid.plugins.core.Complain:

permission = 'awesome'

@authorise()
@match(r'^do\s+awesome\s+things$')
def method(self, event):
   event.addresponse(u'Yes sir, you are awesome!')

ibid.plugins.periodic([interval=0, config_key=None, initial_delay=60])¶

Decorator that runs the method every interval seconds, from timer events. The method won’t be called until initial_delay seconds have passed since startup.

If config_key is set to a string, the IntOption of that name will be used to set interval. This is done in Processor.setup() so if you override that, be sure to call super.

Other Functions¶

ibid.plugins.auth_responses(event, permission)¶

If the event sender has the permission permission, return True.

If not, the event will be marked as having failed authorisation. If no other Processor processes the event, an error message will be returned by ibid.plugins.core.Complain.

This is used internally by @authorise(), but you can call it directly if you need more complex permission handling than @authorise() allows for.

When you use this, you should ensure that permission is listed in Processor.permission or Processor.permissions.

RPC¶

class ibid.plugins.RPC¶

All methods named with the prefix remote_ will be exposed via Ibid’s various RPC mechanisms (including the web interface).

It is common to extend both Processor and RPC in the same class. The handlers can then wrap around the remote_ methods, to provide the same features over IM and RPC.

Note

The RPC code is still experimental and not widely used. Don’t be surprised if it doesn’t work.

End-to-end testing¶

class ibid.test.PluginTestCase¶

A subclass of Twisted Trial’s unittest.TestCase. It sets up an environment much like a running Ibid including a clean SQLite database, and loads the specified plugins,

load¶

List: strings naming plugins to be loaded before running tests.

Default: empty

noload¶

List: strings naming plugins not to be loaded.

Default: empty

load_base¶

Boolean: whether to load a small set of base plugins (currently, just core).

Default: True

load_configured¶

Boolean: load all configured modules (excluding noload).

Default: if load is empty, True; otherwise, False.

username¶

String: the default username/nick in events created by the make_event() method.

Default: u'user'

public¶

Boolean: whether or not the events created by make_event() are public.

Default: False

network¶

Boolean: whether or not the test uses the external network. Used to skip tests in networkless environments (where the environment variable IBID_NETWORKLESS_TEST is defined).

Default: False

setUp()¶

If you override this method, make sure you call PluginTestCase.setUp().

tearDown()¶

If you override this method, make sure you call PluginTestCase.tearDown().

make_event(message=None, type=u'message')¶

Create and return an event on the test source, from the test user, of type type.

responseMatches(event, regex)¶

Process event (either an event or a string to be treated as a message from the test user on the test source), and return a 3-tuple of (result, event, responses); result is a bool indicating whether the response matches regex (either a regex string or a compiled regex).

The other two elements give more information: event is the processed event; responses is a single matching response if result is True, or a list of all responses otherwise.

assertResponseMatches(event, regex)¶

Assert that responseMatches() returns true.

failIfResponseMatches(event, regex)¶

The opposite of assertResponseMatches().

assertSucceeds(event)¶

Process event (either an event or a string to be treated as a message from the test user on the test source), and check that it is processed by some Processor and no complaint is set.

assertFails(event)¶

The opposite of assertSucceeds().

String Functions¶

ibid.utils.ago(delta[, units])¶

Return a string representation of delta, a datetime.timedelta.

If units, an integer, is specified then only that many units of time will be used.

>>> ago(datetime.utcnow() - datetime(1970, 1, 1))
'39 years, 6 months, 12 days, 9 hours, 59 minutes and 14 seconds'
>>> ago(datetime.utcnow() - datetime(1970, 1, 1), 2)
'39 years and 6 months'

ibid.utils.format_date(timestamp[, length='datetime', tolocaltime=True])¶

Convert datetime.datetime timestamp to the local timezone (timestamp is assumed to be UTC if it has no tzinfo) and return a unicode string representation.

length can be one of 'datetime', 'date' and 'time', specifying what to include in the output.

If tolocaltime is False, the time will left in the original time zone.

The format is specified in the plugins.length configuration subtree, as three values: datetime_format, date_format and time_format.

Example:

>>> format_date(datetime.utcnow())
u'2009-12-14 12:41:55 SAST'

ibid.utils.parse_timestamp(timestamp)¶

Parse string timestamp, convert it to UTC, and strip the timezone.

This function is good at parsing machine timestamps, but can’t handle “human” times very well. (It uses dateutil.parser)

Return a naive datetime.datetime.

ibid.utils.human_join(items[, separator=u', ', conjunction=u'and'])¶

Turn iterable items into a unicode list in the format a, b, c and d.

separator separates all values except the last two, separated by conjunction.

Example:

>>> human_join(['a', 'b', 'c', 'd'])
u'a, b, c and d'

ibid.utils.plural(count, singular, plural)¶

If count is 1, return singular, otherwise plural.

It’s recommended to use complete words for singular and plural rather than suffixes.

ibid.utils.indefinite_article(phrase)¶

Use heuristics to determine whether the pronunciation of phrase starts with a vowel or consonant (assuming it is English) and return ‘an’ or ‘a’ respectively.

ibid.utils.decode_htmlentities(text)¶

Return text with all HTML entities removed, both numeric and string-style.

ibid.utils.file_in_path(program)¶

Returns a boolean indicating whether the program of name program can be found, using the PATH environment variable.

Similar to which on the command line.

ibid.utils.get_process_output(command, input=None)¶

Runs command, a list of arguments where the first argument is the process to run (as in subprocess.Popen). The command will be fed input on standard input, and get_process_output() will block until the command exits.

Returns a tuple of (output, error, code): standard output, standard error, and exit code.

ibid.utils.unicode_output(output[, errors='strict'])¶

Decodes output a string, to unicode, using the character set specified in the LANG environment variable. errors has the same behaviour as the builtin unicode().

Useful for parsing program output.

ibid.utils.ibid_version()¶

Return the current Ibid version or None if no version can be determined.

ibid.utils.locate_resource(path, filename)¶

Locate a resource shipped with Ibid. path is specified as a python package (e.g. 'ibid'). filename is the relative path within the package (e.g. 'data/something.txt')

Returns the filename to the resource.

ibid.utils.get_country_codes()¶

Retrieve and decode a list of ISO-3166-1 country codes.

Returns a dict of code -> country_name. The codes are capitalised.

ibid.utils.identity_name(event, identity)¶

Refer to identity naturally in response to event.

URL Functions¶

ibid.utils.url_regex()¶

Returns a regular expression string (not a re.RegexObject) for matching a URL.

ibid.utils.is_url(url)¶

Is url a valid URL? (according to url_regex())

ibid.utils.iri_to_uri(iri)¶

Convert a unicode iri to punycode host and UTF-8 path. This allows IRIs to be opened with urllib.

Web Service Functions¶

ibid.utils.cacheable_download(url, cachefile[, headers, timeout=60])¶

Useful for data files that you don’t want to keep re-downloading, but do occasionally change.

url is a URL to download, to a file named cachefile. cachefile should be in the form of pluginname/filename. It will be stored in the configured plugins.cachedir and the full filename returned. Extra HTTP headers in headers can be supplied, if necessary.

If cachefile already exists, cacheable_download() will do an If-Modified-Since HTTP request. It handles HTTP-compression.

Example:

filename = cacheable_download(
   'http://www.iso.org/iso/country_codes/iso_3166_code_lists/iso-3166-1_decoding_table.htm',
   'lookup/iso-3166-1_decoding_table.htm')

ibid.utils.generic_webservice(url[, params, headers])¶

Request url, with optional dicts of parameters params and headers headers, and return the data.

ibid.utils.json_webservice(url[, params, headers])¶

Request url, with optional dicts of parameters params and headers headers, and parse as JSON.

JSONException will be raised if the returned data isn’t valid JSON.

exception ibid.utils.JSONException(Exception)¶

Raised by json_webservice() if invalid JSON is returned.

ibid.utils.html – HTML Parsing¶

ibid.utils.html.get_html_parse_tree(url[, data, headers, treetype='beautifulsoup])¶

Request url, and return a parse-tree of type treetype. data and headers are optionally used in the request.

treetype can be any type supported by html5lib, most commonly 'etree' or 'beautifulsoup'.

ContentTypeException will be raised if the returned data isn’t HTML.

exception ibid.utils.html.ContentTypeException(Exception)¶

Raised by get_html_parse_tree() if the content type isn’t HTML.