Chapter 4. Programming interface

Recoll has an Application Programming Interface, usable both for indexing and searching, currently accessible from the Python language.

Another less radical way to extend the application is to write input handlers for new types of documents.

The processing of metadata attributes for documents (fields) is highly configurable.

4.1. Writing a document input handler

Terminology

The small programs or pieces of code which handle the processing of the different document types for Recoll used to be called filters, which is still reflected in the name of the directory which holds them and many configuration variables. They were named this way because one of their primary functions is to filter out the formatting directives and keep the text content. However these modules may have other behaviours, and the term input handler is now progressively substituted in the documentation. filter is still used in many places though.

Recoll input handlers cooperate to translate from the multitude of input document formats, simple ones as opendocument, acrobat), or compound ones such as Zip or Email, into the final Recoll indexing input format, which is plain text. Most input handlers are executable programs or scripts. A few handlers are coded in C++ and live inside recollindex. This latter kind will not be described here.

There are currently (1.18 and since 1.13) two kinds of external executable input handlers:

  • Simple exec handlers run once and exit. They can be bare programs like antiword, or scripts using other programs. They are very simple to write, because they just need to print the converted document to the standard output. Their output can be plain text or HTML. HTML is usually preferred because it can store metadata fields and it allows preserving some of the formatting for the GUI preview.

  • Multiple execm handlers can process multiple files (sparing the process startup time which can be very significant), or multiple documents per file (e.g.: for zip or chm files). They communicate with the indexer through a simple protocol, but are nevertheless a bit more complicated than the older kind. Most of new handlers are written in Python, using a common module to handle the protocol. There is an exception, rclimg which is written in Perl. The subdocuments output by these handlers can be directly indexable (text or HTML), or they can be other simple or compound documents that will need to be processed by another handler.

In both cases, handlers deal with regular file system files, and can process either a single document, or a linear list of documents in each file. Recoll is responsible for performing up to date checks, deal with more complex embedding and other upper level issues.

A simple handler returning a document in text/plain format, can transfer no metadata to the indexer. Generic metadata, like document size or modification date, will be gathered and stored by the indexer.

Handlers that produce text/html format can return an arbitrary amount of metadata inside HTML meta tags. These will be processed according to the directives found in the fields configuration file.

The handlers that can handle multiple documents per file return a single piece of data to identify each document inside the file. This piece of data, called an ipath element will be sent back by Recoll to extract the document at query time, for previewing, or for creating a temporary file to be opened by a viewer.

The following section describes the simple handlers, and the next one gives a few explanations about the execm ones. You could conceivably write a simple handler with only the elements in the manual. This will not be the case for the other ones, for which you will have to look at the code.

4.1.1. Simple input handlers

Recoll simple handlers are usually shell-scripts, but this is in no way necessary. Extracting the text from the native format is the difficult part. Outputting the format expected by Recoll is trivial. Happily enough, most document formats have translators or text extractors which can be called from the handler. In some cases the output of the translating program is completely appropriate, and no intermediate shell-script is needed.

Input handlers are called with a single argument which is the source file name. They should output the result to stdout.

When writing a handler, you should decide if it will output plain text or HTML. Plain text is simpler, but you will not be able to add metadata or vary the output character encoding (this will be defined in a configuration file). Additionally, some formatting may be easier to preserve when previewing HTML. Actually the deciding factor is metadata: Recoll has a way to extract metadata from the HTML header and use it for field searches..

The RECOLL_FILTER_FORPREVIEW environment variable (values yes, no) tells the handler if the operation is for indexing or previewing. Some handlers use this to output a slightly different format, for example stripping uninteresting repeated keywords (ie: Subject: for email) when indexing. This is not essential.

You should look at one of the simple handlers, for example rclps for a starting point.

Don't forget to make your handler executable before testing !

4.1.2. "Multiple" handlers

If you can program and want to write an execm handler, it should not be too difficult to make sense of one of the existing modules. For example, look at rclzip which uses Zip file paths as identifiers (ipath), and rclics, which uses an integer index. Also have a look at the comments inside the internfile/mh_execm.h file and possibly at the corresponding module.

execm handlers sometimes need to make a choice for the nature of the ipath elements that they use in communication with the indexer. Here are a few guidelines:

  • Use ASCII or UTF-8 (if the identifier is an integer print it, for example, like printf %d would do).

  • If at all possible, the data should make some kind of sense when printed to a log file to help with debugging.

  • Recoll uses a colon (:) as a separator to store a complex path internally (for deeper embedding). Colons inside the ipath elements output by a handler will be escaped, but would be a bad choice as a handler-specific separator (mostly, again, for debugging issues).

In any case, the main goal is that it should be easy for the handler to extract the target document, given the file name and the ipath element.

execm handlers will also produce a document with a null ipath element. Depending on the type of document, this may have some associated data (e.g. the body of an email message), or none (typical for an archive file). If it is empty, this document will be useful anyway for some operations, as the parent of the actual data documents.

4.1.3. Telling Recoll about the handler

There are two elements that link a file to the handler which should process it: the association of file to MIME type and the association of a MIME type with a handler.

The association of files to MIME types is mostly based on name suffixes. The types are defined inside the mimemap file. Example:


.doc = application/msword

If no suffix association is found for the file name, Recoll will try to execute the file -i command to determine a MIME type.

The association of file types to handlers is performed in the mimeconf file. A sample will probably be of better help than a long explanation:


[index]
application/msword = exec antiword -t -i 1 -m UTF-8;\
     mimetype = text/plain ; charset=utf-8

application/ogg = exec rclogg

text/rtf = exec unrtf --nopict --html; charset=iso-8859-1; mimetype=text/html

application/x-chm = execm rclchm

The fragment specifies that:

  • application/msword files are processed by executing the antiword program, which outputs text/plain encoded in utf-8.

  • application/ogg files are processed by the rclogg script, with default output type (text/html, with encoding specified in the header, or utf-8 by default).

  • text/rtf is processed by unrtf, which outputs text/html. The iso-8859-1 encoding is specified because it is not the utf-8 default, and not output by unrtf in the HTML header section.

  • application/x-chm is processed by a persistant handler. This is determined by the execm keyword.

4.1.4. Input handler HTML output

The output HTML could be very minimal like the following example:

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
  </head>
  <body>
   Some text content
  </body>
</html>
          

You should take care to escape some characters inside the text by transforming them into appropriate entities. At the very minimum, "&" should be transformed into "&amp;", "<" should be transformed into "&lt;". This is not always properly done by translating programs which output HTML, and of course never by those which output plain text.

When encapsulating plain text in an HTML body, the display of a preview may be improved by enclosing the text inside <pre> tags.

The character set needs to be specified in the header. It does not need to be UTF-8 (Recoll will take care of translating it), but it must be accurate for good results.

Recoll will process meta tags inside the header as possible document fields candidates. Documents fields can be processed by the indexer in different ways, for searching or displaying inside query results. This is described in a following section.

By default, the indexer will process the standard header fields if they are present: title, meta/description, and meta/keywords are both indexed and stored for query-time display.

A predefined non-standard meta tag will also be processed by Recoll without further configuration: if a date tag is present and has the right format, it will be used as the document date (for display and sorting), in preference to the file modification date. The date format should be as follows:

<meta name="date" content="YYYY-mm-dd HH:MM:SS">
or
<meta name="date" content="YYYY-mm-ddTHH:MM:SS">
          

Example:

<meta name="date" content="2013-02-24 17:50:00">
          

Input handlers also have the possibility to "invent" field names. This should also be output as meta tags:

<meta name="somefield" content="Some textual data" />

You can embed HTML markup inside the content of custom fields, for improving the display inside result lists. In this case, add a (wildly non-standard) markup attribute to tell Recoll that the value is HTML and should not be escaped for display.

<meta name="somefield" markup="html" content="Some <i>textual</i> data" />

As written above, the processing of fields is described in a further section.

4.1.5. Page numbers

The indexer will interpret ^L characters in the handler output as indicating page breaks, and will record them. At query time, this allows starting a viewer on the right page for a hit or a snippet. Currently, only the PDF, Postscript and DVI handlers generate page breaks.