Email search & indexing

Searching a mailbox from webmail or a mail client works out of the box, but without an index the mail server reads every message on disk to answer it. Reqad can turn on Dovecot's full-text search, backed by flatcurve (Xapian), so searches over a large mailbox come back in a moment instead of a minute.

Two different indexes

Dovecot keeps two kinds of index inside each mailbox, and it is worth knowing which is which:

• The mailbox index (dovecot.index, dovecot.index.cache) is always there. It tracks message flags, sizes and headers so a client can open a folder without re-reading the whole maildir. It needs no setup and no maintenance.
• The full-text index (fts-flatcurve/) is optional and is what this page is about. It holds the words in every message body and attachment, so searching for a word becomes a lookup instead of a scan.

Both live inside the mailbox directory itself (/home/<user>/mail/<domain>/<mailbox>/), which means they are covered by the normal account backup and are removed with the account. Nothing extra to clean up.

Why flatcurve

The usual full-text backend for older Dovecot installs was Solr, which means running a Java service alongside the mail server — far too heavy for a single VPS. Dovecot 2.4 ships flatcurve as an official sub-package: no daemon, no network service, no separate database to administer. The index is just files in the mailbox.

Checking the current state

Open Email in the panel. The Dovecot card shows a Full-text search line with one of:

• Enabled — the plugin, a backend and the configuration are all in place. Search is indexed.
• Installed, not enabled — a backend is present but the configuration does not switch it on.
• No backend installed — Dovecot's own search plugin is there, but nothing to store an index with. On its own it indexes nothing.
• Not installed — the search plugin is missing entirely.

Turning it on

Full-text search is opt-in and is enabled from the shell, not from the panel. Enabling it installs a package, restarts Dovecot and eventually builds an index for every mailbox on the server — that is an administrator's decision, not something an update should do behind your back. Run:

bash /usr/local/reqad/scripts/update/setup_dovecot_fts.sh

The script installs dovecot-flatcurve, writes the configuration, validates it, restarts Dovecot and then proves the backend really initialises against a real mailbox. If anything fails at any step it rolls the change back and leaves Dovecot running as it was. It is idempotent, so it is safe — and worth it — to re-run after a Dovecot package upgrade.

It quietly does nothing when email is disabled for the install, when Dovecot is not present, or when Dovecot is older than 2.4.

Building the index

New mail is indexed as it arrives. Mail that was already in the mailboxes is indexed lazily — the first search that touches an unindexed folder indexes it on the way past — so you can simply enable search and let it fill in. To build everything up front instead:

doveadm index -A -q '*'

-A means every user and '*' every folder. The -q flag queues the work for Dovecot's indexer process and returns immediately; without it the command does the indexing itself and blocks until it is finished. On a server with users connected, prefer -q: it keeps a single writer per mailbox and avoids two processes fighting over the same index.

Check progress, or confirm a mailbox is fully indexed:

doveadm fts flatcurve stats -u user@example.com '*'

The messages= figure is the number of indexed messages per folder. last_uid= is the highest message ever indexed, not a count — a low message count beside a high last UID usually just means mail has been deleted.

Disk space

Expect the index to be roughly 0.5–1% of the mail it covers — a 75 GB mailbox measured at about 450 MB. That is much smaller than the 3–8% often quoted for full-text indexes, because Reqad turns off substring matching: the index stores whole words and prefixes, which is what mail clients actually search for, rather than every fragment of every word.

An optimize pass builds the merged index alongside the existing one before replacing it, so keep about twice the index size free while a large reindex is running.

Configuration

The settings live in /etc/dovecot/fts.conf, pulled in from local.conf. Notable choices:

• Search is case- and accent-insensitive, and stemmed — searching for running also finds run. Email addresses stay searchable as whole addresses.
• Messages over 50 MB are not indexed. One enormous attachment would otherwise stall the indexer for everyone.
• Anything missing from the index is indexed at search time, so searches over old mail are correct even before a full reindex has run.

Troubleshooting

Searches still feel slow

If the backend cannot start, Dovecot does not report an error to the client — searches keep working, silently falling back to a full scan of the mailbox. Check the Email overview badge first, then look in /var/log/maillog (not /var/log/dovecot.log) for fts errors, and re-run the setup script.

“No glass database found”

A doveadm command failing with Cannot open DB ... No glass database found is almost never a corrupt index. Flatcurve writes the index as a series of shards, merging and deleting them as they accumulate; if a second process — a live IMAP session, or the indexer handling new mail — is holding a shard while it is merged away, it ends up pointing at a directory that no longer exists.

Two things matter afterwards. First, index through the queue (doveadm index -q) rather than directly, so there is only ever one writer. Second, and easy to miss: after such a crash doveadm index does nothing at all — Dovecot still believes the mailbox is fully indexed, so it returns instantly while messages are missing from the index. Reconcile it first:

doveadm fts rescan -u user@example.com
doveadm index -q -u user@example.com '*'

Starting over for one mailbox

Deleting the fts-flatcurve directories in a mailbox and running doveadm fts rescan for that user gives a clean slate. Nothing is lost — the index is derived data, and the mail itself is untouched — but the mailbox has to be indexed again from scratch.

Related

Search complements, but is separate from, email filters and the rest of the email stack.

Need a hand? Register for early access or contact us.