Analyzers and tokenizers
Why can you only ask questions in the vocabulary the analyzer kept?
Matching happens on what the analyzer emitted, never on what the author wrote. An analyzer transforms text at write time into tokens through steps such as:
- split (break on spaces and punctuation),
- case-fold (make lowercase),
- stem (cut words to a root: "running" to "run"),
- stop (drop very common words like "the"),
- normalize (for example remove accents).
The query is transformed by the same rules. Both halves must agree, so the analyzer is a contract. It is a schema-on-write contract for language (early binding). Change it and the index no longer answers questions in the same vocabulary it was built from, which forces a full rebuild. This is the schema-evolution problem arriving in a place where nobody calls it a schema.
Normalization is one-way
The deeper constraint is informational: normalization destroys distinctions at write time, and no read-time cleverness recovers them.
- Case folding makes
appleandApplethe same question forever (Amara's fourth oddity). - Stemming makes
universeanduniversitycollide in some stemmer designs.
Each normalization trades precision for recall, and the trade is irreversible because the discarded information is not stored.
The escape: index the text several times
The standard practice is to index the same text under several different analyses and combine them at query time:
| Field | Analysis | Buys |
|---|---|---|
| stemmed field | stem + lowercase | recall |
| exact field | no change | precision |
| n-gram field | cut into overlapping pieces of n characters (for "kurta", 3-grams: "kur", "urt", "rta") | partial matching |
The other place recall is bought back is the query side: rewriting, expansion, synonyms and spell correction. It is a linguistic rewrite, not an algebraic one. If the analyzer discarded a distinction, the query side can sometimes re-add it ("kurti" is rewritten to also search "kurta").
Indexing several times is materialization: you buy back the lost distinctions by storing the text under multiple lossy projections instead of one. The cost is a multiple on index size and write amplification, paid per analysis variant.
Two honest notes
- Tokenization assumptions are linguistically contingent, not universal. Splitting on spaces is an accident of some writing systems. It fails outright for Chinese, Japanese and Korean (CJK), which do not put spaces between words. In compounding languages (such as German) the correspondence between "token" and "word" is a modelling decision.
- Stemming is folklore-adjacent. Aggressive stemmers give mixed results in tests, and the field's own evaluations have gone back and forth for decades. Treat stemmer choice as a tunable with measurable effects, not as a correctness matter.
One line: analysis is schema-on-write for language; you can only ask questions in the vocabulary the analyzer emitted, and what it discarded is gone.