Arabic search that finds أحمد when you type احمد
People type Arabic inconsistently. أحمد comes in as احمد, مدرسة as مدرسه, numbers as ٢٥ or 25, sometimes with diacritics. A plain text search treats each spelling as a different word.
My Android reminders app handles it with one rule from its arabic-rtl skill: the stored text is never changed, a normalized copy is used only for matching, and exactly one function produces it. That function is called in two places: when a message is written to the index, and when a query is built. A third copy, or an inline variant, is a bug.
The normalizer
From the app’s ArabicNormalizer.kt:
object ArabicNormalizer {
fun normalize(raw: String): String = buildString(raw.length) {
for (c in raw) {
when (val cp = c.code) {
in 0x0610..0x061A -> {} // Quranic/Arabic signs: strip
in 0x064B..0x065F -> {} // harakat (tashkeel): strip
0x0670 -> {} // superscript alef: strip
0x0640 -> {} // tatweel (kashida): strip
0x0623, 0x0625, 0x0622 -> append(0x0627.toChar()) // أ/إ/آ → ا
0x0629 -> append(0x0647.toChar()) // ة → ه
0x0649 -> append(0x064A.toChar()) // ى → ي
in 0x0660..0x0669 -> append('0' + (cp - 0x0660)) // ٠–٩ → 0–9
in 0x06F0..0x06F9 -> append('0' + (cp - 0x06F0)) // ۰–۹ (Persian/Urdu) → 0–9
else -> append(c)
}
}
}.lowercase() // for the Latin side of mixed text
}
The logic uses code points, not Arabic literals, so it can’t change meaning if an editor mangles the file’s encoding. Only the comments carry Arabic. The skill stops at these foldings: anything more, like ء variants or ک/ك, counts as a spec change, and the agent has to ask first.
The query builder
From SearchQuery.kt:
object SearchQuery {
private val separators = Regex("[^\\p{L}\\p{N}]+")
fun build(raw: String): String? {
val tokens = ArabicNormalizer.normalize(raw)
.split(separators)
.filter { it.isNotEmpty() }
if (tokens.isEmpty()) return null
return tokens.joinToString(" ") { "\"$it\"*" }
}
}
Splitting on anything that isn’t a letter or digit drops the characters FTS reads as operators (" * ( ) : -), so user punctuation can’t cause a syntax error. Each token is quoted and gets a *, which makes it find-as-you-type. Blank or punctuation-only input returns null, and the app skips the query. The code’s comment says an empty MATCH would be an error. That’s true of FTS5, and the app ended up on FTS4, where it just returns nothing.
The six tests in SearchQueryTest cover the builder: "foo* -bar:" becomes "foo"* "bar"*, ٢٥ becomes "25"*, *:-()" gives null. All six passed on the run of 2026-07-06. They test the builder and that it goes through the normalizer, not a search against the real index on a device.
Room
The index is an external-content FTS4 table over the normalized column:
@Fts4(contentEntity = MessageEntity::class, tokenizer = FtsOptions.TOKENIZER_UNICODE61)
@Entity(tableName = "message_fts")
data class MessageFts(val normalized_text: String?)
The app’s spec asked for FTS5. Room has an annotation for FTS4 and generates the sync triggers itself; FTS5 would mean hand-written DDL, triggers and raw queries. The skill records the call: for a personal reminders app the result is the same, and moving to FTS5 later is a data rebuild, not a schema trap. unicode61 only splits words, because the normalizer has already done the Arabic part on both sides.
For your own skill
These rules combine the skill’s search section with what the code does, so an agent working on the app keeps them intact:
Arabic search: one normalizer, two call sites
Raw text is never changed. Normalization produces a derived copy used only for matching.
- Exactly ONE normalization function. It is called in exactly two places: writing the normalized column on create/edit, and building the FTS query. A third copy or an inline variant is a bug.
- Foldings: strip tashkeel, superscript alef and tatweel; أ/إ/آ → ا; ة → ه; ى → ي; Arabic-Indic and Persian digits → ASCII; lowercase for Latin. Anything beyond that (ء variants, ک/ك) is a spec change: ask, don't slip it in.
- Display always renders the raw text.
- Query: normalize, split on non-letter/non-digit runs (this removes FTS operator
characters), quote each token and add
*for prefix search. Blank or punctuation-only input runs no query. - Accept Arabic-Indic and Latin digits in every numeric input. Parse liberally, store canonically; the normalizer's digit fold doubles as input scrubbing.
## Arabic search: one normalizer, two call sites
Raw text is never changed. Normalization produces a derived copy used only for matching.
- Exactly ONE normalization function. It is called in exactly two places: writing the
normalized column on create/edit, and building the FTS query. A third copy or an inline
variant is a bug.
- Foldings: strip tashkeel, superscript alef and tatweel; أ/إ/آ → ا; ة → ه; ى → ي;
Arabic-Indic and Persian digits → ASCII; lowercase for Latin. Anything beyond that
(ء variants, ک/ك) is a spec change: ask, don't slip it in.
- Display always renders the raw text.
- Query: normalize, split on non-letter/non-digit runs (this removes FTS operator
characters), quote each token and add `*` for prefix search. Blank or punctuation-only
input runs no query.
- Accept Arabic-Indic and Latin digits in every numeric input. Parse liberally, store
canonically; the normalizer's digit fold doubles as input scrubbing.
see also: Rtl Arabic Web Skill, Kotlin For React Devs Skill