Building a Swahili ASR benchmark

It is easy to make a speech recognition model look good. Test it on clean studio recordings of people reading news scripts, and the numbers will be flattering. Then a real caller phones in from a daladala with the windows open, mixes Swahili and English in one sentence, and the model falls apart.

We are building Waridi Sikia (sikia-1) for the second situation. So we need a benchmark built for it too. This post explains how we are designing that benchmark, what we are measuring and the traps we are trying to avoid.

Why existing test sets are not enough

Public Swahili test sets exist, and they are useful. But most of them share a few gaps:

• Read speech, not conversation. People reading prompts speak slowly and clearly. Callers do not.
• Wideband audio. Many sets are recorded at 16 kHz or higher on decent microphones. A huge share of real customer-service audio arrives over phone networks at 8 kHz.
• Narrow accent coverage. Speakers often cluster in one city or one demographic.
• Code-switching is cleaned out. Sentences with English words are sometimes filtered away, which removes exactly the speech we most need to handle.

A benchmark that ignores these gaps tells us nothing about whether Sikia will work in a call centre in Mwanza.

What goes into the benchmark

We are building the test set in slices. Each slice targets one real-world condition, so we can see where the model is strong and where it breaks.

Accents and regions

We sample speakers from across Tanzania and, increasingly, Kenya: Dar es Salaam, the coast and Zanzibar, the northern zone around Arusha and Moshi, the Lake zone, the southern highlands, and Nairobi. Speakers whose first language is not Swahili are included deliberately. A Sukuma or Chaga speaker's Swahili is Tanzanian Swahili, and our model has to handle it.

Narrowband phone audio

A dedicated slice is recorded over real phone calls, or transcoded through telephony codecs, at 8 kHz. Narrowband audio removes high-frequency detail, which makes sounds like s and f harder to tell apart. That matters in Swahili, where single consonants change meaning.

Background noise

We collect recordings in the places people actually call from:

• markets and shops, with overlapping voices;
• buses and daladala, with engine noise and conductors calling stops;
• outdoors in wind;
• homes with radios, TVs and children in the background.

We record noise in context rather than only mixing synthetic noise into clean speech. Artificial mixing is useful for training, but people speak differently when it is noisy. They raise their voices and repeat themselves. A benchmark should capture that.

Code-switching and Sheng

A sentence like "Nataka ku-check balance yangu, lakini app inagoma" is ordinary Tanzanian speech. The benchmark includes a large share of naturally code-switched utterances and a Nairobi slice with Sheng. We cover why this matters in Code-switching is not an edge case.

Transcription conventions

A benchmark is only as good as its reference transcripts. Before anyone transcribes a word, we write down the rules. Some of the questions we had to answer:

• English words inside Swahili. We write them in standard English spelling ("check", "balance"), not phonetically.
• Mixed-morphology words. "ku-check" is one word with a Swahili prefix. We pick one form and apply it everywhere.
• Numbers. Do we write "elfu tano" or "5,000"? References keep spoken words. Normalisation to digits happens in a separate, documented step.
• Hesitations and fillers. "Eeh", "aah" and "sasa..." are marked consistently, and scored separately.
• Unclear speech. Anything a transcriber cannot hear confidently is tagged, not guessed.

Every utterance is transcribed by one person and checked by a second. Disagreements go to a third reviewer and, where needed, become new rules in the guide.

WER, CER and normalisation pitfalls

Word error rate (WER) is the standard metric, and we report it. But Swahili is agglutinative. A single word like hatutawaambia ("we will not tell them") carries subject, tense, object and verb. Get one morpheme wrong and the whole word counts as an error, even if most of it was right.

So we also report character error rate (CER), which is more forgiving of small slips inside long words and better at showing gradual progress.

Normalisation is where benchmarks quietly cheat. Lowercasing, removing punctuation and turning numbers into digits can move scores a lot. Our rules:

1. Publish the exact normalisation code with the benchmark.
2. Apply the same normalisation to every system we compare.
3. Report raw and normalised scores side by side.
4. Never tune normalisation rules after looking at test results.

Held-out speakers

Every speaker in the benchmark is excluded from training data. Not just their test utterances: all of their recordings. If a model has heard a person before, it can learn their voice instead of the language, and the score stops meaning anything. We track this at the speaker-code level described in How we collect speech data with consent.

We also keep a small "locked" slice that no one on the modelling team sees, used only for major releases.

Publishing the methodology

Our goal is to publish the benchmark design, transcription guide and normalisation code, and to report Sikia's results per slice rather than as one headline figure. Where speaker consent allows, we would like to release part of the audio so others can test their own systems.

We have not published numbers yet, and we will not until the benchmark is complete and stable. When we do, you will see where the model is weak as well as where it is strong.

If you are a researcher working on African-language speech and want to compare notes, write to hello@waridi.io.