Have you ever wondered how a machine learns to understand a language, not from text typed neatly on a screen, but from real human speech spoken in buses, markets, busy offices, or quiet homes? Speech full of interruptions, background noise, accents, laughter, and hesitation?
That was the question we wanted to answer when we entered the AI for Good Swahili ASR Challenge on Zindi. We ended up winning the competition.
Our system achieved a word error rate of 17.81% on the private leaderboard and 18.22% on the public leaderboard. It processed each recording in about 1.24 seconds and transcribed the full test set of 4,089 recordings in one hour and twenty-four minutes. We trained the entire system using a single borrowed GPU.
But the score is not the most important part of this work.
What mattered to us was that the challenge reflected many of the same conditions we face when building speech technology for African languages: noisy recordings, limited computing power, and languages that are still poorly represented in the largest AI models.
Although Swahili is not Hausa, but many of the technical problems are similar. Much of the speech data available online is read speech. Someone sits in a quiet room, uses a good microphone, and reads prepared sentences. This data is clean and easy to label, but models trained only on it often struggle when they hear real conversations.
Real people pause, interrupt each other, change their minds, speak in different accents, and pronounce words differently depending on where they come from. They also speak in places with traffic, music, wind, markets, and other people talking nearby.
For this reason, we trained our model using Sunbird’s SALT dataset, a conversational Swahili corpus with several speakers and natural dialogue.
The data was more difficult to work with, but it was much closer to the environment in which the model would eventually be used.
We did not train the model from the beginning.
We started with a distilled Whisper checkpoint that had already been trained on Swahili data from Common Voice 12. This gave the model a basic understanding of how Swahili sounds before we began fine-tuning it.
A distilled model is a smaller model trained to reproduce much of the performance of a larger one. It requires less memory and computing power while keeping much of the original model’s accuracy.
This was important because the challenge was not simply to build an accurate system. The system also needed to be small and efficient enough to run on a device instead of depending on a large cloud server.
Since the starting model already understood some Swahili, we could focus our limited training resources on helping it handle real conversations and difficult audio.
One common way to handle noisy recordings is to clean the audio before transcription. The system tries to remove the sound of traffic, markets, or other conversations before sending the recording to the speech model.
We took a different approach. Instead of trying to remove the noise, we trained the model to expect it.
During training, our custom data collator selected sounds from a collection of urban East African noise, including traffic, markets, and nearby conversations. It mixed those sounds into the training recordings at different volume levels.
The model heard a different combination each time. It could not simply memorize one kind of background noise. It had to learn how to recognize speech even when other sounds were present.
This may be the most important part of the entire system. A model that has only heard clean studio recordings will struggle the first time someone tries to use it on a bus, beside a road, or inside a busy market.
We trained the model on a single NVIDIA T4 GPU with 16 GB of memory using a free Kaggle notebook.
That limitation shaped the way we built the system.
Instead of updating every part of Whisper, we froze the main model and trained small LoRA adapters inside its attention layers. This allowed us to adjust the model while training only a small percentage of its parameters.
We also loaded the model in 8-bit precision to reduce its memory requirements. Mixed-precision training and gradient checkpointing helped us save even more memory.
With these changes, we trained with a batch size of four and used gradient accumulation to reach an effective batch size of eight. We used a learning rate of 1e-5, 500 warm-up steps, and selected the checkpoint with the lowest validation error.
These methods are not unusual or experimental. They are standard tools for training models with limited resources.
That is exactly why they matter.
Building a competitive speech model for an under-resourced language did not require access to a large research laboratory. We trained it using hardware that a student could access for free. For many people working on African languages, that is the reality.
During transcription, we used beam search with three possible outputs and a repetition penalty of 1.2.
A simpler method, called greedy decoding, chooses the most likely next word at every step and never goes back. This can work well on clean audio. On noisy recordings, however, the model may make an early mistake, lose track of the sentence, and begin repeating the same words.
Beam search keeps several possible transcriptions open at the same time. This requires slightly more processing, but it gives the model a better chance of recovering from uncertainty.
We did not use a separate language model or combine several models into an ensemble. Both approaches might have improved the final score, but they would also have made the system larger and more difficult to deploy.
Our goal was not only to perform well on the leaderboard. We wanted to build something that could realistically be used.
A word error rate of 17.81% means that roughly one out of every six words is transcribed incorrectly.
That is not accurate enough for tasks that require a perfect written record, such as professional dictation.
But it can still be useful when the goal is to understand what someone is asking.
Even when one word is incorrect, the rest of the sentence often provides enough context to understand the speaker’s meaning. This distinction between producing a perfect transcript and understanding a person’s request is important for the systems we are building.
The result also belongs to the specific test set used in the competition: 16 kHz mono recordings with an average length of slightly more than six seconds.
We would not expect the exact same result from every microphone, dialect, speaker, or environment. A word error rate only has meaning when the conditions under which it was measured are also explained.
This challenge gave us four lessons that we are now applying directly to our work in Hausa.
First, conversational data is more valuable than perfectly clean recordings when the system will be used in real conversations.
Second, noise should be part of the training process. We cannot assume that people will always speak from quiet rooms or through expensive microphones.
Third, parameter-efficient fine-tuning makes serious speech research possible without large computing budgets. This is especially important for the researchers and communities closest to under-resourced languages.
Finally, decoding. On difficult audio, the difference between greedy decoding and beam search can be the difference between a useful answer and a repeated sentence that goes nowhere.
Hausa faces many of the same challenges as Swahili, but with even fewer datasets, models, and technical resources available.
The complete training and inference code is available at github.com/SalifouAbdourahamane/swahili_asr_sota_model. A more detailed technical explanation of the solution is also available on Zindi.
