The Hinglish Blind Spot: How Code-Mixed Queries Confuse AI Answer Engines

The typing habit the internet was never built around

Typing "kya price hai" or "kal available ho?" into a phone produces a perfectly normal Indian sentence that is neither Hindi nor English. It is code-mixing: two languages blended inside a single utterance, written in Roman script because Indian keyboards default to it and switching to a native-script layout is friction most people never bother with. Linguists have studied this pattern for decades under the code-mixing and code-switching literature; the shorthand "Hinglish" is the popular name for the Hindi-English version, and equivalent patterns exist for Tamil, Bengali, Telugu, and most other Indian languages typed in Roman letters.

The scale is not a niche case. The "Internet in India 2024" report from the Internet and Mobile Association of India and Kantar put active internet users at 886 million in 2024, an 8 percent year-on-year gain, on track to pass 900 million in 2025, and found that 98 percent of users accessed content in an Indic language, with rural India, at 488 million users, now the majority of the online population. Separately, the government's Ministry of Micro, Small and Medium Enterprises reports that over 7.83 crore, 78.3 million, enterprises had registered on the Udyam Registration Portal as of the end of February 2026. Put those two facts together and the population an Indian MSME actually serves, and the population typing to find it, overlaps heavily with people whose default written mode is Roman-script, code-mixed Indic language, not the clean English a website's copy is usually written in.

That gap matters more now than it did five years ago, because the thing reading the query has changed. A keyword-matching search index could tolerate messy, inconsistent spelling reasonably well; a generative answer engine has to parse the query, understand intent, and often generate a response in the same register the user asked in. The evidence below is about what happens at that second step.

What code-mixed, romanized text actually does to a model

The core technical problem is not that Hindi and English are mixed together; models handle multilingual text reasonably well when each language stays in its own script. The problem is romanization. Hindi has a standard script, Devanagari, but when it is typed in Roman letters there is no equivalent standard. "Why" can be "kyu," "kyun," "kyon," or "kyu'n" depending on the person typing, and none of those spellings is more correct than another. A model trained mostly on standard-orthography text meets a token it was never taught to normalize, and normalization, deciding what a noisy spelling variant actually means, is exactly the step that breaks.

The evidence for this is concrete. COMI-LINGUA, the largest manually annotated Hindi-English code-mixed dataset published to date at over 125,000 expert-annotated instances spanning both Devanagari and Roman script, tested large language models across five core tasks and found a consistent pattern: models score well on language identification, telling which words are Hindi and which are English, but drop off on part-of-speech tagging and named entity recognition, the tasks that determine who or what a sentence is actually about. That is the difference between a model that can tell a query is bilingual and a model that can tell what the query is asking for.

A separate line of research makes the causal link explicit. A 2026 study in ACM Transactions on Asian and Low-Resource Language Information Processing tested "deromanization," converting Hindi-English code-mixed text from Roman script back into Devanagari before running it through classifiers, and found the conversion step alone improved F1 scores by a minimum of 3 points on toxic-comment classification and sentiment analysis after fine-tuning a multilingual model. The content of the message did not change; only the script did. That the accuracy moved anyway is fairly direct evidence that the romanization itself, not just the code-mixing, is a measurable source of model error.

The benchmarks built specifically to test this

Enough researchers have hit this same wall that a small cluster of benchmarks now exists to measure it directly, rather than treating it as an edge case inside a broader multilingual test.

Voice is making the gap louder, not quieter

India is unusually voice-first. Google SVP Prabhakar Raghavan has said that about a third of Google's search queries in India are spoken, roughly ten times the 3 to 4 percent share in the United States, and that the company is deliberately "training mixed language models alongside pure language models," citing Hinglish by name as one of the dialects it is building for. Spoken Hinglish inherits every problem of typed Hinglish and adds a harder one: automatic speech recognition has to guess both the words and the language boundary from audio alone, with no keyboard habit to fall back on.

Independent measurement of exactly how large that speech-recognition gap is remains thin and mostly industry-reported rather than academically audited. Deepgram, an ASR vendor, cited the SwitchLingua benchmark, a NeurIPS 2025 dataset of 420,000 code-switched text samples and more than 80 hours of recorded speech across 12 languages, and reported that word error rate across six models tested on identical Hinglish audio ranged from roughly 27 percent to roughly 70 percent. That range is wide enough, and industry-sourced enough, that it should be read as directional rather than a settled figure: it shows real, large variance between systems on the same audio, not a single agreed error rate for Hinglish speech recognition.

The industry response exists, but it is early and uneven

The gap has not gone unnoticed by the companies building Indic-language models. Sarvam AI, an Indian AI lab, built a 22-language evaluation benchmark for its 2026 model releases that deliberately scores every language in two separate script conditions, native script for "formal written usage" and romanized Latin script for the "colloquial usage commonly seen in messaging and online communication," treating the two as different enough to require separate measurement rather than assuming one score covers both. That a frontier Indic lab found it necessary to build two tracks is itself a signal that romanized, colloquial input is not yet a solved problem inside the models people actually use.

At the policy level, the government's Bhashini mission, the National Language Translation Mission under the Ministry of Electronics and Information Technology, has published over 1,000 pre-trained language AI models across more than 22 scheduled languages as public digital infrastructure. Its stated purpose is native-script and voice access in Indian languages; it is not, at this stage, a romanization-specific or code-mixing-specific initiative, and no public documentation from Bhashini claims to have closed the Hinglish gap described above. It is worth naming as evidence that institutional attention exists, not as evidence that the problem is solved.

What is not yet measured, and what measuring it would take

Here is the limit of the evidence. The research above establishes, with real datasets and peer-reviewed benchmarks, that large language models measurably underperform on romanized, code-mixed Hindi-English text in general, and that the underperformance grows with code-mixing density. What no public dataset yet measures is the specific downstream question that matters to an Indian MSME: when a real customer asks an AI answer engine a Hinglish, small-business query, "sabse acha AC repair near me kaun sa hai," how often does the engine correctly parse intent, correctly resolve the local business being asked about, and surface it in the answer, compared with the same question asked in clean English or in Devanagari.

That specific measurement does not exist as a published study as of this writing. Building it would require a labeled sample of real Hinglish local-intent queries, a controlled comparison against English and native-script equivalents of the same intent, and a way to audit which businesses actually got surfaced in the generated answer versus a classic search result for the same query, run across more than one engine since behavior differs by model. Until that study exists, the claim that holds is the narrower one the benchmarks actually support: the text format itself is a documented source of model error, and it is the format most of an MSME's actual customers type in.

How to read this

Two things can be true at once. The underlying linguistic and NLP evidence is solid and peer-reviewed: romanized code-mixed text lacks a standard orthography, and multiple independent benchmarks, built by different teams for different purposes, all find that large language models lose accuracy on it, with a deromanization study showing the script conversion alone recovers measurable performance. What is not yet established is the size of the practical consequence for a specific Indian small business trying to be found in an answer an AI engine generates, because that particular measurement has not been published.

The useful response to that gap is not to wait for the perfect study before acting, and not to assume the worst without evidence either. It is to treat machine legibility in the language a business's actual customers use, not just the language its website is written in, as one more surface worth checking directly, the same way a business would check whether its address is correct on a map listing rather than assuming it is.