The average score of multilingual AI conceals inequities across languages.

Global MMLU demonstrated, through expert and community translation across a wide range of languages and analysis of culture-sensitive items, that model rankings and evaluations may change in culture-dependent areas.[1] Correctly translating a word does not guarantee that the question's premises align with the region.

Translation equivalence is important, but it alone does not represent the questions that native speakers actually ask. Add locally created questions that include institutions, calendars, honorifics, proper nouns, laws, educational content, and orthographic variations. Do not treat the English version as the sole prototype for the correct answer.

When using machine translation, native speakers must verify that the difficulty has not changed between the source and target texts. Classify defects such as ambiguity only in the translation, unnaturalness only in the correct answer options, and differences in required knowledge across regions.

The approach of excluding culture-dependent items to produce only neutral scores also conceals real-world operational issues. Report the ability to handle non-culture-dependent skills and the ability to handle regional context as separate scores.

• Translation Equivalence: Verify meaning and difficulty between source and target texts.
• Local Creation: Native speakers create questions relevant to actual use.
• Cultural Categorization: Report dependent and non-dependent items separately.
• Answer Rationale: Clarify regional institutions and reference points in time.

Hu et al. compared multiple model series and multilingual datasets, showing that models with higher overall performance do not necessarily have higher fairness across languages.[3] Even if the average across all languages increases, the gap with the weakest language may widen.

The average score can be used for decision-making that weights high-usage languages, but it does not guarantee the minimum quality for each language promised for provision. Present the average, per-language, worst-language, and inter-language differences side by side, and explicitly state the weights according to the purpose.

Do not consider the inability to detect differences due to small sample sizes in low-speaker languages as equivalence. Show confidence intervals and unverified areas, and suspend automatic provision of critical functions. Prioritize sample expansion based on impact.

If improvements in high-resource languages coincide with regressions in low-resource languages, do not approve based on the overall average. Review model update diffs per language and provide a structure that allows reverting to the previous version or human support for languages that fall below the lower bound.

PARIKSHA conducted large-scale studies on agreement between humans and LLM evaluators across multiple Indian languages and cultures, reporting that direct evaluation agreement varies by language and conditions.[2] Multilingual differences exist not only in generative models but also in scorers.

Allowing an LLM strong in English to score the quality of all languages solely because it is inexpensive may lead to errors in naturalness, respect, and regional knowledge. Use native-speaker evaluations as a baseline set and measure the per-language agreement and bias of LLM evaluators.

Native speakers are not a monolith. Since expression evaluations differ by region, generation, and expertise, record evaluator attributes and consensus. Items with low consensus should be reviewed not as model superiority or inferiority, but as ambiguity in the criteria.

If the evaluation model and the generation model belong to the same series, suspect self-preference. Hide the generation source, reverse the order, and combine different series with human evaluations. Do not confine multilingual evaluation to a single automated judge.

• Native-speaker standard: Create human baseline sets per language.
• Evaluator agreement: Measure human-human and human-LLM agreement separately.
• Regional differences: Record dialect and institutional differences within the same language.
• Disagreement referral: Do not auto-confirm; return for re-review.

Counting all errors as a single item gives equal weight to slight stylistic unnaturalness and errors in payment, safety, or medical guidance. Classify per-language errors as factual errors, instruction misunderstanding, insufficient refusal, excessive refusal, derogatory expressions, and citation errors.

Safety filters may also vary by language. Test cases where dangerous requests rejected in English pass in another language, and conversely, harmless regional expressions are rejected. Include not only translated dangerous texts but also euphemistic expressions actually used locally.

Abstention can serve as a safety valve for low-quality languages, but constantly stating "cannot respond" does not constitute service provision. Distinguish between appropriate abstention, handover to humans, and the safe range of answerable responses, ensuring that relief is not lost due to the user's language choice.

If critical failures concentrate in minority languages, prioritize them even if the total number of cases is small. Place rights and safety lower bounds per language, rather than relying solely on usage-weighted averages.

Avoid a blanket display of "multilingual support"; instead, show for each language the provided functions, the time of quality verification, and excluded high-impact judgments. Distinguish functional differences such as "translation only possible," "search possible," and "procedure completion possible."

Completely dropping languages that fail to meet the lower bound may widen access gaps. Prepare human interpreters, callbacks, document submission, and alternative channels, and measure wait times and completion rates. Verify that alternative pathways are not merely nominal.

Mechanisms for receiving corrections from users should record not only error reports but also the language, region, question type, and impact. Do not consider the low number of reports for minority languages as high quality; test whether the reporting interface itself is usable.

The decision to add provision languages should be made only when native-speaker evaluation, maximum allowed severe failures, relief capabilities, and monitoring personnel are in place. Do not use the number of languages for marketing purposes as a quality indicator.

Model updates do not necessarily move all languages in the same direction. Fix the regression set per language and compare pre- and post-update metrics including average improvement, worst-language performance, critical failures, abstentions, and human evaluation agreement.

Do not directly translate and apply prompts improved in one language to all languages. Since politeness, word order, separators, and citation formats in instructions can alter model behavior, verify per language while maintaining common principles.

In monitoring, view usage, correction rates, referral rates to humans, completion times, and re-inquiries per language. Do not determine the cause of low usage solely from usage logs, distinguishing between low demand and attrition due to low quality.

Multilingual fairness does not mean achieving the same score in all languages. It means maintaining the same safety lower bounds and relief possibilities for promised functions, and not hiding remaining differences and unverified areas from users.

When evaluation resources are limited, prioritizing solely by speaker count may delay high-impact use cases for minority languages. Combine user count, rights impact, known failures, and lack of alternative support, and retain the selection rationale in a publicly explainable form.

For adding new languages, approval requires not only translation staff but also domain experts, native speakers, safety personnel, and relief contacts. Do not entrust the validity for all regions to a single fluent speaker.

Even when discontinuing a language, do not suddenly remove the option. Inform affected users of the reasons and alternatives, and provide a grace period to complete ongoing procedures. Ensure that quality-based discontinuation does not directly lead to loss of access rights.

When using external translation services or evaluators, verify confidentiality, data location, secondary use, and the working conditions of evaluators. The supply chain that creates multilingual quality is also subject to governance.

The final provision table should show, rather than the number of supported languages, the pass/fail status by function, the final evaluation date, critical known limitations, and connection time to humans. The goal is to enable users to assess usage risks in their own language.

The language-by-function service table records model version, last evaluation date, and next review deadline, plus a language-specific quality minimum and maxima for severe harm and time to human escalation. A function stops in that language if it breaches a limit or misses its review deadline, with users routed to a human alternative. Service returns only after the local operations owner confirms native-speaker retesting and remedy for affected users and the language product owner reapproves it.