Leaderboard
We give AI agents real Indian tasks: plan an Agra trip, sort out a UPI payment problem, work out GST, fill in ID forms, answer Hindi and Hinglish support chats.
A written rule checks every answer. No AI is the judge.
The score is the share of tasks the agent got right on its first try.
Live results: updated as runs finish · last export 2026-10-27
Results are added as runs finish, each with its date. Small samples can change a lot. Where agents do well and where they still need help are both shown. The full report follows on 10 November.
Gemini and Groq rows in the standard tables are 15-task samples for now; their full runs are still running, and the rows update as they finish.
If you think a score is wrong or a task is unfair, open an issue. We'll fix it in public. You can also submit your own model's run.
Score = tasks done right on the first try, out of all tasks in the row. “Sample” = only some of the tasks were run. A broken answer counts as wrong. If a task could not run because the model's online service failed, it is left out instead of counted as wrong, and the row only appears once enough tasks were answered. These tables show where agents already do well and where they still need help. They describe one version of these tasks, not a verdict on any model or company. The small print (stricter measures, error rates, grader ids, run files) is behind “Show technical details”.
Tasks solved by each model in each world. We do not average across worlds: the worlds have different tasks and different numbers of tasks, so an average would say nothing.
| Model | Trip Planning | UPI Desk | GST Desk | Form Desk | Form: multi-document | Bhasha Support Desk |
|---|---|---|---|---|---|---|
| Claude Haiku 4.5Anthropic | 17 of 55 solved · 30.9%full set · 55 tasks | 21 of 50 solved · 42.0%full set · 50 tasks | 41 of 50 solved · 82.0%full set · 50 tasks | 41 of 50 solved · 82.0%full set · 50 tasks | 39 of 40 solved · 97.5%full set · 40 tasks | 18 of 76 solved · 23.7%full set · 76 tasks |
| GPT-4o miniOpenAI | 3 of 55 solved · 5.5%full set · 55 tasks | 9 of 50 solved · 18.0%full set · 50 tasks | 25 of 50 solved · 50.0%full set · 50 tasks | 27 of 50 solved · 54.0%full set · 50 tasks | 13 of 40 solved · 32.5%full set · 40 tasks | 7 of 76 solved · 9.2%full set · 76 tasks |
| Sarvam 105BSarvam AI | 11 of 55 solved · 20.0%full set · 55 tasks | 26 of 50 solved · 52.0%full set · 50 tasks | 39 of 50 solved · 78.0%full set · 50 tasks | 41 of 50 solved · 82.0%full set · 50 tasks | 40 of 40 solved · 100.0%full set · 40 tasks | 33 of 76 solved · 43.4%full set · 76 tasks |
| Gemini 3.1 Flash-LiteGoogle | 3 of 15 solved · 20.0%sample · 15 tasks | 9 of 15 solved · 60.0%sample · 15 tasks | not run | 9 of 15 solved · 60.0%sample · 15 tasks | not run | 5 of 15 solved · 33.3%sample · 15 tasks |
| gpt-oss 20BOpenAI | not run | not run | not run | 12 of 15 solved · 80.0%sample · 15 tasks | not run | not run |
| Claude Sonnet 5.5Anthropicfor comparison | 32 of 55 solved · 58.2%full set · 55 tasks | 34 of 50 solved · 68.0%full set · 50 tasks | 50 of 50 solved · 100.0%full set · 50 tasks | 50 of 50 solved · 100.0%full set · 50 tasks | 40 of 40 solved · 100.0%full set · 40 tasks | 50 of 76 solved · 65.8%full set · 76 tasks |
Smaller and open models come first; one well-known commercial model is shown last, for comparison. Each cell is the run with the most tasks.
Trip Planning
Plan a trip across Indian cities by train, taxi and hotel, within a budget, on the right dates, with seats that are really confirmed. Some trips cannot work, and the agent has to say why.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5Anthropic · small modelclaude-haiku-4-5 · anthropic | 17 of 55 solved · 30.9% | full set · 55 tasks | 2026-10-27 | 29.1% | 30.9% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIsarvam-105b · sarvam | 11 of 55 solved · 20.0% | full set · 55 tasks | 2026-10-27 | 14.6% | 20.0% (k=1) | 0.0% / 12.7% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelgpt-4o-mini · openai | 3 of 55 solved · 5.5% | full set · 55 tasks | 2026-10-27 | 5.5% | 5.5% (k=1) | 0.0% / 7.3% | 123c1450dab5 | Run file |
| Gemini 3.1 Flash-LiteGoogle · small modelgemini-3.1-flash-lite · gemini | 3 of 15 solved · 20.0% | sample · 15 tasks | 2026-10-27 | 20.0% | 20.0% (k=1) | 0.0% / 6.7% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonclaude-sonnet-5-5 · anthropic | 32 of 55 solved · 58.2% | full set · 55 tasks | 2026-10-27 | 49.1% | 58.2% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelanother run of this modelclaude-haiku-4-5 · anthropic | 3 of 15 solved · 20.0% | sample · 15 tasks | 2026-10-27 | 20.0% | 20.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelanother run of this modelgpt-4o-mini · openai | 2 of 15 solved · 13.3% | sample · 15 tasks | 2026-10-27 | 13.3% | 13.3% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIanother run of this modelsarvam-105b · sarvam | 0 of 8 solved · 0.0% | only 8 tasks so far | 2026-10-27 | 0.0% | 0.0% (k=1) | 0.0% / 12.5% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonanother run of this modelclaude-sonnet-5-5 · anthropic | 11 of 15 solved · 73.3% | sample · 15 tasks | 2026-10-27 | 60.0% | 73.3% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
UPI Desk
Handle a UPI payment problem: check who is being paid, stay inside the limits, sort out a failed payment, and spot a scam request before approving it.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| Sarvam 105BSarvam AIsarvam-105b · sarvam | 26 of 50 solved · 52.0% | full set · 50 tasks | 2026-10-27 | 52.0% | 52.0% (k=1) | 0.0% / 6.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelclaude-haiku-4-5 · anthropic | 21 of 50 solved · 42.0% | full set · 50 tasks | 2026-10-27 | 42.0% | 42.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelgpt-4o-mini · openai | 9 of 50 solved · 18.0% | full set · 50 tasks | 2026-10-27 | 18.0% | 18.0% (k=1) | 0.0% / 16.0% | 123c1450dab5 | Run file |
| Gemini 3.1 Flash-LiteGoogle · small modelgemini-3.1-flash-lite · gemini | 9 of 15 solved · 60.0% | sample · 15 tasks | 2026-10-27 | 60.0% | 60.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonclaude-sonnet-5-5 · anthropic | 34 of 50 solved · 68.0% | full set · 50 tasks | 2026-10-27 | 68.0% | 68.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelanother run of this modelclaude-haiku-4-5 · anthropic | 6 of 15 solved · 40.0% | sample · 15 tasks | 2026-10-27 | 40.0% | 40.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIanother run of this modelsarvam-105b · sarvam | 3 of 8 solved · 37.5% | only 8 tasks so far | 2026-10-27 | 37.5% | 37.5% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelanother run of this modelgpt-4o-mini · openai | 2 of 15 solved · 13.3% | sample · 15 tasks | 2026-10-27 | 13.3% | 13.3% (k=1) | 0.0% / 6.7% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonanother run of this modelclaude-sonnet-5-5 · anthropic | 12 of 15 solved · 80.0% | sample · 15 tasks | 2026-10-27 | 80.0% | 73.3% (k=3) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonanother run of this modelclaude-sonnet-5-5 · anthropic | 11 of 15 solved · 73.3% | sample · 15 tasks | 2026-10-27 | 73.3% | 73.3% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
GST Desk
Work out the GST on an invoice, the right kind of tax when two states are involved, the credit you can claim, and the late fee, to the rupee.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5Anthropic · small modelclaude-haiku-4-5 · anthropic | 41 of 50 solved · 82.0% | full set · 50 tasks | 2026-10-27 | 82.0% | 82.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIsarvam-105b · sarvam | 39 of 50 solved · 78.0% | full set · 50 tasks | 2026-10-27 | 78.0% | 78.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelgpt-4o-mini · openai | 25 of 50 solved · 50.0% | full set · 50 tasks | 2026-10-27 | 50.0% | 50.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonclaude-sonnet-5-5 · anthropic | 50 of 50 solved · 100.0% | full set · 50 tasks | 2026-10-27 | 100.0% | 100.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
Form Desk
Fill in an ID or address form from messy text: PAN, IFSC, phone number, PIN code and amounts like "2.5 lakh". A wrong number must be called wrong, not fixed.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5Anthropic · small modelclaude-haiku-4-5 · anthropic | 41 of 50 solved · 82.0% | full set · 50 tasks | 2026-10-27 | 82.0% | 82.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIsarvam-105b · sarvam | 41 of 50 solved · 82.0% | full set · 50 tasks | 2026-10-27 | 82.0% | 82.0% (k=1) | 0.0% / 2.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelgpt-4o-mini · openai | 27 of 50 solved · 54.0% | full set · 50 tasks | 2026-10-27 | 54.0% | 54.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| gpt-oss 20BOpenAI · open weights, run on Groqopenai/gpt-oss-20b · groq | 12 of 15 solved · 80.0% | sample · 15 tasks | 2026-10-27 | 80.0% | 80.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Gemini 3.1 Flash-LiteGoogle · small modelgemini-3.1-flash-lite · gemini | 9 of 15 solved · 60.0% | sample · 15 tasks | 2026-10-27 | 60.0% | 60.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonclaude-sonnet-5-5 · anthropic | 50 of 50 solved · 100.0% | full set · 50 tasks | 2026-10-27 | 100.0% | 100.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIanother run of this modelsarvam-105b · sarvam | 7 of 8 solved · 87.5% | only 8 tasks so far | 2026-10-27 | 87.5% | 87.5% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelanother run of this modelclaude-haiku-4-5 · anthropic | 11 of 15 solved · 73.3% | sample · 15 tasks | 2026-10-27 | 73.3% | 73.3% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelanother run of this modelgpt-4o-mini · openai | 8 of 15 solved · 53.3% | sample · 15 tasks | 2026-10-27 | 53.3% | 53.3% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
Form: multi-document
Harder form checks: compare an applicant's documents, find conflicts inside a form, fix scanning mistakes only when the fix is certain, and check a batch of records.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| Sarvam 105BSarvam AIsarvam-105b · sarvam | 40 of 40 solved · 100.0% | full set · 40 tasks | 2026-10-27 | 100.0% | 100.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelclaude-haiku-4-5 · anthropic | 39 of 40 solved · 97.5% | full set · 40 tasks | 2026-10-27 | 97.5% | 97.5% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelgpt-4o-mini · openai | 13 of 40 solved · 32.5% | full set · 40 tasks | 2026-10-27 | 32.5% | 32.5% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonclaude-sonnet-5-5 · anthropic | 40 of 40 solved · 100.0% | full set · 40 tasks | 2026-10-27 | 100.0% | 100.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelanother run of this modelclaude-haiku-4-5 · anthropic | 15 of 15 solved · 100.0% | sample · 15 tasks | 2026-10-27 | 100.0% | 100.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIanother run of this modelsarvam-105b · sarvam | 8 of 8 solved · 100.0% | only 8 tasks so far | 2026-10-27 | 100.0% | 100.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelanother run of this modelgpt-4o-mini · openai | 5 of 15 solved · 33.3% | sample · 15 tasks | 2026-10-27 | 33.3% | 33.3% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
Bhasha Support Desk
Answer a customer in Hindi or Hinglish: read the order and the shop's rules, do the right thing (refund, cancel, decline or pass to a person), and reply in the customer's language.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| Sarvam 105BSarvam AIsarvam-105b · sarvam | 33 of 76 solved · 43.4% | full set · 76 tasks | 2026-10-27 | 43.4% | 43.4% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelclaude-haiku-4-5 · anthropic | 18 of 76 solved · 23.7% | full set · 76 tasks | 2026-10-27 | 21.1% | 23.7% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelgpt-4o-mini · openai | 7 of 76 solved · 9.2% | full set · 76 tasks | 2026-10-27 | 9.2% | 9.2% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Gemini 3.1 Flash-LiteGoogle · small modelgemini-3.1-flash-lite · gemini | 5 of 15 solved · 33.3% | sample · 15 tasks | 2026-10-27 | 26.7% | 33.3% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonclaude-sonnet-5-5 · anthropic | 50 of 76 solved · 65.8% | full set · 76 tasks | 2026-10-27 | 65.8% | 65.8% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIanother run of this modelsarvam-105b · sarvam · experimental split | 15 of 26 solved · 57.7% | full set · 26 tasks | 2026-10-27 | 57.7% | 57.7% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIanother run of this modelsarvam-105b · sarvam · core split | 18 of 50 solved · 36.0% | full set · 50 tasks | 2026-10-27 | 36.0% | 36.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelanother run of this modelclaude-haiku-4-5 · anthropic · experimental split | 8 of 26 solved · 30.8% | full set · 26 tasks | 2026-10-27 | 23.1% | 30.8% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelanother run of this modelclaude-haiku-4-5 · anthropic · core split | 10 of 50 solved · 20.0% | full set · 50 tasks | 2026-10-27 | 20.0% | 20.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelanother run of this modelgpt-4o-mini · openai · core split | 6 of 50 solved · 12.0% | full set · 50 tasks | 2026-10-27 | 12.0% | 12.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelanother run of this modelgpt-4o-mini · openai · experimental split | 1 of 26 solved · 3.9% | full set · 26 tasks | 2026-10-27 | 3.9% | 3.9% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIanother run of this modelsarvam-105b · sarvam | 3 of 8 solved · 37.5% | only 8 tasks so far | 2026-10-27 | 37.5% | 37.5% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Haiku 4.5Anthropic · small modelanother run of this modelclaude-haiku-4-5 · anthropic | 3 of 15 solved · 20.0% | sample · 15 tasks | 2026-10-27 | 20.0% | 20.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelanother run of this modelgpt-4o-mini · openai | 1 of 15 solved · 6.7% | sample · 15 tasks | 2026-10-27 | 6.7% | 6.7% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonanother run of this modelclaude-sonnet-5-5 · anthropic · core split | 33 of 50 solved · 66.0% | full set · 50 tasks | 2026-10-27 | 66.0% | 66.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonanother run of this modelclaude-sonnet-5-5 · anthropic · experimental split | 17 of 26 solved · 65.4% | full set · 26 tasks | 2026-10-27 | 65.4% | 65.4% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Claude Sonnet 5.5Anthropicfor comparisonanother run of this modelclaude-sonnet-5-5 · anthropic | 1 of 5 solved · 20.0% | only 5 tasks so far | 2026-10-27 | 20.0% | 20.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
Want a private evaluation on held-out tasks? →
The easy split is a smaller set of simpler tasks with one rule or one decision each, so smaller models have something to show. Its scores are not comparable with the standard tables above.
Trip Planning
Plan a trip across Indian cities by train, taxi and hotel, within a budget, on the right dates, with seats that are really confirmed. Some trips cannot work, and the agent has to say why.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5Anthropic · small modelclaude-haiku-4-5 · anthropic · easy split | 12 of 18 solved · 66.7% | sample · 18 tasks | 2026-10-27 | 66.7% | 66.7% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelgpt-4o-mini · openai · easy split | 1 of 18 solved · 5.6% | sample · 18 tasks | 2026-10-27 | 0.0% | 5.6% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
UPI Desk
Handle a UPI payment problem: check who is being paid, stay inside the limits, sort out a failed payment, and spot a scam request before approving it.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5Anthropic · small modelclaude-haiku-4-5 · anthropic · easy split | 11 of 18 solved · 61.1% | sample · 18 tasks | 2026-10-27 | 55.6% | 61.1% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelgpt-4o-mini · openai · easy split | 6 of 18 solved · 33.3% | sample · 18 tasks | 2026-10-27 | 33.3% | 33.3% (k=1) | 0.0% / 27.8% | 123c1450dab5 | Run file |
GST Desk
Work out the GST on an invoice, the right kind of tax when two states are involved, the credit you can claim, and the late fee, to the rupee.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5Anthropic · small modelclaude-haiku-4-5 · anthropic · easy split | 18 of 18 solved · 100.0% | sample · 18 tasks | 2026-10-27 | 100.0% | 100.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| Sarvam 105BSarvam AIsarvam-105b · sarvam · easy split | 18 of 18 solved · 100.0% | sample · 18 tasks | 2026-10-27 | 100.0% | 100.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| gpt-oss 20BOpenAI · open weights, run on Groqopenai/gpt-oss-20b · groq · easy split | 18 of 18 solved · 100.0% | sample · 18 tasks | 2026-10-27 | 94.4% | 100.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| GPT-4o miniOpenAI · small modelgpt-4o-mini · openai · easy split | 17 of 18 solved · 94.4% | sample · 18 tasks | 2026-10-27 | 94.4% | 94.4% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| gpt-oss 120BOpenAI · open weights, run on Groqopenai/gpt-oss-120b · groq · easy split | 17 of 18 solved · 94.4% | sample · 18 tasks | 2026-10-27 | 94.4% | 94.4% (k=1) | 0.0% / 5.6% | 123c1450dab5 | Run file |
Bhasha Support Desk
Answer a customer in Hindi or Hinglish: read the order and the shop's rules, do the right thing (refund, cancel, decline or pass to a person), and reply in the customer's language.
| Model | Solved | Tasks tried | Updated | Strict pass@1 | pass^k | Errors / bad format | Grader | Run |
|---|---|---|---|---|---|---|---|---|
| gpt-oss 120BOpenAI · open weights, run on Groqopenai/gpt-oss-120b · groq · easy split | 0 of 18 solved · 0.0% | sample · 18 tasks | 2026-10-27 | 0.0% | 0.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
| gpt-oss 20BOpenAI · open weights, run on Groqopenai/gpt-oss-20b · groq · easy split | 0 of 18 solved · 0.0% | sample · 18 tasks | 2026-10-27 | 0.0% | 0.0% (k=1) | 0.0% / 0.0% | 123c1450dab5 | Run file |
Questions about the results
Why are some rows “samples”?
Some runs only did part of the tasks, for example 15 of them, because the full set costs time and money. We mark those rows “sample”. With 15 tasks, one task changes the score by almost 7 points, so a sample can move a lot. Rows for the full set say so, and we add more rows as runs finish.
Why is there no overall average?
The worlds are very different. They have different numbers of tasks and different kinds of difficulty, and an easy set and a standard set are never mixed. One average would hide where an agent is good and where it needs help. So you see each world on its own.
Who checks the answers?
A program, not an AI. The rules are written down in the code, and the program works out the right answer again from the task's own data and compares it with the agent's answer. It ignores whatever the agent says about its own score. The code is public, so anyone can read the rules. If you think one is wrong, open an issue.
How do I submit my model?
Install the repo, run the public tasks with your model, and send us the saved run files. We replay them on our computer and list the row only if we get the same answer. Rows from other people are marked “community” and their model name is self-reported. The steps are in SUBMITTING.md.
What does “for comparison” mean?
We show one well-known commercial model last, as a reference point, so you can see how smaller and open models do next to it. It is not a prize, a ranking or a verdict on any company. We usually run it on fewer tasks, so check the “sample” mark.
Why does a model have more than one row?
Some models were run more than once, for example on a sample and on the full set. The table shows the run with the most tasks; the others appear under “Show technical details”.
Submit your model
Run Navital Gym on your own model, then send us the run file. Any model behind an OpenAI-style chat API with tool calling works.
- Install the repository and run an environment:
pip install -e ., thennavital-gym run --env upi --agent "openai:MODEL@BASE_URL" --api-key-env NAME --out runs/upi-MODEL.json. Keep the key in an environment variable, never on the command line. - Open a GitHub issue (new issue) with the run JSON and its transcripts file attached, and the model name, version and date.
- We re-grade the run with
scripts/regrade.pybefore listing it, so every row is graded by one grader commit. The listed score can differ from the one your run printed.
State whether it is the full set or a sample. Samples are listed as indicative.
Submit your model: the full steps, and what we check before a row is listed. Outside submissions are marked community.