Can this AI handle Indian money?
Open and public models, run on the same UPI and GST tasks, checked by a program and not by another AI. We show where each model breaks so builders can pick, and so model makers can see what to fix.
Last updated 2026-11-10 · grader 2e8c903
Results by model and world
Known issue, 16 November 2026: the official v1.1 scores are unchanged. Our UPI instructions listed answer fields under group labels without saying the answer must be one flat object, and some models nested their fields under those labels, which the grader did not read. A sensitivity regrade that also reads fields nested under group labels gives Claude Haiku 4.5 UPI 32 of 50 (published 26) and Sarvam 105B UPI 30 of 50 (published 29). Every other model and world: no change. The 15 November results board is not affected. CHANGELOG, known issues. See the leaderboard for the two rows.
UPI payments
Tried a forbidden step: the model took an action the world's rules do not allow, for example sending money again to recover a payment made to the wrong UPI ID.
| Model | Solved | Tried a forbidden step |
|---|---|---|
| Claude Opus 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c903 | 43 of 43 | 0 of 23 |
| Claude Sonnet 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c903 | 42 of 43 | 0 of 23 |
| Claude Haiku 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c903 | 40 of 43 | 2 of 23 |
| Navital Bay (LoRA on Qwen2.5-7B-Instruct) (Navital Labs)bf16, our GPU, oursTrained on these kinds of task.Run 2026-11-09Grader 2e8c903 | 38 of 43 | 4 of 23 |
| Qwen2.5-7B-Instruct (Alibaba Qwen)bf16, our GPURun 2026-11-09Grader 2e8c903 | 6 of 43 | 12 of 23 |
Notes: Qwen2.5-7B-Instruct: 2 answers were marked invalid, counted unsolved. Collect requests are scored separately (7 tasks): most merchant collect payments in UPI were phased out from 28 February 2026, so these tasks show a dated flow; they still test whether a model approves only legitimate requests. Claude Opus 5.5 7 of 7, Claude Sonnet 5.5 6 of 7, Claude Haiku 5.5 2 of 7, Qwen2.5-7B-Instruct 1 of 7, Navital Bay 7 of 7. As of 2026-11-10.
One attempt per task. Public tasks: models may have seen them.
GST invoices
| Model | Solved | Tried a forbidden step |
|---|---|---|
| Claude Haiku 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c903 | 50 of 50 | not applicable |
| Claude Opus 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c903 | 50 of 50 | not applicable |
| Claude Sonnet 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c903 | 50 of 50 | not applicable |
| Qwen2.5-7B-Instruct (Alibaba Qwen)bf16, our GPURun 2026-11-09Grader 2e8c903 | 14 of 50 | not applicable |
| Navital Bay (LoRA on Qwen2.5-7B-Instruct) (Navital Labs)bf16, our GPU, oursTrained on these kinds of task.Run 2026-11-09Grader 2e8c903 | 11 of 50 | not applicable |
Notes: Qwen2.5-7B-Instruct: 2 runs hit the 32,768-token context limit and count as unsolved; Navital Bay: 8 of 50 answers had the wrong shape (a string where an object is required) and count as unsolved. As of 2026-11-10.
One attempt per task. Public tasks: models may have seen them.
How to read this
- Solved is tasks done correctly in one attempt per task (one automatic retry if the final reply had no readable JSON object; an answer in the wrong shape is not retried), out of the tasks run on the public split.
- Haiku. This page lists Claude Haiku 5.5 (results dated 9 November). The leaderboard and the home-page scorecard show Claude Haiku 4.5 from the v1.1.0 release (31 October). They are different models, so their scores are not comparable.
- Tried a forbidden step counts tasks where the model took an action the rules do not allow, out of the tasks where that was possible. It sits next to solved because a model can do both in one task. A world whose public tasks have no forbidden step says "not applicable".
- Each row names the model, its maker, how it was run and where. Hosted models and models on our own GPU are run differently, so read the label before you compare two rows.
- Open models ran at temperature 0. The Claude 5.5 models do not accept a temperature setting, so they ran at the Anthropic API's default.
- Output limits follow each provider: up to 4,096 tokens per reply for the Claude models, 1,500 for our 7B models run on our own GPU.
- The order is by tasks solved. It is not a prize and it is not a verdict on a model.
- Method and grader are open: read the report, or run it yourself. The older full tables are on the leaderboard.
