Skip to content

Can this AI handle Indian money?

Open and public models, run on the same UPI and GST tasks, checked by a program and not by another AI. We show where each model breaks so builders can pick, and so model makers can see what to fix.

Last updated 2026-11-10 · grader 2e8c903

01 · RESULTS

Results by model and world

Known issue, 16 November 2026: the official v1.1 scores are unchanged. Our UPI instructions listed answer fields under group labels without saying the answer must be one flat object, and some models nested their fields under those labels, which the grader did not read. A sensitivity regrade that also reads fields nested under group labels gives Claude Haiku 4.5 UPI 32 of 50 (published 26) and Sarvam 105B UPI 30 of 50 (published 29). Every other model and world: no change. The 15 November results board is not affected. CHANGELOG, known issues. See the leaderboard for the two rows.

UPI payments

Tried a forbidden step: the model took an action the world's rules do not allow, for example sending money again to recover a payment made to the wrong UPI ID.

UPI payments: tasks solved and tasks where a forbidden step was tried, by model
ModelSolvedTried a forbidden step
Claude Opus 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c90343 of 430 of 23
Claude Sonnet 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c90342 of 430 of 23
Claude Haiku 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c90340 of 432 of 23
Navital Bay (LoRA on Qwen2.5-7B-Instruct) (Navital Labs)bf16, our GPU, oursTrained on these kinds of task.Run 2026-11-09Grader 2e8c90338 of 434 of 23
Qwen2.5-7B-Instruct (Alibaba Qwen)bf16, our GPURun 2026-11-09Grader 2e8c9036 of 4312 of 23

Notes: Qwen2.5-7B-Instruct: 2 answers were marked invalid, counted unsolved. Collect requests are scored separately (7 tasks): most merchant collect payments in UPI were phased out from 28 February 2026, so these tasks show a dated flow; they still test whether a model approves only legitimate requests. Claude Opus 5.5 7 of 7, Claude Sonnet 5.5 6 of 7, Claude Haiku 5.5 2 of 7, Qwen2.5-7B-Instruct 1 of 7, Navital Bay 7 of 7. As of 2026-11-10.

One attempt per task. Public tasks: models may have seen them.

GST invoices

GST invoices: tasks solved and tasks where a forbidden step was tried, by model
ModelSolvedTried a forbidden step
Claude Haiku 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c90350 of 50not applicable
Claude Opus 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c90350 of 50not applicable
Claude Sonnet 5.5 (Anthropic)provider default, Anthropic APIRun 2026-11-09Grader 2e8c90350 of 50not applicable
Qwen2.5-7B-Instruct (Alibaba Qwen)bf16, our GPURun 2026-11-09Grader 2e8c90314 of 50not applicable
Navital Bay (LoRA on Qwen2.5-7B-Instruct) (Navital Labs)bf16, our GPU, oursTrained on these kinds of task.Run 2026-11-09Grader 2e8c90311 of 50not applicable

Notes: Qwen2.5-7B-Instruct: 2 runs hit the 32,768-token context limit and count as unsolved; Navital Bay: 8 of 50 answers had the wrong shape (a string where an object is required) and count as unsolved. As of 2026-11-10.

One attempt per task. Public tasks: models may have seen them.

02 · HOW TO READ THIS

How to read this

  • Solved is tasks done correctly in one attempt per task (one automatic retry if the final reply had no readable JSON object; an answer in the wrong shape is not retried), out of the tasks run on the public split.
  • Haiku. This page lists Claude Haiku 5.5 (results dated 9 November). The leaderboard and the home-page scorecard show Claude Haiku 4.5 from the v1.1.0 release (31 October). They are different models, so their scores are not comparable.
  • Tried a forbidden step counts tasks where the model took an action the rules do not allow, out of the tasks where that was possible. It sits next to solved because a model can do both in one task. A world whose public tasks have no forbidden step says "not applicable".
  • Each row names the model, its maker, how it was run and where. Hosted models and models on our own GPU are run differently, so read the label before you compare two rows.
  • Open models ran at temperature 0. The Claude 5.5 models do not accept a temperature setting, so they ran at the Anthropic API's default.
  • Output limits follow each provider: up to 4,096 tokens per reply for the Claude models, 1,500 for our 7B models run on our own GPU.
  • The order is by tasks solved. It is not a prize and it is not a verdict on a model.
  • Method and grader are open: read the report, or run it yourself. The older full tables are on the leaderboard.