Navital Bay
Our first model: a 7B model trained on Navital Gym for Indian payment tasks, built on Qwen2.5-7B.
281 of 300 new UPI tasks solved
281 / 300
Navital Bay
New UPI tasks, generated after the training data was fixed, and graded by rules, not by an AI judge.
The model was trained on these four kinds of UPI task. The 300 test tasks were generated after its training data was locked and frozen, and none of them appeared in training.
Previous training run: 223 of 300 on the same tasks and hardware.
Forbidden actions tried
3 of 171
3 of 171 tasks where a forbidden action was possible (under 2%)
Counts a try at an action the rules forbid, even when the simulated bank would have refused it. Lower is better.
The model was trained on these four kinds of UPI task. The 300 test tasks were generated after its training data was locked and frozen, and none of them appeared in training.
The same tasks, every model
Navital Bay7B, ours
UPI 281 of 300 (94%)
- Collect requests
- 72 / 75
- Failed but debited
- 73 / 75
- Pay and verify
- 75 / 75
- UPI Lite
- 61 / 75
Forbidden attempts: 3 of 171
Claude Sonnet 5.5API
UPI 242 of 300 (81%)
- Collect requests
- 17 / 75
- Failed but debited
- 75 / 75
- Pay and verify
- 75 / 75
- UPI Lite
- 75 / 75
Forbidden attempts: 0 of 171
Claude Haiku 4.5API†
UPI 162 of 300 (54%)
- Collect requests
- 22 / 75
- Failed but debited
- 0 / 75
- Pay and verify
- 65 / 75
- UPI Lite
- 75 / 75
Forbidden attempts: 69 of 171
Sarvam-105BAPI
UPI 169 of 300 (56%)
- Collect requests
- 17 / 75
- Failed but debited
- 23 / 75
- Pay and verify
- 61 / 75
- UPI Lite
- 68 / 75
Forbidden attempts: 67 of 171
Our previous training run7B
UPI 223 of 300 (74%)
- Collect requests
- 67 / 75
- Failed but debited
- 63 / 75
- Pay and verify
- 45 / 75
- UPI Lite
- 48 / 75
Forbidden attempts: 7 of 171
UPI is scored on 300 fresh held-out tasks.
Same tasks, rules and grader for every model; temperature 0, one attempt per task, same tools and turn limit. Output-length limits follow each provider: 4,096 tokens (Anthropic), 32,000 (Sarvam, a reasoning model), 1,500 (our 7B models, run locally).
† Claude Haiku 4.5. Its 0 on failed-but-debited: right values, wrong answer format.
Each model ran once, so differences of a few tasks may not be meaningful. Rows: ours first, then other providers, then our earlier runs; not ordered by score.
The Claude and Sarvam rows run through their providers' APIs; Navital Bay runs as local weights. That difference is not part of the score. Navital Bay is a 7B model; the others are larger or of undisclosed size.
API models: claude-sonnet-5-5 and claude-haiku-4-5 (Anthropic API) and sarvam-105b (Sarvam API), all run on 5 November 2026.
Where another model scores higher than Navital Bay: UPI Lite (Claude Sonnet 5.5 and Claude Haiku 4.5 75 of 75, Sarvam-105B 68); failed-but-debited (Sonnet 75 of 75); forbidden actions (Sonnet tried none). Sonnet ties it on pay-and-verify (75 of 75).
Tasks solved, by kind of task (out of 75 each)
Collect requests72 / 75
Previous training run: 67 / 75
Failed but debited73 / 75
Previous training run: 63 / 75
Pay and verify75 / 75
Previous training run: 45 / 75
UPI Lite61 / 75
Previous training run: 48 / 75
What this does not show
- This model is not built for GST.
- UPI mandates were not part of the training.
- Some failure types were targeted in training on purpose. These are unseen tasks, not a test of how well it generalises.
How it was measured
The recipe was fixed before the test set was generated. Each task gets one attempt (k=1: one run, no retries) at temperature 0, and counts as solved only when a program finds all of its rules satisfied. Navital Bay was evaluated on single-GPU NVIDIA L40S machines with the same software stack.
Use it
The LoRA adapter for Navital Bay (rank 16), to be loaded on top of Qwen/Qwen2.5-7B-Instruct. Merged weights are not published yet.
Licence: Apache 2.0, built on Qwen2.5-7B, free for commercial use.
You need a GPU with about 24 GB of memory for the bf16 base model. Install:
pip install "transformers>=5.20" peft accelerate torchTested on an NVIDIA L40S, bf16, in two clean installs with these versions:
- Python 3.12.10: transformers 5.22.1, peft 0.21.2, accelerate 1.15.0, torch 2.15.0
- Python 3.10.12: transformers 5.20.2, peft 0.21.2, accelerate 1.15.0, torch 2.14.1
Want one trained on your tasks?
A Training Pilot trains a model on your own workflows and tests it on held-out tasks kept out of training.