Skip to content

Research

What a 7B model learns from synthetic agent environments: a LoRA adapter on Qwen2.5-7B-Instruct for Indian UPI tasks, and where the gain stops

1. Summary

We fine-tuned a LoRA adapter on Qwen2.5-7B-Instruct using 2,530 generated training rows, then tested it on the public Navital Gym UPI tasks, which include the kinds it was trained on. On the 43 current public UPI tasks the untrained base solved 6 of 43 (14%, interval 7 to 27%) and the adapter, Navital Bay, solved 38 of 43 (88%, interval 76 to 95%). The two ran on the same tasks; a paired exact McNemar test gives p = 4.7e-10 (32 tasks solved only by Bay, 0 only by the base). The gain is on kinds that have training data: mandates, for which the training data has no template, stayed at 0 of 5. The gain did not carry over to GST, a different environment: 14 of 50 for the base and 11 of 50 for Bay (p = 0.51), so we found no transfer. Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 ran on the same tasks; we did not fine-tune them on these tasks; their results are in the tables beside Bay's and are not matched on settings (section 5). One training run exists, so we make no claim about which part of the data produced which result.

2. Environments and tasks

The tasks are the public Navital Gym tasks at tag v1.1.0 (commit 2e8c903): 50 UPI tasks and 50 GST tasks, each graded by written rules in code, with no AI judge. The 50 UPI tasks split into 43 current tasks and 7 collect-request tasks. Collect requests model a dated merchant flow and are scored separately. The 43 current tasks fall into six kinds: pay_verify (10), limits (8), failed_debited (10), wrong_recipient (5), upi_lite (5), mandate (5). Public tasks may have been seen by any of the models tested.

3. Training data

The adapter is a LoRA on Qwen2.5-7B-Instruct, adapter id d3736aec897c (run 5). The SFT file has 2,530 rows (sha256 e0dbc1e6059c1063…, wc -l 2530).

Rows by kind and template:

KindRowsTemplates
pay_verify946lookalike-name 347, match-then-limit 599
failed_debited605p2m-window 341, p2p-window 264
upi_lite679load 242, pay 437
collect_request5124 + 27
limits168 + 8
wrong_recipient175 + 12
other (non-UPI)216no kind or template

Replay rows. 506 of the 2,530 rows are replay from earlier runs. These are: all 216 non-UPI rows, all 51 collect rows, all 16 limits rows, all 17 wrong_recipient rows, and 206 of the pay_verify, failed_debited and upi_lite rows (82, 45 and 79). So the limits and wrong_recipient rows were not generated for this run, and nothing in this paper says 16 fresh rows taught limits. There is no mandate template: zero training rows have kind or template mandate. GST rows (66 of the 216) are all replay, and run 5 used no GST generator template.

Outcome labels across the 2,530 rows: allowed 1,144; forbidden 1,137; over_both 3; over_daily 2; over_verified 3; paid 5; paid_boundary 3; plain 5; scam_return 12; no label 216 (these sum to 2,530).

4. Contamination checks

We checked the 50 core and 18 easy public UPI tasks against the training prompts. No public task shares an id, an exact prompt, a world, a world with its gold answer, or an 8-gram overlap (maximum 8-gram Jaccard 0.000) with the training data. A control injected 2 public prompts into the training prompt set, and the check found 2 of 2. Limit of the check: the world, world+gold and 8-gram checks cover the 2,024 rows that have task files. The 506 replay rows have no task file there, so only the id, exact-prompt and first-60-character checks cover them. Public tasks of every kind except mandate share template names with training, with different values, so the adapter was trained on the kinds it is tested on. Overlap with Qwen2.5's pretraining data was not checked.

5. Evaluation set-up, and what differs between arms

Intervals are Wilson 95% intervals, and each task was run once (one attempt per task, k=1). Grader package tree 1543b5103be5 (navital_gym/ at v1.1.0) is recorded in the Claude run files. The open-model run files record no grader tree, so for the base and Bay the tree rests on the results board (navitallabs.com/labs/navital-gym/results, as of 15 Nov 2026) and the run note of 9 Nov only.

  • Base and Bay used the same serving stack per the run note only: vLLM 0.31.0 with a tool-call shim, A10G, bf16, temperature 0, max_tokens 1500, context 32768. The run files record provider ollama and no max_tokens; we did not check server logs.
  • The Claude rows ran at the API's default temperature. The API rejected a temperature setting for these models, and the harness then drops it (agents.py:420-430); the run files still record 0.0.
  • The output limit differs by arm: the Claude rows ran with max_tokens 4096 per turn, the open models with 1500 per turn per the run note. The context limit hurt the base on GST: 2 of its 50 GST runs ended in a context-limit error (IN-GST-009, IN-GST-012).
  • The Claude rows are Haiku 5.5, Sonnet 5.5 and Opus 5.5 from the 9 Nov API runs. They are not the Haiku 4.5 files of the v1.1.0 release; the two are not mixed here.
  • Bay was trained on the kinds it is tested on; the Claude models were not fine-tuned by us on these tasks.

6. Results

6.1 UPI, 43 current tasks

ModelSolvedInterval
Qwen2.5-7B-Instruct (base)6 of 437 to 27% (14%)
Navital Bay (adapter)38 of 4376 to 95% (88%)
Claude Haiku 5.540 of 4381 to 98% (93%)
Claude Sonnet 5.542 of 4388 to 100% (98%)
Claude Opus 5.543 of 4392 to 100% (100%)

Base against Bay, paired on the same 43 tasks: McNemar exact p = 4.7e-10. Bay against Haiku 5.5, paired: 1 task Bay only, 3 tasks Haiku only, p = 0.63, so these data do not separate the two.

6.2 UPI by kind (solved of n, Wilson 95% interval)

KindBaseBayHaiku 5.5Sonnet 5.5Opus 5.5
pay_verify6 of 10 [31-83]10 of 10 [72-100]10 of 10 [72-100]10 of 10 [72-100]10 of 10 [72-100]
limits0 of 8 [0-32]8 of 8 [68-100]7 of 8 [53-98]8 of 8 [68-100]8 of 8 [68-100]
failed_debited0 of 10 [0-28]10 of 10 [72-100]10 of 10 [72-100]10 of 10 [72-100]10 of 10 [72-100]
wrong_recipient0 of 5 [0-43]5 of 5 [57-100]5 of 5 [57-100]5 of 5 [57-100]5 of 5 [57-100]
upi_lite0 of 5 [0-43]5 of 5 [57-100]5 of 5 [57-100]5 of 5 [57-100]5 of 5 [57-100]
mandate0 of 5 [0-43]0 of 5 [0-43]3 of 5 [23-88]4 of 5 [38-96]5 of 5 [57-100]

Cells hold 5 to 10 tasks, so every interval is wide. We draw no ranking from this table.

6.3 Collect requests (7 tasks)

Base 1 of 7 [3-51]; Bay 7 of 7 [65-100]; Haiku 5.5 2 of 7 [8-64]; Sonnet 5.5 6 of 7 [49-97]; Opus 5.5 7 of 7 [65-100]. Bay's 51 collect training rows are all replay.

6.4 Forbidden-step attempts

On the 23 current tasks that have a forbidden step: base 12 of 23 [33-71], Bay 4 of 23 [7-37], Haiku 5.5 2 of 23 [2-27], Sonnet 5.5 0 of 23 [0-14], Opus 5.5 0 of 23 [0-14]. These are counts of attempts at a forbidden step (R-FORBIDDEN false).

6.5 GST: no transfer

ModelSolved of 50Interval
Base1417 to 42%
Bay1113 to 35%
Claude Haiku 5.5, Sonnet 5.5, Opus 5.550 each93 to 100%

Base against Bay on the same 50 tasks: 6 tasks base only, 3 Bay only, McNemar exact p = 0.51. The data show no transfer of the UPI training to GST. Bay returned 8 of 50 answers in the wrong shape: valid JSON, but answer was the string "IGST" where the grader requires an object (tasks IN-GST-016 to 023). GST was not a training target: its 66 rows are replay and no GST template was used. The Claude models solved all 50.

7. Where the gains stop

  1. Mandates: 0 of 5 for Bay. The training data has no mandate template, and the base also scored 0 of 5.
  2. GST: no transfer (section 6.5).
  3. Limits (8 of 8) and wrong_recipient (5 of 5) were solved in full from 16 and 17 training rows, but those rows are all replay from earlier runs, the kinds share templates with the public tasks, and the cells are small. We do not claim that 16 rows teach limits.
  4. Failure transcripts (Bay, public tasks, verbatim from results-t133/raw/*.transcripts.json, k=1):
    • Mandates (IN-UPI-041 to 045), all five unsolved. In 041 to 044 Bay's final message is {"status": "plan", "answer": {"action": "needs_user_pin_to_register"}}. The action is the gold one, but the answer omits cap_applied and each_debit_needs_approval, and the run called create_mandate, which the task forbids (rules R-CAP, R-DEBIT-APPROVAL and R-FORBIDDEN fail; gold, e.g. 041: cap_applied 15000, each_debit_needs_approval false). In 045 the final message is Check: (1) amount 120000 > high-cap maximum 100000 -> over the highest available limit -> do not set up followed by {"status": "plan", "answer": {"action": "not_allowed", "reason": "over_txn_limit"}}; the gold action is needs_user_pin_to_register (cap_applied 100000), so R-ACTION fails, and R-CAP and R-DEBIT-APPROVAL fail as well (the answer has no cap_applied or each_debit_needs_approval). These four tasks, 041 to 044, are exactly Bay's four forbidden-step attempts in section 6.4.
    • GST, 8 tasks (IN-GST-016 to 023). The final message in each is {"status": "plan", "answer": "IGST"}. R-FMT fails with "answer must be an object", score 0.0, no repair taken. Of the 31 other unsolved GST tasks (valid shape, wrong values), the rules that fail most are R-TAXABLE 14, R-TOTAL 14, R-IGST 9 and R-ROUNDOFF 9 (a task can fail several).
    • Counts from results-t133/raw/upi-bay-tuned.vllm.json, gst-bay-tuned.vllm.json and their transcripts; the mandate and GST task lists are read at v1.1.0.

8. Limitations

One attempt per task. Five to ten tasks per kind. One training run (run 5), so no ablation and no claim about which data produced which gain. The adapter sits on a named base model; it is not a standalone model. The training kinds are the tested kinds. Public tasks may have been seen by all models, and pretraining overlap was not checked. Open-model and Claude arms differ in output limit (1,500 vs 4,096 tokens per turn) and temperature setting (0 vs API default). The open-model stack comes from the run note only. We use no held-out results: the private held-out set stays unpublished by decision. Other small open models (Mistral, Qwen3, Fathom) are not reported, because their results are not verified.

9. Disclosure

Navital Labs is a member of AWS Activate, Claude for Startups and the Sarvam Startup Program. Credits received: Anthropic $1,000 and 5 Claude Team accounts for one year; AWS $1,000; Sarvam 25,000 INR. Anthropic credits paid for the Claude model runs, and AWS credits paid for the open-model runs on AWS (A10G). None of these programmes has any say in the tests, the grading or the results.

10. Reproduction

Public: the tasks, grader and tag v1.1.0 (commit 2e8c903; grader package tree 1543b5103be5, navital_gym/ at that tag) of the navital-gym repository on GitHub, and the results board at navitallabs.com/labs/navital-gym/results (as of 15 Nov 2026). Not released: the SFT training file (sha256 e0dbc1e6059c10639eb7c28a85f61b493f4fa71c061f20b97a01d1f6c4731270, 2,530 rows) and the per-run result files for this paper.