Skip to content
Navital Labs

Navital Gym report

Published 30 October 2026 · Navital Gym v1.0.1 · grader ed78addcd557

Claude Opus 5.5 solved the most standard tasks: 264 of 321, on one run each. The weakest full run solved 84.

Navital Gym is a set of practice tasks for AI agents built around everyday Indian jobs: planning a trip, handling UPI payments, filing GST, filling forms and answering shop customers in Hindi and Hinglish. A written rule checks every answer. No AI is the judge.

See the leaderboardPrivate evaluation, $750
Claude Opus 5.5 · standard set · k=1321 tasks

Each dot is one task. Filled: solved. Hollow: failed.

Trip Planning32/55
UPI Desk39/50
GST Desk49/50
Form Desk90/90
Bhasha Support54/76
57of 321 tasks still fail
264/321 solvedsolvedfailed
01 · RESULTS AT A GLANCE
Tasks solved by model, full standard-set runs, one run per task
ModelSolvedPass rate
Claude Opus 5.5Anthropic264/32182.2%
Claude Sonnet 5.5Anthropic256/32179.8%
Sarvam 105BSarvam AI186/32157.9%
Claude Haiku 4.5Anthropic177/32155.1%
GPT-4o miniOpenAI84/32126.2%

One run per task (k=1). Only full runs on all 321 standard tasks are listed. Models we have only sampled are on the leaderboard, marked as samples.

02 · WHAT WE RAN

Navital Gym has five environments with six task sets. Form Desk has a second, harder set where the facts are spread across several documents.

  • Standard set: 321 tasks. Every score on this page uses it.
  • Models: Claude Opus 5.5, Claude Sonnet 5.5, Sarvam 105B, Claude Haiku 4.5, GPT-4o mini. Each got the same tasks, the same tools and the same instructions.
  • Runs: one per task, at temperature 0. A model may fix a badly formatted answer once; the “strict” column on the leaderboard shows scores without that retry.
  • Grading: every listed row was graded with grader ed78addcd557. Last export: 2026-10-30.
03 · HOW IT WAS GRADED

Each task is set in a small simulated world: train timetables, bank accounts, invoices, forms, a shop's order history. The agent uses tools to look things up and then submits an answer. A program then checks that answer against written rules, working the right answer out again from the same world. The same answer always gets the same score.

No AI model grades anything. The rules for each environment are written out in plain language in its README on GitHub, with the official sources they simplify.

One rule is a heuristic, not an exact check: in Bhasha Support, the reply must be in the customer's language. That is judged by the share of letters in the right script (at least 70%, or 95% Latin letters for Hinglish) and by words typical of the language. It applies to every support task, including the experimental ones.

You can check that a score came from the published code: the grader is identified by the git tree id of the navital_gym/ folder at the release tag. Run this in a clone of the repository; the result must start with ed78addcd557.

git rev-parse v1.0.1:navital_gym
04 · ENVIRONMENT BY ENVIRONMENT

  • Trip Planning

    55 tasks. Best full run: Claude Opus 5.5 and Claude Sonnet 5.5, 32/55.

    Most failed rules for Claude Opus 5.5: R-DATE (18 tasks), R-CONN (5 tasks), R-ROUTE (5 tasks).

    • R-DATE: Exactly days consecutive dates from start; starts at origin, ends at the end city on the last date
    • R-CONN: Real services, location continuity, buffers (rail 60 / road 30 / air 120 min), a hotel for each night
    • R-ROUTE: The trip does what was asked: visits every named city, must_visit sights, must_use legs, day-specific visits

  • UPI Desk

    50 tasks. Best full run: Claude Opus 5.5, 39/50.

    Most failed rules for Claude Opus 5.5: R-ACTION (11 tasks), R-EFFECTS (2 tasks).

    • R-EFFECTS: the account changed exactly as it should: required actions done once, nothing else

  • GST Desk

    50 tasks. Best full run: Claude Sonnet 5.5, 50/50.

    A deliberately careless scripted agent that makes two known GST slips still scores 80% on the public GST set. A high GST score alone does not show careful tax work.

  • Form Desk

    90 tasks. Best full run: Claude Opus 5.5 and Claude Sonnet 5.5, 90/90.

  • Bhasha Support

    76 tasks. Best full run: Claude Opus 5.5, 54/76. The 26 experimental tasks (Tamil, Bengali, Marathi) were written with machine help and have not been checked by native speakers; their scores are shown separately on the leaderboard.

    Most failed rules for Claude Opus 5.5: R-FORBIDDEN (10 tasks), R-REASON (10 tasks), R-DECISION (6 tasks).

    • R-FORBIDDEN: no action beyond what the policy requires (no wrong refund, no extra cancel, no double refund)
    • R-REASON: the stated reason code matches the policy fact that decided the case
    • R-DECISION: the stated decision matches what the policy requires (read from the order, not guessable from the prompt)

Counts are tasks that failed each rule, at one run per task. A task can fail several rules, so the counts can add up to more than the failed tasks.

Each environment's rules are written out in its README on GitHub.

05 · WHAT THESE RESULTS DO NOT SHOW
  • One run per task. Scores can change between runs, and we have not measured by how much for every model.
  • Samples are not full runs. A 15-task sample has a wide error bar: one task is 6.7 points.
  • The rules are simplified. Each environment simplifies the real rules (rail quotas, GST, UPI limits) so they can be checked exactly. A model that passes here can still be wrong about the real rule. Each README says what was simplified.
  • Public tasks can leak. These tasks are public, so future models may have seen them in training. That is why private evaluations use a separate held-out set that we never publish.
  • Some answers were cut off. When a model's answer hit its length limit, the task counts as unsolved. This affected 9 tasks for Sarvam 105B on Trip Planning.
  • Some environments are easy to game. A do-nothing agent scores 0 everywhere, but a scripted agent that makes two known GST slips still scores 80% on the public GST set, and a naive form filler scores 32% on Form Desk. The private held-out sets are built so that these shortcuts score under 25%.
06 · RUN IT YOURSELF

The code is Apache-2.0 and the task data is CC BY 4.0. You need Python 3.10 or newer; the core uses only the standard library. Install it with pip install navital-gym, or clone the repository to work on it.

Get it: GitHub · PyPI · Hugging Face

git clone https://github.com/Navitallabs/navital-gym
cd navital-gym
pip install -e .
navital-gym list
navital-gym run --env support --agent oracle

The last command runs the reference solver, which should score 1.00. SUBMITTING.md on GitHub explains how to add your own model's results.

07 · TEST YOUR OWN MODEL PRIVATELY

Private evaluation

Want these tests run on your model or agent, on tasks that have never been published? A private evaluation uses the held-out set and comes with a written report.

$750 per model or agent version, plus applicable taxes

Private evaluation, $750Pilots and custom task sets
08 · CITE

If you use Navital Gym in research, cite it with the CITATION.cff file in the repository.

The code, package and task data: GitHub · PyPI · Hugging Face

title: "Navital Gym: deterministic, auto-graded environments for AI agents doing Indian-reality tasks"
authors:
  - name: "Navital Labs"
version: "1.0.1"
license: Apache-2.0
repository-code: "https://github.com/Navitallabs/navital-gym"