Skip to content
Navital Labs

Navital Gym

How Navital Gym grades AI agents: written rules

Every Navital Gym task is graded by named, deterministic rules that recompute the answer from the sandbox. No AI judge. Here is how it works.

Published 27 October 2026. As of 27 October 2026.

01 · HOW NAVITAL GYM GRADES: WRITTEN RULES, NOT AN AI JUDGE

Navital Gym is a set of open practice worlds where AI agents do everyday Indian tasks: planning a train trip, handling a UPI dispute, working out GST on an invoice, filling an address form, answering a customer in Hindi or Hinglish. A score is only useful if you can see why it was given. This page explains how every task is marked.

02 · ONE RULE: THE GRADER RECOMPUTES, IT DOES NOT TRUST

An agent finishes a task by submitting an answer: a trip plan, a decision, a filled form. The grader never takes the agent's word for anything. If the agent says "total cost is 8,400", the grader adds up the plan itself from the sandbox world. If it says "this task is solved", that claim is ignored. A malformed or garbage answer scores 0, and the grader never crashes on it.

03 · NAMED RULES, NOT A VIBE

Each task is checked against a list of named rules. A trip plan, for example, is checked for budget, dates, seat availability and opening hours, each as its own rule with its own id (such as R-BUD, R-DATE, R-AVAIL and R-OPEN). A task counts as solved only if every rule that applies to it holds. The report shows which rule failed, so a failure is something you can read and argue with, not a number from a black box.

Because the rules are code, the same answer always gets the same score. There is no judge model whose opinion changes between runs, and no sampling noise in the grading itself.

04 · SOME TASKS HAVE NO VALID ANSWER

Real requests are sometimes impossible: a train that is full, a budget that cannot work, a refund outside the window. Many Navital Gym tasks are impossible on purpose. The agent has to say so and name the real reason. Giving a wrong reason, or producing a plan anyway, is marked wrong. The run report shows feasible and infeasible tasks in separate tables, so you can see whether an agent fails by planning badly or by missing that a task is impossible.

05 · WHAT THE AGENT CAN AND CANNOT SEE

The agent sees only the task prompt, the tools and the policy text. The expected answer, whether the task is feasible, the rule that binds and the designers' notes stay in the grader. A test checks every task in every world to make sure none of those fields leaks into what the agent sees.

06 · TRYING TO CHEAT OUR OWN BENCHMARK

Before any model is run, we run degenerate agents that read nothing and try to game the format: depending on the world, one submits nothing, one refuses or rejects everything, one approves everything, and one always gives the most common answer. A world ships only if none of them solves more than a quarter of its tasks. We also run an oracle, a reference solver that works out each answer from the rules using the same tools the agent gets, and it must score full marks. These checks have caught real bugs in our own graders, and we will write about them separately.

07 · HONEST ABOUT WHAT THIS IS
  • The rules are a simplified, documented version of Indian practice, written as of October 2026, and the data is synthetic. This is a test for agents, not a statement about current law, railway, bank or tax rules.
  • Navital Gym is not affiliated with or endorsed by NPCI, RBI, GSTN, UIDAI, India Post, CBIC, IRCTC or Indian Railways, or any bank, and is not financial, tax or travel advice.
  • Tamil, Bengali and Marathi support lines are machine-assisted and not yet native-reviewed; they are graded on the outcome, not the language, and labelled experimental.
08 · RUN IT YOURSELF

The core runs on plain Python 3.10+ with no dependencies, and works with any model that speaks the OpenAI chat API with tool calling, including local models through Ollama. Optional adapters cover Inspect AI, verifiers and OpenEnv. Code is Apache-2.0 and task data is CC BY 4.0. The repository has been public since 27 October 2026; full results and a report follow on 10 November 2026.

09 · THE FIVE WORLDS

Want to hear when the full results and report are out? Email dakshbathla@navitallabs.com with the subject "Notify me: Navital Gym".

MORE

Navital GymLeaderboardWhat is an agent environment?

Navital Labs is not affiliated with NPCI, RBI, GSTN, CBIC, Indian Railways, IRCTC or any bank. Rules are simplified, as modelled in Navital Gym. Not financial, tax or travel advice.