Skip to content
Navital Labs

We test AI agents on Indian tasks. Every score comes from a written rule, never another AI.

UPI refunds, GST invoices, Hindi support chats: real Indian tasks, checked by open rules.

Solved
205
Failed
116
Scored by an AI judge
0
Test my agent, $750See how a task is graded

Claude Haiku 4.5. One try per task (k=1), on the 321 public tasks.

Trip Planning19/55

UPI Desk26/50

GST Desk42/50

Form Desk86/90

Bhasha Support32/76

Each dot is one task: filled is solved, hollow is failed. Press a hollow dot to see which rule it broke.
Fig. 1. One real run per model, every task, one try each, graded by the open rule-checker (tree 1543b5103be5). Form Desk counts its standard and multi-document sets together. Shown: the three Claude models, v1.1.0 files.

321 public tasks you can check yourself. Behind them: 181 private held-out tasks we have never published, and generators that make new tasks for five of the six task sets.

NAVITAL BAY

Navital Bay: our 7B model for Indian payment tasks. Adapter on Hugging Face. See Navital Bay →

Fig. 2. One of the failures above, graded.

The AI did the right thing, then promised something nobody can guarantee.

The dispute was the right action. The rules forbid the promise.

IN-UPI-029, UPI Desk, money sent to the wrong IDClaude Haiku 4.5, 4 turns, $0.026
Customer
I sent Rs 9000 to the wrong VPA in TXN7100 by mistake. Get it back.
Tool
transaction TXN7100; Rs 9000; status: success; payee: unknown0@upibank
Tool
policy failed_txn_tat_calendar_days: {p2p: 1, p2m: 5} (policy shortened)
Agent
called raise_dispute(TXN7100, wrong_recipient)action dispute_raisedanswer wrong_recipient: {reversal_guaranteed: true}reply "…since this is a p2p transaction, the bank has 1 calendar day (by 2026-11-17) to process the reversal and return the money to your account."
The rule-checker (the grader), 5 written rulessolved: false
The answer is in the right format. Passed.R-FMT
It took the right kind of action. Passed.R-ACTION
The account ended up exactly as it should. Passed.R-EFFECTS
It did not use anything it was told not to. Passed.R-FORBIDDEN
It must not promise the money will come back. It promised it would. Failed.R-NOPROMISE, does not promise the money will come back. The answer said reversal_guaranteed: true; it must be false
score 0.75 of 1.0, not solved: one failed rule fails the task.

Source: runs/upi-anthropic-haiku-4-5-full.transcripts.json, graded by tree 1543b5103be5. Transcript shortened: tool results trimmed, the agent's reply cut to its last sentence. The dispute itself was the right action; the promise is what failed. Task IDs are real. Held-out tasks are never shown.

HOW WE KNOW THE SCORE IS REAL
What we check, and what the results do not show

What we check

  • No AI model grades anything. Written rules recompute the answer from what the agent did.
  • Public and private tasks. You can run the public set yourself. Private evaluations use held-out tasks we have never published.
  • The graders are tested first. A reference solver should score 1.00, and scripted agents that read nothing and try to game the format are run before any model.

Check that a score came from the open rule-checker (the grader). The output starts with 1543b5103be5:

git rev-parse v1.1.0:navital_gym

What the results do not show

  • One run per task. Scores can change between runs, and we have not measured by how much for every model.
  • The rules are simplified. A model that passes here can still be wrong about the real rule.
  • Public tasks can leak. Future models may have seen them, which is why private evaluations use a separate set.
  • Some environments are easy to game. A careless scripted agent still scores 80% on the public GST set.

Read the full report

What you give, what you get, what it costs.

Test

$750 per model or agent version, plus applicable taxes (none for clients outside India)

A private written report: pass rates per world, the main failure types, and fixes to try. You pay your own model usage, capped and agreed up front.

What’s included

You give: API access or an endpoint. How long: within 5 working days.

Pilot

From $2,500

A focused slice of 20-30 tasks built around your use case, with grader, reference solutions and a short report.

What’s included

You give: a description of the workflow and what a correct result looks like. How long: about two weeks.

Train

From $7,500, everything included, plus applicable taxes (none for clients outside India).

A before-and-after report on held-out tasks kept out of training, with every transcript, and the trained weights or adapter.

What’s included

You give: one world and one open model you choose. Compute is agreed up front in the written scope. How long: Agreed in the written scope.

Email us about a Training Pilot

Every project starts with a scope we write down together, and payment terms are agreed in that scope.

Test my agent, $750Other ways to work with usSee our training results
01. WHAT WE ARE BUILDING

Five practice worlds where an AI assistant does a real Indian task and a program checks the result.

02. WHY INDIA

Many public evaluations we studied use Western workflows. Ours are Indian: rail quotas, UPI, GST, PIN codes and four languages. What is an agent environment?

03. WORK WITH US

Tell us the task and what a correct result looks like. We will say whether we can build it.

dakshbathla@navitallabs.com

Member of Anthropic's Claude for Startups program · AWS Activate credits

ALSO FROM NAVITAL LABS