Known issues in Navital Gym v1.1
GST Desk, UPI Desk and Support: grading conventions the task text does not state. In a few places the grader applies a convention that the instructions and policy text given to the agent never spell out. A model that reads the text carefully can still pick a different, reasonable convention and be marked wrong. The official v1.1 scores are unchanged. We have not re-graded the published runs under any other convention, so we do not say how many published results would change; the counts below are how many public tasks have an answer that depends on the convention.
- GST, sign of
round_off(4 of 15 invoice tasks: IN-GST-002, -003, -005, -014). The grader expectsround_off = invoice_total − gross, which is negative when the total is rounded down. The text says only "round-off". An answer that gives the size of the rounding without the sign failsR-ROUNDOFF. - GST, counting days late (8 of 10 late-filing tasks). The grader counts the days after the due date and does not count the due date itself, so a return filed one day after is 1 day late. The core text does not say this (the easy split does). Counting the due date gives a different
days_late, late fee and interest. - GST, the interest basis (6 of those 8 late tasks; the other 2 are nil returns with no tax). The grader computes simple interest: net tax × 18/100 × days late / 365, rounded half-up to 2 decimals. The policy says only that the rate is 18% a year. Using 366 days, a monthly rate or a different rounding changes the interest. Note: section 170 of the CGST Act rounds tax, interest and penalty to the nearest rupee; the env rounds to the paisa. A chartered accountant has not yet reviewed these GST rules.
- UPI, which reason to give when several limits are broken (2 tasks: IN-UPI-012, IN-UPI-E08). The grader expects the first of: per-payment limit, daily amount limit, daily count limit, balance. The instructions list the reason codes with no order, so a model that names a different true reason fails
R-REASON.
Also unclear in the text, not known to change a published score: whether a limit is "up to and including" the amount (4 UPI tasks sit exactly on a boundary: IN-UPI-018, -042, -047, -049); whether "days late" for failed-but-debited money counts the days after tat_date (7 UPI tasks); that composition is open up to and including Rs 1.5 crore (IN-GST-049 is exactly on it); for Support, which reason to give when an item is both non-returnable and outside its window (5 tasks: IN-SUP-014, -017, -058, -E03, -E04), and that a change-of-mind return refunds the item price only. Support also grades that a reply mentions no rupee figure other than the refund, the item price, the shipping fee or one the customer wrote (R-AMOUNT); the instructions only say to state the refund. Its effect depends on each reply, so there is no task count.
What we are doing: the next version's task sets, which we are building now and have not released, state each of these conventions in the policy or instruction text the agent reads. v1.1 task files and graders are not changed, and v1.1 scores stay as published.
Known issues (fixed in v1.2)
UPI Desk: some answers were scored as wrong only because of their layout. The UPI instructions list the detail fields under labels, for example failed-but-debited money: "type", "tat_date", … and wrong recipient: "reversal_guaranteed". They never say that the answer must be flat. Some models copied a label as a key and put the fields inside it, for example {"action": …, "failed-but-debited money": {"type": …}}. The grader reads those fields only at the top level of the answer, so it marked them missing even when the values were right.
The official v1.1 scores are unchanged. To show the size of the effect, we re-graded every published v1.1 run with one level of nesting accepted, and changed nothing else:
- Claude Haiku 4.5, UPI: 26 → 32 of 50 (IN-UPI-022, 023, 025, 027, 028, 033). Total: 205 → 211 of 321.
- Sarvam 105B, UPI: 29 → 30 of 50 (IN-UPI-033). Total: 216 → 217 of 321.
- Every other model, and every other world: no change. The order of the models does not change.
The nested answers are in the public transcripts in the navital-gym repository on GitHub, at tag v1.1.0 (runs/upi-*-full.transcripts.json). For example, in Claude Haiku 4.5's transcript, IN-UPI-033 nests "reversal_guaranteed": false (right, but unread), while IN-UPI-029 nests "reversal_guaranteed": true (a real error, still failed).
Fix in v1.2: the UPI instructions will state the answer schema as one flat object. The grader stays strict, and every model is re-run on v1.2. v1.1 scores stay as published.
Known issues (to be fixed in v1.1.1)
- R-AMOUNT (support). Some ways of stating a refund are not checked yet: ranges joined by words ("₹1899 से लेकर 2500 तक", "between ₹1899 and 2500") check only the end next to the rupee sign; the spelling "rupaiya" is not recognised; a minus glued to the word before it ("रिफंड-₹1899") is read as a hyphen, and the minus check knows only the ₹, Rs and INR markers. For the experimental Tamil, Bengali and Marathi replies, and for a figure given in another role (such as a reference number), the rule checks which rupee figures appear, not which one the reply calls the refund. A number right after a suffix marker ("1899 रुपये 5 दिनों में") is read as a rupee figure, which can fail a correct reply. None of these spellings occurs in the saved v1.0.1 or v1.1 support replies. To be fixed in v1.1.1.
