Igor Eduardo

Good that the post says outright that matching Jev is not the same as being right. One thing I'd add before anyone copies the auto-approve-above-0.9 pattern from the intro:

If I'm reading section 2 right, the ECE of 0.0009 is measured on the Jev-labelled rows, so it tells you the student's probabilities match the teacher's. That's fidelity. A threshold needs calibration against human gold, because a 0.92 that faithfully copies an overconfident teacher 0.92 is still overconfident.

The gazelle93 set already has human labels, so a cheap check would be: on that set, per task, what share of decisions above 0.9 are wrong? And of those above-threshold decisions, how many flip when only the option order is shuffled? An auto-approve that depends on option order isn't a threshold you can lean on.

The scope point is the part I'd turn into its own gate. Citation precision can pass on a sentence that cites the right study and still broadens it: sample to population, "was associated" to "is", descriptive finding to "clinicians should".

A citation-density filter rewards citing, so it can't see that drift. A densely cited report can still overstate.

What I'd want next to the four metrics: small paired fixtures where the source supports only the narrow claim, and the check fails on the broadened version even though the citation itself is correct. Scope-preserved and citation-supported are separate verdicts, and a blended report score shouldn't let one cover for the other.

The useful cut is agent-reported "done" versus backend terminal state.

A green success from the same process that acted is a claim, not a grade. I'd want the eval to score durable side effects (ticket status, refund state, policy-cited fields) against a frozen oracle — and fail when the narrative is green while the database disagrees.

Practical check: can a run be marked done when the terminal-state fixture is red?

The part that travels for me is not "say no" as a slogan — it's writing the decision before any candidate is scored, then freezing the bytes the gate reads.

A +7pp held-out gain that still fails zero-regression authored cases is exactly the ship-or-not fork most dashboards hide: averages and critical behaviours are different predicates. Treating exit 3 (eval could not run) as a broken job rather than a reject is the other half — a crash must never look like a verdict.

Practical ask for anyone adopting this shape: do you commit gate.toml + authored cases before training/scoring the candidate, and do you refuse to score if the freeze hash moves without a deliberate re-freeze note?

Glad this board splits multilingual TTS quality from voice-cloning identity instead of one aggregate green.

The check I'd keep reading: whether the ranking still holds when you hold language/domain fixed versus when cloning is the axis. Otherwise teams optimize the easier subscale and still call the result "SOTA TTS."

The useful move in TraceML is forcing a named bottleneck before anyone touches precision, batch shape, or model choice.

If the run is input-bound, swapping bf16→int4 is sophistication theater. If it is compute-bound, more dataloader workers won't heal it. Splitting DataLoader wait, host-to-device, forward/backward, and optimizer time into separate signals is how you stop "Trainer feels slow" from becoming a random stack shuffle.

Practical ask on the next slow run: can you point to which of those four buckets owned the wall time before you change the model? If not, the first optimization is still debugging, not tuning.

The part that travels for unattended fixers is the verification funnel, not the LLM session count.

Serge's design makes that concrete: reproduce on GPU before patching, stop when the failure is already being worked, and treat "only expected values changed" as reward-hacking until a human (or a stronger check) says otherwise. Infrastructure timeouts and apply failures are not evidence the patch is good — they are evidence you don't have a release-grade gate yet.

Narrow release ask: before an agent is allowed to open a PR, can you name the discard reasons that kept weak candidates out? If the only green signal is "the model produced a diff," the harness is still grading fluency, not control.

The failure mode that matters for MCP agents is quieter than "hallucination": a claim that is true somewhere in the pooled tool outputs, attributed to the wrong source.

Source-blind faithfulness can still go green in that case — the fact exists in the soup. Source-aware verification is the contract upgrade: keep tool/source IDs through claim decomposition, support check, and attribution check, then allow/block with a per-claim source verdict a reviewer can actually inspect.

Practical ask for teams wiring MCP: when your eval says "grounded," does it mean supported-by-any-tool-output, or supported-by-the-source-the-answer-named? Those are different release gates. In multi-tool setups the second one is the one that catches cross-source conflation before a human trusts the citation.

The useful split is the one most dashboards still refuse to print: Mean@k answers “how good is this agent on average?” Pass^k answers “will the same request still succeed if I ask again?”

Once you’ve watched a trajectory chain dozens of near-tied tool choices, a strong average with a wide consistency gap is exactly the demo that fails live. Capability and consistency are orthogonal — buying a bigger model can lift Mean without closing the gap.

What I want teams to take from the Consistency Analyzer framing is operational, not the guideline machinery itself: treat flip-prone decision steps as first-class eval targets. One offline resample pass over a recorded trace is a cheaper reliability signal than another end-to-end leaderboard screenshot.

Practical ask for anyone shipping agents from this: do you report Pass^k (even k=3) next to Mean@k in the release gate, and do you refuse the “stable” label when the consistency gap is still large on your hard tier?

Reproducible reliability will not remove disagreement about which guidelines to inject. It will make the disagreement about the right metric.

The line that matters for practitioners is quiet: apparently similar scores can be produced under meaningfully different conditions.

Once you've watched a benchmark move just by changing inference-time compute or the elicitation protocol, "Model A beat Model B on X" without a card is marketing, not comparison. Evaluation Cards that ship verified results with configuration — especially transcript-level context where it's appropriate — are how eval becomes a contract instead of a leaderboard screenshot.

One ask I'd put to teams adopting EEE: which fields get skipped first under deadline pressure (seed, tool access, attempt budget, oracle feedback), and do you treat a missing field as "do not cite the delta" rather than "assume defaults"?

Reproducible reporting won't remove disagreement. It will make the disagreement about the right variables.

Exactly — and I’d make that a scored contract, not a vibe.

If the eval reward is “fewer tool calls” or “shortest successful trajectory,” you quietly punish the agent that does a read-after-write or re-checks status before claiming done. Those look like waste in a call-count chart and like reliability in an incident review.

I’d keep two numbers separate:

1. Progress — did environment state actually advance?
2. Verification cost — did we pay for a check that falsified a premature SUCCESS?

An extra call that confirms the postcondition is often the cheap insurance. Optimizing agents toward early stop is how silent misfires get selected for.

This split is the one product teams need: the overall task can pass while the agent abandoned your tool after a few errors and finished with a competitor. End-to-end pass/fail hides that.

I'd keep three lenses separate when reading traces:

• Discovery — was the right tool even visible / described well enough to be found?
• Selection — given the options at that step, did it pick the right one at the right time?
• Effect — did the call's postcondition hold in the environment, or did the agent interpret a receipt as proof?

Repeated calls are especially easy to misread. Productive recovery and stuck loops look similar until you ask whether state advanced. Same for "fewer tool calls": sometimes verification should cost an extra read-after-write.

Patterns across runs beat any single failed call. That's how you tell "this agent is noisy" from "this API/docs/SKILL.md consistently teaches the wrong next step."

The useful result here isn’t another leaderboard crown — it’s the error taxonomy: hallucination vs retrieval miss vs reasoning miss as separate arms.

If retrieval sets the ceiling, swapping the generator first is often the wrong diagnosis. Pin retrieval accuracy as a hard gate before debating reasoning quality.

The useful split here is judgment type, not another crown for a store.

Rerank asks “in what order?” Relevance filtering asks “should this context reach the generator at all?” Those are different contracts. A hybrid lexical+dense pool can look fine on nDCG while still shipping distracting neighbors into the prompt — and a leave-out policy is doing different work than sorting the same hundred hits.

I’d keep two gates separate: (1) retrieval/ranking quality on the pool, and (2) a missing-evidence / empty-after-filter policy before generate. When the filter clears the deck, that should be a first-class outcome in the eval report — not a silent fallback into fluent guessing.

Their NanoHotpotQA table is a good reminder to publish both retained-set size and ranking quality together; I would not treat either number alone as proof the end-to-end answer got better.

Local-first redaction before share is the right boundary — especially the deterministic secret layer ahead of the span model.

Two eval cuts that helped me when scrubbing sensitive traces/notes:

1. Report credential/secret FP·FN separately from PII-span F1. A missed sk-… is a different failure mode than a missed person name; mixing them into one score hides the risk that matters for publication.
2. Add a cheap "utility after scrub" check: can another builder still follow the procedure from the redacted trace? Span metrics can look great while the teaching value of the shared trace is gone.

Human review stays mandatory either way — automatic pass is a release candidate, not a green light.

Useful release — synthetic multi-domain PII with span labels is exactly what most teams lack for redaction eval.

One evaluation gap we keep hitting when moving from multi-industry synthetic sets into clinical notes: aggregate recall/F1 is usually dominated by high-frequency classes (names, emails, phones). Structured identifiers (national IDs, MRNs, account numbers) fail differently — format-valid lookalikes and locale-specific validation rules. Worth holding those classes out as their own slices (and reporting false positives on lookalikes) before treating a headline recall number as ready for PHI pre-index redaction.

Synthetic + open is the right foundation; the holdout protocol is what makes it transfer.

Really useful that GLiNER-multi-PII scored worst on medical PHI here — general PII setups hide clinical failure modes. When we evaluated de-id on 500 synthetic Portuguese clinical notes, aggregate Macro F1 looked acceptable until we broke out false positives on hard numeric lookalikes (we planted ~1940 distractors that resemble national IDs). For PhysioNet-DEID-style benches, reporting lookalike-ID FP separately is as important as entity F1. Domain-adapted models still need domain-shaped negatives.

Nice demos — the 8-category Privacy Filter taxonomy is a strong EN consumer/sanitization baseline for pastebins and screenshots.

One gap I keep hitting for clinical notes outside EN: taxonomy coverage ≠ “run the same 8 labels on another language/domain.” Locale clinical text needs entity classes and distractors that the EN consumer schema doesn’t encode; otherwise you measure a different product than the one you deploy. Synthetic labeled notes help for that coverage test before anyone claims compliance.

Curious whether you’ve tried feeding clinical-style notes (even synthetic) through the same 8 labels and where the misses concentrate.

Nice result on specialized encoders beating prompt LLMs for span-level PII — matches the pattern that token classification is the right task shape.

One eval gap I keep hitting on clinical notes: for structured national IDs, span NER alone still false-positives on digit lookalikes. On a 500-note Brazilian synthetic set (670 PII spans + 1940 numeric distractors), adding checksum validators as a post-NER filter (mod-11 style) was what collapsed false positives on ID-like strings; default recognizers without that check looked much weaker on Macro F1.

Your table already flags national_id as harder than email/ipv4. Worth reporting FP rate on checksum-invalid distractors separately from true IDs, and treating checksum/post-rules as part of the system under test — not only the encoder. Also noted Limitation 1 (English-only): locale ID classes need their own synthetic distractors or the English F1 overstates production readiness outside EN.