Pre-registered audit

Two instruments, checked in advance

A pre-registered audit of the KKBOX / WSDM Cup 2018 churn dataset and the τ-bench / τ²-bench evaluation harnesses. The defect I went looking for was in neither. Seven other things were.

What to check first

If you check nothing else

The plan below was written, hashed, and publicly registered before either instrument was opened. The registration is osf.io/ksm3n, DOI 10.17605/OSF.IO/KSM3N, timestamped 2026-08-24T15:33:01Z. The plan is published here in full. Verify the hash against the document, and the timestamp against the registration.

  • Frozen plan SHA-256b15f472ef72994970957eda1e1ebc53f5480b759eb6ab71e076496145a7fda0e

01What was fixed in advance

I have a paper about a defect I found in a benchmark I built myself. A label rule recorded a unit as degraded only when a failure condition and an eligibility condition held together; eligibility was one-directional, so once a unit crossed the threshold it could never satisfy the rule again. Units that were failing continuously became structurally unlabelable and were filed as healthy controls. 69.2% of the control class turned out to be those units.

The obvious question is whether this happens anywhere else. The obvious way to get that question wrong is to go looking for it and find it.

So before opening anything, I wrote down: eleven questions about how each instrument handles units it cannot evaluate, a decision rule with seven branches mapping evidence to verdicts, a materiality threshold of 5% of units in the observation window, the expected result — that both would come back negative — and a commitment to publish either way, including an indeterminate outcome. Then I hashed the document and registered the hash publicly.

The decision rule turns on one question: does the unevaluable unit leave the denominator, or stay in it and get counted as a success? Only the second is the defect. Leaving is exclusion — a real problem, a different one.

There is also a persistence requirement: the unit must keep accruing real failures while being structurally unrecordable. Termination is not absorption. A trial that crashes and stops has simply ended.

02The two instruments

KKBOX / WSDM Cup 2018 churn dataset — a public churn-prediction dataset from a music streaming service, distributed through Kaggle, with roughly a million labelled users. Its churn label is a conjunctive rule with a window condition, which is the shape the paper describes.

τ-bench and τ²-bench — Sierra's agent-evaluation harnesses. Scores from these appear in frontier-model reporting and on a public leaderboard. The relevant object is pass^k, computed across repeated trials of the same task.

Both were pinned by commit or release before reading. Nothing was executed: no model was called, no benchmark was run, no cost was incurred. For KKBOX this was a data read under the competition's rules; for τ-bench it was a code read at three pinned commits.

03The result: not found, in either

Result

Neither instrument carries the paper's mechanism.

InstrumentAssignment
KKBOX / WSDM Cup 2018NEGATIVE — exclusion
τ-bench (tau-bench, dormant) — instrumentMISSCORING
τ-bench (tau-bench) — pass^kNEGATIVE — correctly handled
τ²-bench (tau2-bench v1.0.1) — instrumentMISSCORING
τ²-bench (tau2-bench v1.0.1) — pass^kNEGATIVE — exclusion

KKBOX excludes out-of-window users from the labelled cohort rather than retaining them as successes. The documentation states this outright and the data agrees: 92.46% of out-of-window users are absent from the label file entirely. That is the correct treatment.

τ²-bench removes trials it cannot evaluate from the metric. Infrastructure-error trials are filtered out before pass^k is computed. The unit that cannot be evaluated is the trial, and it leaves. The unit that stays — the task — can be evaluated, on its surviving trials. There is no unit here that is both unevaluable and retained.

With three earlier gate checks on other instruments, that is five instruments I did not build examined, and five negatives. Two of the five — these two — were pre-registered. The other three were earlier checks run without registration, and I do not present them as more than that.

The prediction was right about direction and wrong about mechanism

On record before the read: "I expect both to come back negative, and I expect KKBox in particular may turn out to be window censoring rather than absorption."

Censoring was specifically tested for and found absent in the release where it could be tested — churn is flat at roughly 6% across expiry days 1–27 while observability falls from 27 days to 1. The predicted mechanism was examined and ruled out. The finding is exclusion.

For τ-bench the plan predicted the exact place to look and the code had the structure there. What it did not anticipate is that the structure can be present without the conditions being met — a code path is not an instantiation. That distinction is the most useful thing this exercise produced about the protocol itself.

04The seven other findings

None of these is the paper's mechanism. All are checkable by a third party against a fixed public artefact.

Evidence grades are carried throughout. Each says how far a claim could move if someone re-derived it: [E] is read off a fixed artefact and will not move; [I] is traced rather than executed and might.

  • [E]  read directly off code or documentation
  • [E†]  exact given a stated cutoff
  • [R]  dependent on a reconstruction that reproduces the cohort at 88.78%
  • [I]  inferred from traced control flow, not executed

KKBOX

1About a million published labels cannot be verified against the files shipped with them. [E]

transactions.csv ends at 20170228. The February expiries it is meant to explain need observation through 30 March under the stated 30-day rule. No label in the file can be checked against the transactions distributed alongside it. For the original release this is a publishing artefact rather than a labelling one — KKBOX's stated log history runs to 2017-03-31, so the labeller had what the rule requires. For the _v2 release I cannot say the same: its window is March expiries needing observation to 30 April, beyond that stated history, and only one of the thirty-one expiry days has full observation inside it.

2The cohort rule does not reproduce. [R]

Best reconstruction matches 881,477 of 992,931 labelled users — 88.78%. The distributed labeller script has its dates modified for local running and is explicitly not the production configuration, which likely explains the gap. Every count marked [R] in this audit inherits it.

3The headline churn rate moves 41% between adjacent months. [E]

6.392% for February (n = 992,931) against 8.994% for March (n = 970,960). Nothing in the distributed material establishes whether that is seasonality, cohort composition, or a difference in how the two label sets were produced.

490.45% of the _v2 cohort cannot have their cohort-entry expiry established from that release's own files. [E†]

This is the firmest number in the audit, and what it establishes is that a question cannot be answered. It is where the defect was most likely to appear, and it was not testable there. Blocked, not negative — and reported prominently rather than buried, because the alternative is assigning on the convenient half of the evidence.

τ-bench / τ²-bench

5pass^k has two documented conventions and the instrument does not reconcile them. [E]

At the pinned commit, three in-tree sources say the metric excludes infrastructure-error trials — the code itself, the code's own warning string, and CHANGELOG.md:103. One says the opposite: RELEASE_NOTES.md:83 states that "pass^k values are recomputed counting infrastructure-error simulations as failed trials (the leaderboard convention)." Nothing in either document states which convention governs a published figure. A reconciliation is available — that the leaderboard convention is a recomputation separate from the library metric — but neither document says so and this audit does not supply it.

Under the exclusion convention, which is what the code does: pass_hat_k is applied per task with that task's surviving trial count as the denominator, and the suite figure is an unweighted mean across tasks. Since the function returns C(successes, k) / C(trials, k) and C(n,k) grows with n, the same success count contributes more from a task with fewer survivors — at k=2, two successes give 1.0000 on two survivors and 0.1000 on five. A task whose failed attempts were dropped contributes the maximum, then averages with equal weight against tasks measured on their full record.

How often this bites in practice depends on infrastructure-error rates, and I executed nothing. The mechanism is in the code; the frequency is not something a code read can produce.

6The exclusion is applied on an attribution the code never establishes — and the maintainers found one instance themselves. [E]

progress.py:92 catches with a blanket except Exception as e:; :120 assigns INFRASTRUCTURE_ERROR unconditionally, with no inspection of the exception; agent_metrics.py:145 then filters on that label alone.

And :124-129 retains error_type, the full traceback, and the attempt count — precisely the information that would distinguish a provider outage from a model emitting malformed JSON. The instrument keeps the evidence and uses the assertion.

CHANGELOG.md:103 documents that the maintainers hit this: hallucinated tool calls propagating through run_with_retry and being "binned as INFRASTRUCTURE_ERROR." Agent failure, excluded under an infrastructure label. Fixed in v1.0.1. A second route appears still open at the pinned commit: llm_utils.py:440 parses json.loads(tool_call.function.arguments) outside any guard, so a model emitting invalid JSON is retried, relabelled, and dropped.

7"τ-bench" does not identify a version. [E]

The name designates the original repository, the tau2-bench repository, and v1.0.1 of that repository — while v1.0.0 is titled τ³-bench and no tau3-bench repository exists. A reader comparing published results across papers cannot resolve which artefact produced them from the name alone. Not a defect, but the v1.0.1 grading update required leaderboard submissions to be re-graded, which suggests version identity already matters here.

One finding that runs in the maintainers' favour

The Q7 question asks which tasks have a goal state satisfiable by taking no action — where an agent that terminates without acting scores a perfect 1.0.

In the original repository: 24 of 165 test tasks, 14.55%, and 38% in the airline domain. In v1.0.1: 11 of 2,556, 0.43% — airline and telecom at zero, banking at 4.12%, retail at 5.26%, and a ten-task mock fixture domain at 10.00%, the highest share of any domain. Excluding the fixture, 10 of 2,546, 0.39%. Whether a fixture domain belongs in a suite-level figure is a question the frozen plan did not anticipate, so both are reported.

Both counts are [I], not [E], resting on a static analysis that was wrong twice before it was right. And the narrowing matters: what passes by inaction is a trial terminating normally without acting, which is materially smaller than the question's premise.

I assumed I was measuring a static property and found I was measuring an improvement. The surface shrank by more than an order of magnitude, and the release notes credit the task fixes to outside analysis by others. It shrank because the maintainers acted on someone else's audit.

05The deviations that made the findings smaller

Eleven deviations are logged. Four record decisions that reduced what could be claimed, and they are the part of this record I would point a skeptical reader at first. Anyone can publish a null; a documented trail of points where a finding could have been made larger and was not is harder to manufacture.

  • A confirmatory test was designed that was structurally incapable of failing. Two sets stood in a subset relation by construction, so the check could not have come out any other way. It was caught by the executing agent, not by the party that designed it, and the substantive conclusion was recovered from a different, non-degenerate measurement. (D-5)
  • The only positive-shaped result in five checks was drafted, then withdrawn. τ²-bench's pass^k cell was written as POSITIVE and reversed on re-reading the frozen plan's requirement that a unit "must persist and keep accruing" real failures. Establishing that would require running the benchmark, which the plan forbids. The reversal moved away from the more interesting finding. (D-9)
  • A finding was nearly sent that the maintainers had already addressed. The release documentation was never swept during the read. Sweeping it afterwards found the second pass^k convention and the maintainers' own record of the mislabelling — which strengthened one finding and rewrote another, and would have been embarrassing to discover after publication. (D-10)
  • A prevalence claim was not made where the evidence was [R]-graded and the reconstruction gap reached every magnitude in the section.

06What this does not support

It does not support a claim that the defect is common. Five instruments, five negatives — and all five are public, built by people who expected outside scrutiny. This says little about private internal metrics no outsider has read.

It does not support a prevalence claim of any kind. Seven findings across two instruments is not a population.

The one figure clearing the materiality threshold on the firmest evidence establishes only that a question could not be answered. Nothing that clears the threshold anywhere in this audit is read rather than inferred.

And it does not support reading a present code path as a live mechanism. That call cost the only positive-shaped result available, and it was right.

07Provenance, stated plainly

The code reading and the analysis were performed by an AI coding agent working to a brief fixed before the read. The evidence summaries and the draft branch assignments were prepared with AI assistance.

I commissioned the work, held the procedural gates, ruled at the decision points the frozen text does not settle, and caught errors — including several where accepting the analysis would have produced a more flattering result. This is not expert review, not peer review, and confers no warrant.

Total effort across both instruments: 3.55 hours. Twenty-six minutes of wall clock on the first, thirty-five on the second. That number is recorded plainly because the claim that this check is cheap to run is unsupported without it.

08Correspondence

Both instruments' maintainers were contacted before this write-up, with a correction window closing 6 September 2026. Neither replied. What follows is the full account, including where the contact failed.

Sierra — contacted, no reply. Via the project's own documented channel, a public GitHub issue on tau2-bench, 28 August. (The repository publishes no security contact and names GitHub Issues as its route for questions.) A direct note went to two named authors of the τ²-Bench paper on 31 August, after the issue had sat five days without a reply, a label or an assignee. As of 16 October the issue remains open, unlabelled and unassigned, with no comment from the maintainers.

The KKBOX notification was only partly delivered, and this is the accurate account

§8 of the registered plan names the dataset's publishers (KKBox). The WSDM Cup 2018 competition had four organisers: Shou-De Lin (National Taiwan University), Xing Xie (Microsoft), and Yian Chen and Yuh-Ming Chiu (KKBOX). The competition site names all four and publishes no addresses for any of them.

On 28 August an email went to the two for whom public addresses could be found:

Shou-De Lin — National Taiwan University Delivered. No reply.
Xing Xie — Microsoft NOT DELIVERED. The address used was inferred from his institutional profile and was never published by him; it returned, twelve seconds later, "the address couldn't be found, or is unable to receive mail."
Yian Chen, KKBOX No public address found. Not contacted.
Yuh-Ming Chiu, KKBOX No public address found. Not contacted.

Addresses are not reproduced here. Two of the four organisers had addresses that could be found; one was published by its owner and one was inferred from a profile URL. Neither is evidence of anything a reader could check, and one of them may not belong to the person it was sent to.

So the two people §8 names — the KKBOX employees — were never reached, and of the two academic co-organisers, one delivery failed silently.

The bounce notice arrived in the same inbox that recorded the send, on the same day, and was not noticed until 8 September.

This is a partial discharge of §8 and is recorded as one. The effort was genuine — all four organisers identified, both findable addresses used — but "the organisers were notified" would not be a true sentence, and this write-up will not contain it.

On 9 September, after publication, a notification also went to KKBOX's corporate business and press addresses. It did not bounce, and no reply has come as of 16 October. It does not change the partial discharge recorded above: sent after publication, it offered no correction window.

Any reply is published below, in full and unedited, whenever it arrives — including after publication, marked with the date it came. Where there is no reply, that is stated rather than omitted.

An independent report on the same harness, which I did not solicit and had no part in

On 3 September, six days after I finished, a researcher I had not been in contact with before or during this audit filed issue #502 on tau2-bench and cross-referenced mine from it. I opened contact afterwards, on 11 September, through the channel his repository's README designates.

Where I read code and ran nothing, he re-graded public trajectory files across four builds. He found that evaluate-trajs replays certain telecom files in the wrong environment and scores them anyway — four files with recorded values of 0.7478, 0.7215, 0.9035 and 0.9890 re-grade to 0.1601, 0.0965, 0.1754 and 0.1732, with no error raised and a database match rate of 3 in 456. He controls for it properly: the companion file that is classified correctly re-grades to its recorded value. He also reports that four of 136 pairwise leaderboard orderings reverse between builds.

None of that is the defect I went looking for. In his cases the trial stays in the denominator and is scored as a failure — not relabelled and counted as a success. Under my own decision rule that is a misscoring, a different branch. I am not claiming his findings as mine and there is no overlap in what we assert. He says the same in his report, and characterised my work accurately in substance; the one detail he understated — what the registration actually fixed — I have since corrected through the channel his repository designates.

I am including it because it changes what my null means. My gate came back negative on this harness. That is a statement about one specific defect, not a clean bill of health — and six days later somebody using a method I did not use found substantial faults in the same instrument. A negative result is evidence about the question you asked. It is not evidence that there was nothing to find.

09How to check this

Everything below is public. The record lives in two places on purpose — GitHub is browsable and diffable, OSF is an archive with a DOI. A record whose whole claim is verifiability should not depend on one host staying up.

The full record: github.com/davidmullett/loopzero-audit-2026 archived copy at osf.io/4uxz3

Where
The frozen plan, in full, with its SHA-256 PRE-REGISTRATION.md PRE-REGISTRATION.sha256
The registration — the timestamp anchor osf.io/ksm3n DOI 10.17605/OSF.IO/KSM3N
The answers — every question, with citations to paths and line ranges at the pinned commits, and an evidence grade on each claim KKBOX-ANSWERS.md TAU-ANSWERS.md
The branch assignments, with the reasoning for each condition and the two hard calls stated as calls KKBOX-VERDICT.md TAU-VERDICT.md
The deviations log — all eleven, including the four that reduced what could be claimed DEVIATIONS.md
The analysis scripts — runnable against your own copy scripts/
What was sent to the maintainers, and when correspondence/

Verify the plan yourself. The registered artifact is a hash, so this is the check that matters:

curl -sL https://raw.githubusercontent.com/davidmullett/loopzero-audit-2026/main/PRE-REGISTRATION.md | shasum -a 256
b15f472ef72994970957eda1e1ebc53f5480b759eb6ab71e076496145a7fda0e

The plan was frozen at 2026-08-24T15:33:01Z and registered 27 minutes later, at 15:59:52Z. Both reads were completed on 27 August. The hash was published before the instruments were opened; the plan itself was withheld until now, which is why the registration records a hash rather than a document.

No KKBOX data is published. Not a row, an extract, a derived dataset or a sample file. §7.A of the competition rules permits use of the Competition Data for "academic research and education" and for nothing else; §7.B requires participants not to "transmit, duplicate, publish, redistribute or otherwise provide or make available the Data to any party not participating in the Competition."

The scripts are published under §8.B, which permits public sharing of "source or executable code developed in connection with or based upon the Competition Data" and deems anything so shared licensed under an eligible open-source licence. They are MIT. So anyone who has accepted those rules can obtain their own copy and point the scripts at it with KKBOX_DATA. Verification transfers; the data does not move.

The τ-bench repositories are MIT-licensed, so the code is quoted directly and every claim cites a path and line range at a pinned SHA.

Any reader can apply the stated rule to the published evidence and see whether they reach my branch. Disagreement is invited, and substantive disagreement gets published alongside the finding.