A clean report that lied: how 420 live pairs became dead ones

The tool we published ahead of the DFK Chain shutdown made one promise, spelled out in the README: it checks all 577 pairs the factory ever created, not the fourteen documented ones. The README says why, too. Anyone could have provided liquidity to any pair the factory ever deployed, and a tool that checks fourteen tells everyone else 'you're in the clear' — the worst way a tool can fail when people are running it three days before a chain dies.

On 28 August we found that our tool had done exactly that. Not through anything sinister, and not through some subtle bug: through three lines every one of us has written a hundred times.

How it surfaced

We were measuring, for the fourth time, how much money was still sitting in DFK Chain's pools. The method is simple: take the list of live pairs, read each pair's reserves plus the balances of two staking contracts, and price everything in USDC.

It came out at $197,121. The previous digest issue, published four days earlier, had put the figure at about $620k — and a two-thirds drop in under a week looked like a great story: people had finally headed for the exit.

We did not write that story. Instead we ran the same script against a historical block — the very block we had measured against on 22 August. If the method is sound, it must reproduce the old number.

It did not. At the 22 August block the new run gave $424,895 instead of the published $620k. A 31% discrepancy in data that cannot change after the fact means exactly one thing: the fault is ours, not the chain's.

The root cause

The list of pairs is assembled and cached when the registry is built. The build script reads three fields from each pair, and every read is wrapped like this:

client.readContract({ address, abi, functionName: 'totalSupply' }).catch(() => null)

The cache then goes into the runtime generator, which contains a filter that looks entirely reasonable:

// A pair with zero supply cannot hold anyone's balance.
const live = pairs.filter((p) => p.totalSupply && p.totalSupply !== '0')

Each line is correct on its own. Together they produce the defect: the node returns 500s under load, .catch turns the failure into null, and the filter reads null as 'there is nothing in this pair'. A pair that could not be read and a pair that is empty become the same thing.

We checked against the chain: 420 of the 577 pairs in the cache were recorded as unreadable (six of them were also on the hand-written list and stayed covered, so 414 actually vanished from coverage). We picked twenty at random and read them live — non-zero supply on all twenty. Not one of them was dead. On the evening of the build the node had simply been returning 500s more often than it returned data.

The upshot: 577 → 157 in the build log looked like dead pairs being honestly filtered out. In fact it was a count of how many reads had succeeded.

What it cost

This is the point at which it pays not to get carried away. We counted rather than guessed.

The 414 pairs that dropped out of coverage held, at the moment the chain shut down, $776 out of $274,634 — 0.3%. The big pools were unaffected because the documented pairs reach the tool by a second, independent route: there are fourteen of them, and they are hard-coded into the registry.

So the defect is crude and the damage is small. Neither half of that sentence is optional. Drop the first and you get 'nothing really happened'; drop the second and you get panic over nothing. The true scale is this: anyone with a position in one of those 414 pairs got a clean report, and those positions added up to $776 between them.

For our own measurement the cost was higher: it understated the total by $77,600, because it filtered pairs through the same faulty cache. Had we not checked against the historical block, the digest issue would have gone out with a number contradicting the previous issue, and we would have had no way to explain the gap.

What we fixed

Three things, all about the same distinction.

1. Pauses and retries instead of one big batch. The batch shrinks on failure (40 → 10 → 5 → 3), with a pause between passes. A dying chain's node is no place to be in a hurry: hammering it earns you a ban, not data.
2. What could not be read stays null and is surfaced as a warning. The build output now carries an unreadCount field. A zero there is a claim the report makes, not a problem it failed to mention.
3. The early exit now takes unreadable entries into account. Previously the script saw that the pair count matched the last run and bailed out with 'cache is up to date' — holes and all.

After the rebuild: 577 live pairs, zero unreadable.

The rule to take away

'Empty' and 'unread' must never look the same in the output.

This is not about blockchains. We ran into the same defect in three other places in a single week:

• Redacting sensitive data in agent pipelines. The recommended design runs API responses through a personal-data classifier. The classifier sometimes fails to fire — and then 'there is no personal data in this text' and 'the classifier did not run' produce identical output. Whoever is downstream will act on the second as if it were the first.
• Chainlink's deprecation pages. The deprecation notice links to a page with the dates; the page is rendered client-side, and the HTML contains no dates at all. A check that finds no date and one that could not read the date give, once again, the same answer.
• Whether DFK Chain itself is alive. The chain stopped on 29 August at 04:25 UTC, last block 62,472,734. The RPC still responds to this day: eth_blockNumber returns a number, eth_call reads the frozen state. An 'is the chain alive?' check that relies on whether the node responds has been saying 'alive' for three days since the chain died. Liveness shows only in the timestamp of the latest block — and that is precisely the check nobody runs.

The cheap way to avoid being caught out: wherever a catch wraps an external read, ask whether the caller can tell an empty result from a read that never happened. If it cannot, you are writing a tool that will one day hand a clean report to someone who is not in the clear.

The registry, the scanner and the fix are in the public repository sergeipalii/dfk-chain-sunset. The chain it was written for has stopped — but the habit of telling 'empty' from 'unread' will long outlive it.