I Stayed in the Judgment Loop
A near-20-hour agent experiment in which I handled reasoning and major corrections while Codex handled execution and adaptation.
Recently, I asked Codex to build an archive of public documents from Chinese government websites.
I wanted to find potential opportunities in time: project applications, industry support programs, talent programs, digital transformation pilots, or directories looking for AI service providers.
Previously, I had two ways to do this. I could browse the websites myself, or I could write and maintain scripts myself. Either way, I was still operating at the execution layer.
This time, I gave Codex the goal and a few basic requirements. I let it break down the work, write scripts, run jobs, handle exceptions, and continue. I did not tell it how to solve every technical problem, and I did not inspect every request or every line of code.
I stayed in the judgment loop.
I was responsible for deciding whether the work was right. Codex was responsible for figuring out how to do it.
The judgment loop and the execution loop
| Loop | Owner | Work |
|---|---|---|
| Judgment | Me | Define the goal, test the logic, identify major deviations, and correct direction |
| Execution | Codex | Break down the task, choose methods, write scripts, debug, and handle exceptions |
| Deterministic work | Scripts | Inventory, fetch, parse, deduplicate, and validate |
| Evidence | Databases and logs | Show what was collected, what was missed, and whether a conclusion could be trusted |
A general-purpose LLM should not process all 140,000 documents one by one. Repetitive and deterministic work still belonged in scripts. Codex was useful because it could write, invoke, and adjust those scripts toward a goal while adapting to problems during execution.
The moments that required my intervention were not ordinary errors. They were moments when the conclusion stopped making logical sense.
First, the old configuration contained only 62 sources but claimed that the major sources were already covered. I did not think an old configuration could prove its own completeness. Coverage had to be checked against each agency’s full public disclosure directory. I corrected the definition of coverage. Codex then inventoried the section trees again and expanded the scope to 432 collectible nodes.
Second, the queue and document counts kept growing, but parts of the content distribution did not make sense. I pushed the task back toward data correctness instead of accepting 200 OK as success. Codex found that some requests were missing the real CHANNELID, removed 5,095 contaminated links, and repaired the missing and orphaned records.
Third, the system used backfill_complete=1 to claim that the Nanjing task was finished. But “source discovery is complete” does not imply that “the document queue is empty.” Codex audited the full database and found 1,342 pending jobs that had never been attempted.
The division of labor was the same in all three cases: I identified a logical contradiction and corrected the direction. Codex found the technical cause, chose the repair, and carried it out.
Codex handled execution-level problems such as WAF misclassification, duplicated request budgets, Reader database aggregation, DNS failures, and a database lock. I did not need to predict every exception in advance.
What it produced—and what it did not
After an engineering effort on the order of 15 to 20 hours, the system had built a historical archive containing 142,081 deduplicated public government documents, including 39,764 documents classified as notices.
Data coverage and timing
| Scope | Deduplicated government documents |
|---|---|
| Nanjing city-level and district databases | 113,789 |
| Shanghai unified policy platform | 10,399 |
| Changshu | 17,893 |
| Total | 142,081 |
This was not an exact 20-hour stopwatch measurement. The verifiable Jiangning pilot took about 2 hours and 39 minutes, and the Nanjing district-wide run took about 13 hours and 12 minutes. Other work was distributed across city-level collection, diagnostics, Reader integration, and data audits.
The Nanjing collection also preserved original responses, HTML, Markdown, plain text, structured metadata, version relationships, and SHA-256 hashes. Integrity checks for all 14 current Nanjing SQLite databases returned ok.
But my original goal was not to collect 140,000 documents. It was to find opportunities worth acting on.
The system currently has events=0 and briefs=0. The archive already contains at least three opportunities with near-term deadlines and one service-provider program with continuing business-development value, but a person still has to find them.
The accurate conclusion is: the collection layer worked, but the opportunity-discovery layer has not yet been productized.
At first, I thought I was simply handing a manual task over to AI. I later realized that I was moving myself to a more useful layer of abstraction.
I did not need to solve every implementation detail for Codex. But I had to make sure the goal was not quietly replaced by the execution process, that “done” had a precise meaning, and that a large result could survive logical scrutiny.
Codex could adapt the execution. It could not decide what outcome was truly valuable to me.
Perhaps the most important skill in working with an agent is not writing better prompts. It is knowing the layer at which I need to stay clear-headed.