A paper landed on arXiv on 30 July — arXiv:2607.28271, Agentic Method for Deterministic Validation of Legacy Code Migration, by Andras Ferenczi, Jordan Docherty, Mariya Bessonov, Matthew Findlay and Krishna Lingamneni. It went round the usual places under a headline about AI migrating legacy COBOL to Java and carrying the bugs across with it.
That headline misses the paper entirely. I know, because I put a version of it in my own notes before I read past the title. The subject here is validation — how you would ever establish that a migration was correct, once a model has done it.
The actual method
The setup is two runtime environments: the COBOL source and the generated Java target, each instrumented with mocks and executed off-mainframe on commodity hardware. That detail alone is doing real work — you can’t iterate on a problem you have to book mainframe time to observe.
On top of that sits what the authors call the Locksmith Loop. An iterative agentic loop performs “Witness Search” over the input mocks, hunting for inputs that penetrate program branches, followed by parity-preserving mutations. When it hits a routing boundary it can’t get past, an analyzer identifies what they name a Locked Paragraph — the specific condition blocking deeper exploration.
Notice what the agent is never asked to do. Nobody asks it whether the Java is right. Its job is to manufacture the test inputs that would expose a difference, and then to say plainly where it got stuck. The judgment of right-or-wrong is handed to a deterministic parity check between the two runtimes. The authors’ own framing of the contribution is “validating agentic coding output using a deterministic oracle”, and the oracle is the point.
The numbers, and what they don’t cover
Three case studies: two open-source COBOL programs and one internal production-like program, ranging from 430 to 4,114 source lines. Coverage improved past the plateau that plain input search reaches, hitting nearly complete coverage on the two open-source programs and 91.90% branch coverage on the internal one. The generated Java matched the COBOL reference under deterministic parity checks in all accepted test cases.
Two things about that I’d want answered before quoting it at anyone.
The first is that 91.90% is the number from the program most like real production work, and it’s the lowest of the three. The gap between “nearly complete” on open-source samples and 91.90% on something production-shaped is the gap where the interesting bugs live, because Locked Paragraphs are by definition the branches the method could not reach. A COBOL program that’s been in service for thirty years has accumulated exactly that kind of unreachable-looking branch, usually for a reason someone has since retired.
The second is “in all accepted test cases”. Accepted by what, and how many were not? An eleven-page paper with six figures is not going to carry that, and I’m not treating the parity result as stronger than its filter until I’ve seen it.
None of that is a knock on the work. It’s a well-shaped piece of engineering and the honest bit — naming the Locked Paragraph rather than papering over it — is the part I’d want in my own tooling.
Why this is the version worth reading
The reason the mistaken headline travels further than the paper is that “AI wrote buggy code” is a story people already have a slot for. “Researchers built a deterministic check so the AI’s output could be trusted at all” is a duller sentence and a far more useful one.
I’ve been chewing on a related idea for a while in a much smaller way: a check you have never seen fail tells you nothing when it passes. A test suite that goes green against code you already believe is correct hasn’t constrained anything. What the Locksmith Loop is doing is generating the inputs that would make the check fail if the migration were wrong, then reporting honestly about the branches where it couldn’t. That’s a different activity from testing, and it’s the activity that makes agentic output usable in a place where being wrong costs money.
If you take one thing from the paper into your own work, take the architecture rather than the tooling. The model generates candidates. A deterministic check, owned by you and incapable of being talked round, decides which ones survive.