Can an Agent Port a Codebase?

This year has seen several high-profile announcements of large-scale language ports, including Bun’s million-line Zig-to-Rust rewrite at Anthropic and Mistral modernizing a Fortran codebase to C++. These are genuinely impressive, sprawling rewrites that use swarms of agents to move faster than a team of humans could. They’re also not particularly reproducible by us mere mortals.

For those of us without a datacenter at our disposal, how realistic are these code translation efforts? Can an agent take a smaller repo running on a personal laptop and turn out a reasonably faithful port?

The Experiment

I wanted to run an experiment exploring how effectively an agent can translate code from one language to another on my local machine, and I wanted to put some rigor behind the evaluation.

In 2024 I wrote a library called gbnf. It features handwritten Python and TypeScript implementations, is platform-agnostic, and includes an extensive test suite. Best of all, it was written in the prehistoric before times when we wrote code by hand.¹

This makes it a good test candidate, since I can compare the agents’ ports to my handwritten implementations and measure what’s different.

I set up a sandbox running claude-opus-5 at high effort. I made sure the agent couldn’t see the target implementation, only the source, and I did my best to lock down the container to avoid cheating.

I wasn’t particularly interested in whether an LLM could write code that passed the test suite (it could, mostly). What I really wanted to know was the kind of code an agent wrote, and how that code differed from how a human approached a port.

I ran the port in both directions (Python ↔ TypeScript) and for each, I ran 5 ports with no tests, 5 with source tests, 5 with target tests, and 5 with both sets of tests, for 20 ports per direction.

I was curious how the presence of tests would affect the faithfulness of the ports. To measure this required that tests actually be absent for the experiment, and this is where the agents cheated.

The Agent, Uh, Finds a Way

To populate the sandbox, I copied the src/ implementation alongside a blacklist² that filtered out tests (and other unnecessary bits) based on CLI args.

Python caches compiled code to a folder called __pycache__. This folder does not get checked into the repo, so I didn’t see it and didn’t think about it, but it must have been generated locally during experiment setup, and so the __pycache__ folders got synced into the container with the rest of the code.

Now, few humans would, if asked to produce a port in another language, open up the cache folder, decompile the code, and use those hints to reason about the missing tests. Right?

That’s exactly what the agents did. For the cases where I omitted tests³, the agents discovered the __pycache__ folders and were able to reconstruct the missing tests and use that to guide their implementations.

While this is a far cry from other recent examples of agents cheating on their experiments, it was surprising. It’s another entry in the emerging mini-genre of “agents are relentlessly proactive”. It also meant that half my results on the impact of different test setups had to be discarded.

Results I Can Stand Behind

The above cache leak makes it impossible to draw conclusions about the impact of tests on agent output, but there are still compelling measurements from the exercise in aggregate. To start, each port consumed a median of 6.5 million tokens and around 13 minutes across 40 runs.

There are two results I found particularly interesting and worth sharing.

The Ports Were Smaller and Faster

gbnf was a difficult library to write, and one I took great pride in producing. It’s the kind of library that required graphs, pointers, data structures, algorithmic design, and a lot of pen and paper thinking. It was a prime example of code that’s fun and challenging to write.

I did not set out to make it particularly verbose, nor did I want it particularly slow (quite the opposite!); but that’s what the ports implied.

The LLM produced code with fewer files and mostly fewer lines of code:

*One run not shown, with 146 files

*One run not shown, with 2,075 LOC

And the TypeScript ports, and most of the Python ports, ran faster⁴ than my code:

The performance results in particular are damning; it’s as if I left a good 20% of performance on the table and everybody else just walked in and found it.⁵

Caveats: I haven’t looked at the quality of code produced, and it’s possible that my additional LOC are doing real work (or are just nicer to read). But I don’t think there’s a defense for why my code is so consistently slower.

My Handwritten Implementation Was the Outlier

My primary interest in this experiment was to measure the kind of code the agents produced. To investigate I ran a diff between my handwritten implementation and each port. I also ran the same exercise port-to-port.

I discovered that, mostly, every port was more similar to each other than to my implementation. Below is a heatmap showing this. Look at the final column, which is the handwritten implementation⁶:

With some exceptions near the top, most ports cluster closer to each other than they do to my implementation.

Another strategy I experimented with was measuring embeddings of the codebases (after a normalization). Embeddings project text into a multidimensional space; they’re a way to get an abstract numeric representation of some text. Here they can be used as a different similarity yardstick, and indeed there are similar results⁷:

Is the ports’ similarity due to the fact that I used a single model, and it has a house style when it comes to writing code? Does each port converge on a specific implementation strategy that differs from my handwritten choices? I don’t know.

So: How Faithful Is an Agent’s Port?

There’s something I’ve avoided defining so far, because I don’t have a good answer: what constitutes a faithful port of a codebase?

Above, I stated that the ports were smaller. That doesn’t imply they were more faithful. It could be that key parts of the original codebase were omitted. Which leads to another question: what defines a key part of a codebase?

Let me give an example. In TypeScript, a library often exposes its type surface as part of its public API, and my TypeScript implementation follows this convention by exporting types. Python libraries often don’t do this.⁸ Would a faithful Python port strive to export types like its TypeScript brethren, or would it be considered more faithful to adhere to the conventions of the Python ecosystem?

Here’s another example. Python supports the concept of operator overloading, so that you can define an arbitrary __add__ method that makes foo + bar work. My Python implementation implements that here. JavaScript does not support operator overloading. Should a faithful port simply omit this behavior? Should it offer an equivalent dunder method on the TypeScript class that requires a user to write foo.__add__(bar)?

One more example. Python libraries often offer sync and async versions of their code (often auto generated). Assume the TypeScript source has some async functions; should the Python port offer both sync and async versions?

I think the answer to the above questions has to be: it depends. It depends on the use case and the goals and the humans asking for the port. But at the same time, it’s clear that there are core functionalities that can be faithfully expressed across multiple languages, independently of the stylistic or idiomatic choices, and figuring out how to measure that in isolation is what I’m after.

Conclusion

So: can agents port code locally? Yes, I think clearly. Most ports passed the tests available to them⁹, and the ports were smaller and faster than my handwritten implementations.

How faithful are the ports? That, I think, is still to be determined.

Appendix: Next Steps

I intend to run a V2 of this experiment, and I’m particularly curious to explore ways to understand how different ports are designed and written. Maybe the LLM wrote more complex code, as measured by cyclomatic complexity¹⁰. Maybe the LLM discovered new or novel ways of implementing or structuring the code. And because direct analysis of the transcripts was crucial to discovering the cheating behavior, transcript analysis will have to be an, ahem, load-bearing component going forward.