Hashline Edit

A simple way to improve model accuracy when editing using a hash anchor, surprisingly works pretty well.

Summary

• The harness seems to be one of the bottleneck for the model.
• Codex uses its own format though apply_path, which is probably embedded in the model for better accuracy. This format fails when applied to other models.
• Most harnesses use a simple string replace, which is basically like :s/old/new/g in Vim. Not perfect since it has to be the same, it’s string replacement after all. It fails when where’s multiple match or non-matching whitespace, which isn’t really important given that we only care about the actual content, but hey that’s how it works.
• Gemini does something similar, but it takes into account the whitespace, it does fuzzy matching.
• Cursor uses a separate 70B model for model apply, see this blog post.
• Aider shows in their benchmark that the edit format matters.
• There’s no real consensus on the best format. See EDIT-Bench and Jetbrain’s Diff-XYZ Benchmark
• The author introduces a new format which improves the model edit accuracy by giving each line a unique hash anchor, which is only 2 characters! So it shouldn’t be token hungry.
• A bunch of models improved, some significantly, some only marginally, but there’s improvement!
• See the harness problem for the original post, this one is focused on the Hashline Edit.

My Thoughts

Based on the result it looks promising, been testing for a few sessions, so far looks okay. The only downside is that I see the hash for each line. I could hide it but it’s probably good for verbosity in case things go wrong.

There were a few incorrect edits but I think it’s because the session was a bit flaky, had some internet connection, idk if it’s related but it’s probably that. It works fine on sessions that are not flaky.

I’m using pi-hashline-edit which is the extracted or simplified version based on the one in oh-my-pi.