Very interesting outcome. Shows how far it’s possible to get without being idiomatic.
You say the docs contain guidance on structuring, high-level constructs, and how to make things idiomatic. It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”.
If it picks up the same things that’s great news for novice programmers and anyone new to a framework!
LLMs will improve at making these fixes over time - even if they’re currently bad at it that won’t last.
But that's kind of my point - when I was originally working on it - I didn't know there's a more idiomatic approach - why would I copy paste the docs to ask it to improve the results ? Especially given the LLMs tendency to hallucinate and gnerate noise when asked to "improve code".
Also without reading the docs would I be able to spot why the idiomatic approach is better from single use case ?
You're basically depending on LLM to make choices for you - and in my experience so far it makes very suboptimal ones.
> It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”.
For your info, a few days ago, I was trying to migrate Tailwind 3 to 4 in a codebase where I had just set up the boilerplate, and I went through hell and eventually gave up. I used the Tailwind 4 documentation and Claude 3.5 sonnet in Windsurf IDE, and the part related to migration in my codebase was probably less than 50 lines. All of those tasks would take just a few minutes if a person did them directly (excluding reading the documents)
You say the docs contain guidance on structuring, high-level constructs, and how to make things idiomatic. It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”.
If it picks up the same things that’s great news for novice programmers and anyone new to a framework!
LLMs will improve at making these fixes over time - even if they’re currently bad at it that won’t last.