I couldn’t agree more and I’ve said the same thing many times.
I have yet to see an LLM-generated app not collapse under it’s own weight after enough iterations/prompts. It gets stuck in loops (removing and adding the same code/concept), it gets hung up on simple errors, etc.
For greenfield it’s amazing, no doubt, but unless you are watching it very closely and approving/reviewing the code along the way it will go off the rails. At a certain point it’s easier to add the new feature or make the modification yourself. Even if the LLM could do it, it would burn tons of money and time.
I expect things to get better, this will not always be the state of things, but for now “vibe coding” (specifically not reviewing/writing code yourself) is not sustainable.
Most people doing it have a github profile that is a mile wide and a meter deep.
LLM’s are amazing and useful, but “vibe coding” with them is not sustainable currently.
>I expect things to get better, this will not always be the state of things, but for now “vibe coding” (specifically not reviewing/writing code yourself) is not sustainable.
It will not.
And I say this as someone whose been building internal LLM tools since 2021.
The issue is their context window. If you increase the context window so they can see more code costs skyrocket as n^2 the size of the code base. If you don't then you have all the issues people have in this thread.
The reason why I have a job right now is that you can get around this by building tooling for intelligent search that limits the overfill of each context window. This is neither easy, fast, or cheap when done at scale. Worse the problems that you have when doing this are at best very weakly related to the problems the major AI labs are focusing on currently - I've interviewed at two of the top five AI labs and none of the people I talked to cared or really understood what a _real_ agentic system that solves coding should look like.
I can't help but wonder whether the solution here is something like building a multi-resolution understanding of the codebase. All the way from an architectural perspective including business context, down to code structure & layout, all the way down to what's happening in specific files and functions.
As a human, I don't need to remember the content of every file I work on to be effective, but I do need to understand how to navigate my way around, and enough of how the codebase hangs together to be able to make good decisions about where new code belongs, when and how to refactor etc.. I'm pretty sure I don't have the memory or reading comprehension to match a computer, but I do have the ability to form context maps at different scales and switch 'resolution' depending on what I'm hoping to achieve.
> building tooling for intelligent search that limits the overfill of each context window
I'm interested to know what you mean by this, in our system we've been trying to compress the context but this is the first I've seen about filtering it down.
For general text you run some type of vector search against the full-text corpus to see what relevant hits there are and where. Then you feed the first round of results into a ranking/filtering system which does pair wise comparison between each chunk that you've had a good score from the vector search. Contract/expand until you've reach the limit of the context window for your model and run against the original query.
For source code, you are even luckier since there are a lot of deterministic tools which provide solid grounding, e.g., etags, and the languages themselves enforce a hierarchical tree-like structure on the source code, viz. block statements. The above means that ranking and chunking strategies are solved already - which is a huge pain for general text.
The vector search is then just an enrichment layer on top which brings in documentation and other soft grounding text that keeps the LLM from going berserk.
Of course, none of the commercial offerings come even close to letting you do this well. Even the dumb version of search needs to be a self-recursive agent which comes with a good set of vector embeddings and the ability to decide if it's searched enough before it starts answering your questions.
If you're interested drop a line on my profile email.
I'll preface this comment with: I am a recent startup owner (so only dev, which is important) and my entire codebase has been generated via Sonnet (mostly 3.7, now using 4.0). If you actually looked at the work I'm (personally) producing, I guess I'm more of a product-owner/project-manager as I'm really just overseeing the development.
> I have yet to see an LLM-generated app not collapse under it’s own weight after enough iterations/prompts.
There's a few crucial steps to make an LLM-generated app maintainable (by the LLM):
- _have a very, very strong SWE background_; ideally as a "strong" Lead Dev, _this is critical_
- your entire workflow NEEDS to be centered around LLM-development (or even model-specific):
- use MCPs wherever possible and make sure they're specifically configured for your project
- don't write "human" documentation; use rule + reusable prompt files
- you MUST do this in a *very* granular but specialized way; keep rules/prompts very small (like you would when creating tickets)
- make sure rules are conditionally applied (using globs); do not auto include anything except your "system rules"
- use the LLM to generate said prompts and rules; this forces consistency across prompts, very important
- follow a typical agile workflow (creating epics, tickets, backlogs etc)
- TESTS TESTS AND MORE TESTS; add automated tools (like linters) EVERYWHERE you can
- keep your code VERY modular so the LLM can keep a focused context, rules should provide all key context (like the broader architecture); the goal is for your LLM to only need to read or interact with files related to the strict 'current task' scope
- iterating on code is almost always more difficult than writing it from scratch: provided your code is well architected, no single rewrite should be larger than a regular ticket (if the ticket is too large then it needs to be split up)
This is off the top of my head so it's pretty broad/messy but I can expand on my points.
LLM-coding requires a complete overhaul of your workflow so it is tailored specifically to an LLM, not a human, but this is also a massive learning curve (that take's a lot of time to figure out and optimize). Would I bother doing this if I were still working on a team? Probably not, I don't think it would've saved me much time in a "regular" codebase. As a single developer at a startup? This is the only way I've been able to get "other startup-y" work done while also progressing the codebase - the value of being able to do multiple things at a time, let the LLM and intermittently review the output while you get to work on other things.
The biggest tip I can give: LLMs struggle at "coding like a human" and are much better at "bad-practice" workflows (e.g. throwing away large parts of code in favour of a total rewrite) - let the LLM lead the development process, with the rules/prompts as guardrails, and try stay out of it's way while it works (instead of saying "hey X thing didn't work, go fix that now") - hold its hand but let it experiment before jumping in.
This document outlines the standardized approach to ticket management in the <redacted> project. All team members should follow these guidelines when creating, updating, or completing tickets.
## Ticket Organization
Tickets are organized by status and area in the following structure:
TICKETS/
COMPLETED/ - Finished tickets
BACKEND/ - Backend-related tickets
FRONTEND/ - Frontend-related tickets
IN_PROGRESS/ - Tickets currently being worked on
BACKEND/
FRONTEND/
BACKLOG/ - Tickets planned but not yet started
BACKEND/
FRONTEND/
## Ticket Status Indicators
All tickets must use consistent status indicators:
- *BACKLOG* - Planned but not yet started
- *IN_PROGRESS* - Currently being implemented
- *COMPLETED* - Implementation is finished
- *ABANDONED* - Work was stopped and will not continue
## Required Ticket Files
Each ticket directory must contain these files:
1. *Main Ticket File* (TICKET_.md):
- Problem statement and background
- Detailed analysis
- Implementation plan
- Acceptance criteria
1. Create tickets in the appropriate BACKLOG directory
2. Use standard templates from .templates/ticket_template.md
3. Set status to *Status: BACKLOG*
4. Update the TICKET_INDEX.md file
### Updating Tickets
1. Move tickets to the appropriate status directory when status changes
2. Update the status indicator in the main ticket file
3. Update the "Last Updated" date when making significant changes
4. Document progress in IMPLEMENTATION_PROGRESS.md
5. Check off completed tasks in IMPLEMENTATION_PLAN.md
### Completing Tickets
1. Ensure all acceptance criteria are met
2. Move the ticket to the COMPLETED directory
3. Set status to *Status: COMPLETED*
4. Update the TICKET_INDEX.md file
5. Create a completion summary in the main ticket file
### Abandoning Tickets
1. Document reasons for abandonment
2. Move to COMPLETED/ABANDONED directory
3. Set status to *Status: ABANDONED*
4. Update the TICKET_INDEX.md file
## Ticket Linking
When referencing other tickets, use relative links with appropriate paths:
markdown
@TICKET_NAME
Ensure all links are updated when tickets change status.
## Ticket Cleanup and Streamlining
### When to Streamline Tickets
Tickets should be streamlined and cleaned up at major transition points to maintain focus on remaining work:
1. *Major Phase Transitions* - When moving between phases (e.g., from implementation to testing)
2. *Milestone Achievements* - After completing significant portions of work (e.g., 80%+ complete)
3. *Infrastructure Readiness* - When moving from setup/building to operational phases
4. *Team Handoffs* - When different team members will be taking over the work
### What to Streamline
*Replace Historical Implementation Details With:*
- Brief completed tasks checklist ( high-level achievements)
- Current status summary
- Forward-focused remaining work
*Remove or Simplify:*
- Detailed session-by-session progress logs
- Extensive implementation decision histories
- Verbose research findings documentation
- Historical status updates and coordination notes
### Why Streamline Tickets
1. *Git History Preservation* - All detailed progress, decisions, and implementation details are preserved in git commits
2. *Clarity for Future Work* - Makes it easier to quickly understand "what needs to be done next"
3. *Team Efficiency* - Anyone picking up the work can immediately see current state and next steps
4. *Maintainability* - Shorter, focused tickets are easier to read, understand, and keep updated
### How to Streamline
1. *Archive Detailed Progress* - Historical implementation details are preserved in git history
2. *Create Completion Summary* - Replace detailed progress with a brief "What's Complete" checklist
3. *Focus on Remaining Work* - Make current and future phases the primary content
4. *Update Status Sections* - Keep status concise and action-oriented
5. *Preserve Essential Context* - Keep architectural decisions, constraints, and key requirements
*Goal*: Transform tickets from "implementation logs" into "actionable work plans" while preserving essential context.
## Maintenance Requirements
1. Keep the TICKET_INDEX.md file up to date
2. Update "Last Updated" dates when making significant changes
3. Ensure all ticket files follow the standardized format
4. Include links between related tickets in both directions
## Complete Documentation
For detailed instructions on working with tickets, refer to:
- @Ticket Workflow Guide
- @Ticket Index
- @Tickets README
I have yet to see an LLM-generated app not collapse under it’s own weight after enough iterations/prompts. It gets stuck in loops (removing and adding the same code/concept), it gets hung up on simple errors, etc.
For greenfield it’s amazing, no doubt, but unless you are watching it very closely and approving/reviewing the code along the way it will go off the rails. At a certain point it’s easier to add the new feature or make the modification yourself. Even if the LLM could do it, it would burn tons of money and time.
I expect things to get better, this will not always be the state of things, but for now “vibe coding” (specifically not reviewing/writing code yourself) is not sustainable.
Most people doing it have a github profile that is a mile wide and a meter deep.
LLM’s are amazing and useful, but “vibe coding” with them is not sustainable currently.