Part of it seems to be that LLMs are used in a linear, tool-oriented way. You give them prompts, and it responds, in a linear fashion.
Brains are always thinking and processing. What would happen if we designed an LLM system with the ability to continuously read/write to short/long term memory, and with ambient external input?
What if LLMs were designed to be in a loop, not to just run one "iteration" of a loop.
I think you're 100% on the right track here. The key is memory, loops, and maybe a few other things like external interfaces which are just plain code and not deep learning voodoo. Many things do indeed run LLM's in a loop and attach external sources. See for example AutoGPT, the ReAct paper[1], and the Reflexion paper[2].
ReAct one line summary: This is about giving the machine tools that are external interfaces, integrating those with the llm and teaching it how to use those tools with a few examples, and then letting it run the show to fulfill the user's ask/question and using the tools available to do it.
Reflexion one line summary: This builds on the ideas of ReAct, and when it detects something has gone wrong, it stops and asks itself what it might do better next time. Then the results of that are added into the prompt and it starts over on the same ask. It repeats this N times. This simple expedient increased its performance a ridiculously unexpected amount.
As a quick aside, one thing I hear even from AI engineers is "the machine has no volition, and it has no agency." Implementing the ideas in the ReAct paper, which I have done, is enough to give an AI volition and agency, for any useful definition of the terms. These things always devolve into impractical philosophical discussions though, and I usually step out of the conversation at that point and get back to coding.
Or if they were just constantly prompted by outside stimulus. And if they could interact with the real world allowing them to observe cause and effect. In other words, if they were embodied.
Brains are always thinking and processing. What would happen if we designed an LLM system with the ability to continuously read/write to short/long term memory, and with ambient external input?
What if LLMs were designed to be in a loop, not to just run one "iteration" of a loop.