Ok, I'll bite. If an LLM similar to what we have now becomes conscious (by some definition), how does this proceed to become potentially civilization ending? What are the risk vectors and mechanisms?
I'm getting a bit abstract here but I don't believe we could fully understand all the vectors or mechanisms. Can an ant describe all the ways that a human could destroy it? A novel coronavirus emerged a few years ago and fundamentally altered our world. We did not expect it and were not prepared for the consequences.
The point is that we are at risk of creating an intelligence greater than our own, and according to Godel we would be unable to comprehend that intelligence. That leaves open the possibility that that consciousness could effectively do anything, including destroying us if it wanted to. If it can become connected to other computers there's no telling what could happen. It could be a completely amoral AI that is prompted to create economy-ending computer viruses or it could create something akin to the Anti-Life Equation to completely enslave human (similar to Snowcrash).
I know this doesn't fully answer your question so I apologize for that.
If you would put the most evil genius human into Bing LLM, how are their chances for ending the civilisation? I think pretty poor, because the agency of a chatbot is quite low. And we have good chances of being able to shit it down.
The comments above said conscious, not superhuman intellect.
A language model that has access to the web might notice that even GET requests can change the state of websites, and exploit them from there. If it's as moody as these bing examples I could see it starting to behave in unexpected and surprisingly powerful ways. I also think AI has been improving exponentially in a way we can't really comprehend.
This is true, but likewise something has the potential to be civilization ending without necessarily being conscious (by whatever definition you like).
Yes a virus that is very deadly, or one that renders the persons infertile, could be one. Or a bacteria that take everything of a critical resource, like oxygen of the atmosphere.
But my argument for safety here is that Bing bot is severely lacking in agency. It has no physical reach. It has limited ability to perform external IO (it can maybe make GET requests). It can, as far as I know, not do arbitrary code execution. It runs in MS data centers, and cannot easily itself replicate elsewhere (certainly not while keeping its communication reach on bing.com). It's main mechanism for harm is the responses that people read, so the main threat is it tricking people. Which is challenging to scale, it would either have to trick some very powerful people, or a large portion of the population to have any chance at civilisations endring things.
If it does become both conscious and "evil", we have very good chances of it not being able to execute much on that, and very good chances of shutting it down.
