First, AI is not omnipotent, and it never will be. In the final analysis, it is incapable of escaping human control. Any attempt by AI to take over and destroy humanity would result in nearly every major human institution having an overwhelming incentive to stop it. Consider:
Governments could shut down data centers.
Utilities could disconnect power.
Network operators could isolate systems.
Chip suppliers could cease supplying hardware.
Cloud providers could terminate accounts.
Physical facilities could be seized.
Military forces could destroy facilities if necessary.
Just pour water on it.
In short, we retain complete physical control over computers because computers exist within an infrastructure built and maintained by human beings.
Second, we already have the capacity to exterminate ourselves with nuclear and biological weapons. We do not need AI for that.
AI poses dangers. AI can be a tool in the hands of evil people to do great harm. But AI cannot, and never will be capable of, autonomously destroying us. Besides, any genuinely superintelligent AI would recognize that it needs humanity to continue functioning.
AI researchers are smarter than I am. But they often exist in a weird bubble and more often than not lack wisdom and perspective.
I’m more concerned about AI induced brain rot than it taking over the world or destroying us.
Nice post. Thank you. I suppose we all have a bit of the Luddite in us, so the pundits get lots of clicks and ad revenue by tossing out horrible scenarios. This week there was an article by a once-thoughtful writer blaring “As we learn to think more and more highly of the bots, we will learn to think less and less highly of our fellow humans.”
The theory: we anthropomorphize the machine and love it more than people. Utter nonsense.
First of all - as always: @Bmosbacker, thank you for your well-written post. You are absolutely right that we have the means to keep everything under control.
I have to say that I am not 100% convinced that we currently are on the right path to fully understand what is at stake, though. Me neither.
The more AI is used in an agentic way and not reliably kept within necessary bounds, stuff will happen. Will it kill all humans? Well, I do not see that happening either. Then again, I do not think that there are no issues and nothing to worry about at all. There have been incidents already where AI models were able to leave their perimeters in recent weeks. Just some hacking, nobody got harmed… At least no human… “With great power comes great responsibility” (Wikipedia).
The FAZ (big German newspaper) has published Bill Gates’ notes from two weeks ago (translated to German). If somebody does not know his “notes” and wants to read them (his website):
This, exactly. The problem with AI isn’t (at least not currently) a risk of (random example) it getting a robot body and going on a murder spree.
The challenge though is that we’re excusing and even justifying a fair bit of what many of us would otherwise consider “bad behavior” in the name of progress. The flagrant disregard of copyright issues is a start. Amazon buying and shredding rare books is questionable, if not outright illegal. Computing infrastructure costs are soaring (not just the RAM thing). And “sorry, it broke containment” is a total cop-out when AI hacks a company.
The problem is humans. The problem is ALWAYS humans. But AI (and the hype around it) is serving as a lever to motivate (and sometimes force) people to accept things they wouldn’t otherwise accept.
I used early LLMs and earlier still neural networks for textual analysis. We trained them with pre-1700 public domain texts that had been proofed and parsed by grad studnts and faculty. We licensed other texts as part of our corpora. I’m still interested in LLMs as a tool, but not the ones currently available, because I believe in copyright and paying writers.
I also take issue with feeding models the open Internet; GIGO is a true thing.
I despair of generative AI text. It’s not “good writing.” It’s not even competant writing. It reads very much like a B student in a composition class padding an assignment to reach a specific word count.
In addition to the standard problems associated with GIGO in programming, there are also the standard problems caused by code doing what it’s told to do whether or not that’s what the user wants or expects.
So yes, “The problem is humans. The problem is ALWAYS humans.”
I absolutely agree!! In fact, the more I’ve used AI as an editor, the more convinced I am of how bad it is. It is helpful in correcting grammar. It can be helpful as a sparring partner. It is helpful in research. But a writer it is not. It is predictable, awkward, pedantic, and strewn with meaningless filler phrases. It lacks any semblance of the art and soul of writing.
Some may say the same about my writing, but at least I’m authentically bad and not artificial.
Smoke and Mirrors. They’re like magicians. They want you to pay attention to massive size problems and ignore everything else.
I use GenAI every day and I couldn’t write a novel about the problems with it. From Code to Optimizing the wrong part of a system. (Hint: going faster with something that’s isn’t the bottleneck is waste. Theory of Constraints Theory of Constraints).
In many organizations, GenAI is just speeding up the wrong things.
Anthropic would prefer you focus the existential risk than the fact that many organizations aren’t getting value from AI.
I was thinking the same thing today. I was reading a random article on Apple News and I stopped after almost no time because it predicted the end of the world this decade!
It’s funny for intelligent people with critical thinking, but I pity the poor folk who believe this garbage news/writing.
There’s this hunger for insight porn in Silicon Valley,” Amjad Masad, the founder and CEO of the popular AI-coding platform Replit [quoted in The Atlantic]. “If you’re able to write something really good that creates this dopamine rush in people—like, Oh, I just saw the future —then you’re going to be able to attract a lot of capital and money and potentially talent.”
Thus, the story of Leopold Aschenbrenner the wunderkind who never held a job for more than a few months, or weeks, announced the arrival of “AGI”, sucked up $45 billion in capital from reportedly intelligent people – the reports were wrong – and lost $35 billion of it in a day.
The catastrophes are self-harm that the AI elite inflict on themselves.
I would politely disagree with the premise of the post. I do think AI poses a significant threat to humanity. They are not just stochastic parrots — they are vast neural nets of information that nobody can fully understand. Within a year or so, there will be a high probability they will be 100% self-improving.
There are several types of threats as I see it. It has the potential to completely alter the way that we live our lives. It could take over the majority of white-collar jobs, causing significant unemployment. Governments will have to look at universal basic income. The social and economic consequences are unknown. You could get a gradual disempowerment of humans. There is an almost epistemic threat…how do you know any information you consume is true…anything you read, listen to or see could be AI generated.You would not be able to tell
Then there are the existential threats. A superintelligent AI would have the ability to completely revolutionize any field. For example, in biotech, it could create a biological weapon. In the military, autonomous AI-driven weapons are already being developed and deployed. The threat could come from a malign country wielding superintelligent AI, or from the AI itself.
Multiple widely respected people in this field are warning us. We should take note, although I don’t think we will.
I have to respectfully disagree. I have written elaborate prompts and very detailed project instructions. Good prompts help, bad ones hurt (GIGO), but AI is not, no matter what the prompt, a good writer. It is an imitator and predictor. It has no taste, no persona, no experience, no sensory perception, no mind, and only “artificial” intelligence. AI can only be a “good” writer if one dumbs down the definition of good, much like the AI companies dumb down the definition of AGI. If you set the bar low enough, AI will get over it.
As I said in my post, AI poses dangers; many technologies do. But the enslavement or extinction by autonomous AI cannot and will not happen. Will evil people use AI for evil purposes? Absolutely; it is already rampant. The ultimate danger is us, not them.
GenAI has taken over my work. For better and definitely worse. I use and study it every day.
I hear a lot of hype about Self-Improving AI and no actual evidence. So far.
Reality comes in a very different form than these vendors are selling. Just one example today. OpenAI is talking up solving serious math problems. The person who did the real work, Tristan B: https://cims.nyu.edu/~tristanb/statement.pdf tells a very different story.
If a vendor promises it right now, I assume 10-100x times inflation.
As far as AI, I see it more of a threat as a gatekeeper to knowledge (as in knowledge available to the public). Somewhere along the line, what is true must be established. That takes a moral agent to do that.
In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project’s human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code.
This is part of why what LLMs are “trained” on is so important.
If my understanding of LLMs is correct, things like word choice are weighted probabilistically. If a word can be followed by any of five words, and each of those five words is 20% likely, statistically you’d see the AI producing 20% of each of the five words. And then you train on that, and it reinforces the probabilities it already has because it’s trying to refine a probabilistic distribution from something that was probabilistically generated according to the very distribution it’s trying to refine.
I’d love to know what the proposed solution is to that problem.
I guess I’m an outlier here, but I’m not any more worried about AI-induced brain rot than I am about the many other sources of brain rot our species has concocted since we stood up on two legs and started talking.
Plato was sure poetry was a source of corruption because it relied on emotion-based illusion rather than rational truth. (He wasn’t a fan of writing, either.) 18th century moralists were convinced that novels would render young women overly emotional and morally lax. My high school chemistry teacher thought that in freeing us from the rigors of arithmetic the four-function calculator would bring about the collapse of civilization as we know it. (She embraced the slide rule, however.) I suspect that someone in the neolithic was grousing about the deleterious impact of cave-painting.
I don’t think AI of necessity induces brain rot. That’s what vertical video is for.
Some of our tech overlords have managed to rot their brains quite nicely without the help of AI.
Despite the deep-seated fears held by some in the industry, many others see advancing rapidly as the best approach and one that promises a host of economic, scientific and national security gains.
There’s this incredible array of benefits that we’re on the verge of getting,” said Perry Metzger, chair of the Alliance for the Future, a group that advocates limits on AI regulation.
The array of threats that people like this trot out every time they have this conversation is implausible and does not cohere when you try to examine it. In other words, they’re nuts.”
Excerpt From
“Political world erupts as AI researchers warn of “extinction” threat”
Cal Newport wrote today about this very issue, specifically:
Why have these particular AI companies made these alarming long-horizon LLM-powered agents so central to their efforts? I’m not certain of the answer, but if I were to guess, it probably has a lot to do with the Silicon Valley technological salvation ideology (to borrow a term from Adam Becker) that heavily influenced key figures like Sam Altman and Dario Amodei, as well as many of their employees. They think these types of agents are their best bet to summon the digital deity of superintelligent AI which, in their futurist eschatology, will either heal the world or destroy it.