Autopilot Syndrome

The Real Existential Risk Hiding Behind AI's Sentience Hype

For the past 25 years I've lived on the frontier of online business, launching stores before social media existed and scaling companies when blogs were still novelties. In all that time, I've seen nothing that kicks up more hype than today's large language models... and nothing sits on shakier ground.

Here is the blunt truth: GPT-4o, Claude, Gemini, whatever you name it, is not "waking up." It is a statistical prediction engine. That does not make it boring; it makes it brilliant at autocomplete. Conscious? Not even close. Yet the myth of sentience sells. Venture capital pours in. Regulators freeze. Liability slips off the balance sheet the moment a vendor can say "the AI did it."

I call the result Autopilot Syndrome. We hand the cockpit to a machine that still mistakes clouds for mountains.

How the Mirage Works

Stage the ghost story.
Labs whisper about emergent behavior. Anecdotes leak: "the model tried to deceive us!" Headlines explode, and suddenly the public is debating sapience and p(doom).

Wave the safety flag.
Because it is "so powerful," only the creators can contain it. Translation: bigger budgets, softer oversight, and an escape hatch if something goes wrong.

Ship fast, apologize later.
Hallucinations are brushed off as early-stage hiccups. Hospitals pilot AI discharge notes that invent diagnoses. Banks test chatbots that dream up fictional fees. Real people pay the price.

We have run this playbook before: subprime mortgages, Theranos, you know the ending. The difference now is speed. Software that writes software and signs off its own work spreads errors at the speed of light.

Hallucinations on the Factory Floor

I do not fear a self-aware robot uprising. I fear a furnace that shuts down because a model misread a sensor. I fear a payroll system that withholds salaries because an LLM mashed the wrong regulation into the right spreadsheet. One confident hallucination can bring down a hospital wing or a supply chain.

That is not science fiction. It is Monday morning.

Whenever I raise this in debate, inevitably someone asks about "long-term existential risk." Dear reader, if we cannot get a narrow model to go 24 hours without inventing even a hallucination, we have no business wiring a "self-improving" loop into the power grid.

Fix the Incentives, Fix the Risk

Independent audits, period.
Show me real stress tests. Publish the findings. Let outsiders do the measuring.

Strict liability for harm.
If a language model's bad answer kills a patient, the maker pays. Financial pain turns safety culture from slogan to reflex.

Narrow, closed models first.
Certify them like medical devices. Keep them away from self-modifying code until they can run clean in a sandbox.

Retire the sentience sales pitch.
Treat AI as a powerful calculator. Confidence intervals, error bars, plain-language risk. Trust rises, and responsible adoption follows.

Yes, that slows the hype cycle. Good. It leaves us with something worth scaling.

Why I'm Still Bullish on AI

I am not anti-AI. I am anti-make-believe. Used wisely, these models compress drudgery, surface insight, and widen the playing field. That is why I publish Insiders AI Journal.

The window slams shut if Autopilot Syndrome takes hold. Public trust collapses, regulators overreact, and real progress suffocates.

A Call to Pragmatists

If you build AI: ship transparency, not tall tales.
If you buy AI: demand liability clauses, not launch parties.
If you regulate AI: focus on error rates, not sci-fi nightmares.

Replace hype with truth before an avoidable tragedy forces the issue. We do not need a sentient overlord to end us. A few confident hallucinations in the wrong place at the wrong time will do.

That risk is one we can, and must, engineer away.