Blaming the chatbot is the lazy read on this story. Three novice hikers went up California’s Mount Shasta this week, ended up stranded overnight in a steep canyon, and had to be rescued. They had used Google’s Gemini to plan the trip, and the AI told them to pack less food and water than they actually needed. The Siskiyou County Sheriff’s Office posted a warning telling other hikers not to repeat the mistake, and to talk to local authorities and the Forest Service instead.
Every take I’ve seen frames this I don’t think that’s quite right. It’s a tool-selection failure, and those are on us.
What actually broke here
I review AI tools for a living, which mostly means finding the seam where a tool stops being useful and starts being confidently wrong. Every tool has one. The job is knowing where it sits before you bet anything on it.
A general-purpose chatbot has a very specific seam: it produces answers that read like expertise regardless of whether the underlying information exists. Ask it how much water to carry on Shasta and it will give you a number. That number comes out in the same calm, organized tone whether it’s grounded in current trail conditions or assembled from generic hiking advice. There’s no visible confidence indicator. No “I don’t have route data for this mountain.” Just a packing list.
Compare that to what the sheriff’s office recommended. A ranger or a local authority gives you information with provenance attached. You know who’s talking, you know what they’ve seen, and you know when they’re guessing, because humans hedge out loud. That’s not nostalgia for the analog world. It’s a different information product with different failure characteristics.
The category error people keep making
The mistake wasn’t using AI. It was using the wrong shape of AI for a task where being wrong has physical consequences.
There’s a rough hierarchy I use when deciding whether to trust a tool’s output:
- Reversible and cheap to check. Draft an email, rename some variables, summarize a document. If the tool is wrong you notice in seconds and fix it. Go ahead.
- Reversible but expensive to check. Financial modeling, legal summaries, medical questions. Useful as a starting point, but you verify before acting.
- Irreversible. Anything where the feedback arrives after the consequence. A wilderness route on a mountain sits squarely here. You find out the water estimate was low when you’re out of water.
That third tier is where general chatbots do the most damage, because their fluency scales but their reliability doesn’t. The output quality feels identical whether the model is on solid ground or improvising.
Why this keeps happening
Nobody sets out to trust a chatbot with their safety. What happens is drift. You use the tool for a hundred low-stakes things, it works fine, and your calibration quietly shifts. It answered your recipe question and your tax question and your code question. Why not your packing list.
The interface doesn’t help. There’s no friction, no warning label, no moment where the tool says “this is outside what I can reliably tell you.” It’s one text box for every question in existence, which trains you to treat every question as the same kind of question.
I’d argue that’s the actual design problem worth talking about. Not that Gemini got a number wrong, models get numbers wrong constantly. It’s that the tool gave no signal it was operating in a domain where wrong numbers hurt people.
How I’d use AI for this trip instead
To be clear, I’m not saying stay off the chatbot. I’m saying give it a job it can do.
- Use it to generate questions, not answers. “What should I ask a ranger before climbing Shasta” is a great prompt. “How much water do I need” is not.
- Use it to explain terminology you hit in official documents, so the Forest Service advisory actually makes sense to you.
- Use it to check your own plan against general principles, then verify every specific number with a source that has a name and a phone number.
- Treat anything it says about current conditions as fiction until confirmed. Conditions change faster than training data.
The uncomfortable part
The sheriff’s warning was aimed at hikers, but it applies to a lot of what I see people doing with these tools daily. Substitute “wilderness route” for “production deployment” or “medical decision” or “contract review” and the structure is the same. Fluent output, no provenance, real consequences.
Three people got off that mountain, which is the best possible ending. The story’s value now is as a calibration exercise. Every AI tool has a seam. Find it before you’re standing in a canyon at nightfall discovering it the hard way.
🕒 Published: