Don't trust. Verify#

Posted 2026-09-03 by Garrett Smith

I hesitated to use AI for engineering tasks for a while. My early experience was mostly negative. The generated code was okay but required cleanup. At least that was my attitude. In the end I felt I was better off writing the code myself.

That changed with Opus 4.6. But I think more accurately, that changed when I started paying closer attention to the gains that were happening. I was late to the party.

At some point though the models went from “okay, but not ready” to “this is worth using consistently”.

And so I use models consistently. ¹

I was late to the party because I was too busy writing code by hand. I was satisfied that the models weren’t “good enough yet”. I suspected they might never be good enough.

But I missed something critical to the whole apparatus. I missed the role of tools.

#The role of tools

First, by tool I mean a program that runs a deterministic task and exits. Think POSIX toolchain. g++ is a tool. grep is a tool. cargo is a tool.

Tools narrowly constrain the world they operate in. They implement the laws of physics for a system. If you run bash -c "if true { echo hello }" you get an error. You violated the law. The tool constrains your inputs to a base syntax. Comply or fail.

AI’s success in programming tasks is due only in part to advances in language models. AI can enjoy any real success only because it operates in a well-ordered ecosystem of rules and consequences.

You can see this in action with lowly Sonnet and with highbrow Fable, or Mythos, or whatever God model comes next. They fly through a task, succeeding in parts and failing in others. The tools keep them operating within the laws of the system.

#Don’t trust

Bash doesn’t really care about trust. It’s not a topic of concern. Bash isn’t going to up and delete your production database. You need to tell it, specifically. Then it will.

We care about trust — specifically trusting that AI won’t harm us. We want to trust models. Anyone running in Auto mode with Claude Code is doing that.

Except this isn’t trust. If we really trusted LLMs we’d be harmed within hours if not minutes. And we’d stop using them.

LLMs work in programming tasks precisely because we don’t trust them. Every action a model attempts that affects your system is gated behind a tool — often a long pipeline of tools. It’s compelled to follow strict laws or fail. That’s not trust.

#Verify

The verify layer consists of consistent, provable, correct (or potentially correct) gates. I’m calling these tools.

Claude needs tools — they’re literally called this — to do anything outside forward pass token generation. If you want to verify AI, this is the point.

Let’s consider this expression, “Don’t trust. Verify.” I think it shifts our thinking.

Instead of thinking in terms of instructions that detail a set of rules the model must follow, we might instead think about a tool that imposes rules.

This is a pattern I run into in my use of Gage. Gage is good at finding things that Claude did that are wrong. Claude breaks rules all the damn time. This should come as no surprise, but it’s still a source of wonderment and frustration.

I have added this to CLAUDE.md:

Number 1 rule: DO NOT EVER EVER I MEAN NEVER DO...

And guess what.

No count of EVER is sufficient to enforce a rule.

Since then I’ve learned that “do not” rules work up to a point and their effectiveness degrades as context grows. Positive statements apparently work better.

You're so great Claude you're doing an amazing job!

Now, here's what I'm going for ...

And guess what.

Claude will break every rule you give it. It’s a matter of time.

The solution?

Don’t trust. Verify.

I build a tool. If I want Claude to do something really important, I think carefully about how to impose a strict verify step in the form of a tool. A law. Something immutable. And just like running Bash with the right syntax or compiling Go with the right file layout I impose order to the system with a deterministic program that imposes a ruthless barrier to the model. A model I don’t trust as far as I can throw it.

Claude Code lets you register hooks at various points to impose order. This is the sure fire way to implement the don’t trust, verify regime.

If you’re using context instructions, you’ll mitigate the chaos but you cannot tame it. I tend to start there because it’s easy to write rules, or positive and encouraging directions with examples. When I run into something that needs order, I turn to a tool.

#Why LLMs can’t be trusted

Here are some papers that shed light on the structural challenges of language models.

#Calibrated Language Models Must Hallucinate

The paper shows that a language model satisfying standard calibration properties must hallucinate on arbitrary facts. The hallucinated facts have no learnable pattern and occur at a rate close to the fraction of such facts that appeared exactly once in training data. The result holds regardless of model architecture or data quality. It does not apply to facts that recur in training data or to systematic facts such as arithmetic.

arxiv.org/abs/2311.14648

#Why Language Models Hallucinate

The paper argues that training and evaluation pipelines reward confident guessing over acknowledging uncertainty. Benchmark scoring treats an abstention as equivalent to a wrong answer. Models are optimized to behave like test-takers who guess rather than admit they do not know. This dynamic sustains hallucination even as benchmark scores rise.

arxiv.org/abs/2509.04664

#A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

The paper catalogs hallucination types and their causes across the training pipeline. It surveys detection methods and mitigation techniques. It states open research problems. It serves as the standard reference survey for the field.

dl.acm.org/doi/10.1145/3703155