Judgment and verification
When Claude gets it wrong
The answers that get someone in trouble are usually subtle. The answer is formatted well, cites credible sources, and sounds confident. The problem is the content is wrong.
Claude predicts patterns from data, then presents what it found as conversational text. It will sound confident in the answer it found, but confidence does not equal accuracy. The model could be using incorrect reference material or not have the full context, but will present its answer as authoritative. This is the reason our human judgment matters so much when using AI like Claude.
A confident wrong answer
Client received a large deposit for a project that does not start until a few months from now. Should I recognize it as income when they received it?
Yes, any amount over $600 is required to be reported as income upon receipt according to the IRS.
Read it again. It is specific, it is calm, it names a threshold and points to an authoritative source, and you could paste it into a client email without a second thought. And it was wrong. The $600 threshold is for issuing 1099s, not for when a client receives an advance deposit. It also did not consider if the client was on a cash basis or an accrual basis. The IRS citation makes the response seem credible, but a human review would catch the errors.
You may not catch this by reading harder. It initially reads fine because it was built to read fine, and rereading a well-worded wrong answer only makes you more comfortable with it. The one thing a wrong answer cannot fake is your judgment.
Wrong arrives in many different shapes, and knowing what they are helps to catch them.
- 1Fabricated authority
A cite in the right form for a rule that is not real.
- 2Invented numbers
A threshold or figure stated flatly, simply made up.
- 3Flawed reasoning
Reasoning that looks sound but is not.
- 4Incomplete context
An answer that depends on context not given.
All four have the same tell, which is that they have no tell. They all appear to be correct, but after digging deeper it becomes apparent that they are not. That is why the defense is not just sharper reading. It is the habit of checking the claim against the source before you rely on it.