The Deep Dive
What your news reporter didn't mention. Primary documents, named defendants, and the gap between the caption and the docket.
The Deep Dive
Calculator vs Chatbot
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
You type two times two into a calculator, you get four. You type it again, you get four. You type the same tax question into a chat bot. First time it gives you one number, second time it gives you another. That is not a glitch, that is the product. This is the deep dive. Today is not about hating the tool. It is about the gap between what people think the tool is and what it actually is. Especially when the number you send the IRS is on the line. How good is the bot at figuring taxes, quarterly estimates, first year books, married filing jointly, real QuickBooks numbers? Short answer, not good enough to set the number you mail. Independent tests put general chatbots in the coin flip range on tax questions. Full return tests put the whole class of models in the mid 20s to low thirties percent correct. On a professional tax exam set, Grok was the only model to pass. 56%. That is still a failing grade for a return you mail. One study had Grok regress about 17% of the time, wrong on a question it had already gotten right. The New York Times ran scenarios and found chat bots off by an average of more than $2,000. NerdWallet watched one bot say tens of thousands and another say thirty two thousand. Same facts. Direction both ways, over and under. The two thousand dollars is the size of the miss, not a bias toward the taxpayer. Fine for explaining a concept, fine for a rough ballpark, not fine for the check. A first year of quarterly estimates is the hard case. No prior return, no safe harbor to hide under. The IRS dates are the fifteenth of April, June, September, and January. Miss September and the penalty starts stacking while you are still guessing what the year will look like. People say it will get worked out at filing. That is true. The penalty is also true, and the bot being two thousand dollars off in either direction is how you get a letter instead of a refund. Give it the real numbers, people say. Married filing jointly, income, deductions, access to the books. Shouldn't that be enough? The failure is not arithmetic. Two plus two still equals four. The failure is judgment. Classify the income, pick the deduction, apply the entity rule, know which safe harbor you actually qualify for. That is where it hallucinates with a straight face. It can nail the spreadsheet and still call a distribution a wage. That is the two thousand dollars error wearing a clean column of numbers. This is the part that blows people's minds. Turbo tax, tax act, H and R block, tax cut. People call them AI because they live on a screen. They are not AI. They are calculators with a questionnaire bolted on. They ask, you answer. The answer opens the next branch. Same input, same output every time. There is a guarantee behind the math because the math is a fixed path. If their engine is wrong, they pay. If you lied on the intake, you pay. A language model does not run that path. It predicts the next word. Same prompt can yield a different answer on the second try because the machine is sampling, not computing. Set the temperature to zero and you can force the same sentence twice. You also lock it onto a wrong pattern with more confidence. The randomness is a feature for conversation. It is a defect for a 1040. There is no native pipe from the books into a tax engine inside the chat. A connector can feed numbers. It does not turn the chat into the software those numbers belong in. So why can't a shop full of engineers bolt a real engine under the chat and be done? They can. Parts of that work are already happening. Specialist tutors, public talk about checking returns. The bottleneck is not IQ. A language model predicts what sounds right. Tax software computes what is right. Even the better models on public tax benches still miss a full return most of the time, often because they use a bracket percentage instead of the IRS table and let a small miss cascade across forms. The smart product is not a smarter guesser, it is a splitter. The model gathers and classifies. A deterministic engine calculates. That is a different product than a chatbot. It is also the one that can carry a guarantee. HR Block can guarantee the engine. They cannot guarantee your facts. Grok has no engine like that, so there is nothing to stand behind. The terms say it out loud. No liability for agentic action that touches money. The IRS does not care what the bot said. The AI told me is not reasonable cause. That is why there is no Grok accuracy guarantee. Not because the engineers cannot count, because they will not bet the company on a number they cannot reproduce the same way twice. Here is the real problem, and it is not the model. They call it a chat bot. People treat it like QuickBooks, payroll, and a CPA in one icon. The marketing talks like a one person company. The terms talk like a toy. Both are in writing. Most people only read the first one. In practice, the bot is a remote control. It presses buttons in software that already exists Twilio, Zapier, QuickBooks. The books still live in Intuit. The withholding still lives in the payroll app. The bot is the hands, you are still the brain. That is useful. It is not the same as one app that owns the math. Supervision is the tax you pay for that. You wanted to stop checking. The product is built so you cannot stop checking. Slow tool calls, extra confirmations, disclaimers. That slowness is not the model being dumb, it is the plumbing, and it keeps the liability on your side of the table. Covering their ass? Yes. Also accurate. Those two things can be true at the same time. Ask the model to do something clearly illegal and you hit a wall. That wall is real. Hard filter, consistent. Ask it whether your deduction survives an audit and the wall is gone. Soft prompt. A few thousand words of guidance the model can follow, bend or quietly skip. That asymmetry is not an accident. A refusal is cheap. It costs nothing and protects the brand. A guaranteed number is expensive every time the model is wrong, so the hard gate goes up where it is free and stays down where it costs money. People hear that and say the model should never be wrong. This architecture cannot do never. Sampling from probabilities always leaves a chance the next token is the wrong one. The only way to get never is to stop sampling and run code. That is turbotax. That is not the chat. If the only reason you ask the bot is that you do not know the answer, you cannot be the person who catches the miss. A wrong deduction looks like a right one on the page. That is the failure mode that matters. So what is honest? You can get a ballpark, you can get the questions a good CPA would ask. You can get the code sections in plain English, carried interest, opportunity zones, step up and basis. Those are not secret books, they are the tax code, public. You cannot get a promise of zero audit risk. No software sells that because the IRS does not sell that. What survives an audit is a defensible position, cited sections, documented facts, a paper trail that shows good faith, not the absence of a letter. Aggressive planning is legal, fraud is not. The line is whether you can defend the position with the code, not whether a bot sounded sure. Verifiable beats merely right. A number you can trace to a form beats a number that happened to land. That is the gap, not intelligence. Proof. Two times two is four, every time. Because the calculator is not guessing. Your quarterly estimate is not two times two. It is a stack of judgments. The chat is good at sounding like it finished the stack. It is not good at proving it. Use it to learn the map. Do not let it mail the check. This is the deep dive. I will see you next time.