The Making of the IRS: ‘Nobody Likes a Tax Collector’

Robert Goulder and Joseph Thorndike of Tax Notes explore tax collection throughout U.S. history, from tariffs to tax returns, and how the past has shaped Americans’ perception of tax.

**

Credits
Hosts: Robert Goulder, Joseph Thorndike
Executive Producer: Paige Jones
Producer: Jordan Parrish
Editors: Christopher Trigo, Carolyn Kuimelis

Transcript

Copy Transcript Copied

Joseph Thorndike: Nobody likes a tax collector.

Robert Goulder: No.

Joseph Thorndike: That's millennia of truth behind that.

Robert Goulder: Of course. Nobody likes paying taxes.

Joseph Thorndike: Nobody likes somebody else failing to pay their taxes, and we rely on the IRS to stop that from happening.

Robert Goulder: Hello and welcome to the podcast. I am your host, Bob Goulder, a contributing editor with Tax Notes. Joining me is my colleague, Joe Thorndike, the tax history guy over at Tax Notes. Joe runs our tax history project. That's where you can find all sorts of things, a timeline of major tax historical events and a fascinating catalog of all publicly available presidential tax returns.

Serious topic though today, Joe. How did we end up with the federal tax agency that we have today? It didn't just come out of nowhere. Now, I know some of the history: Right after the Constitution was ratified, they had the Treasury Department and they appointed this really interesting guy to run it, Alexander Hamilton. But if I'm correct, the IRS didn't come around until the 1950s. So that is a century and a half after Alexander Hamilton. So what was going on back then? Who was collecting taxes in the early American republic?

Joseph Thorndike: You're right. There was no Internal Revenue Service until 1953, but there were definitely federal tax collectors of one kind or another.

So to start at the beginning, the founders get together and they create this new plan for government and the Constitution. And the Constitution starts working, and they get out there and they pass a law. The second law ever passed by the new Congress was in fact a revenue bill. They had to actually fund this new government. And what do they do? They pass a tariff law because that's what people thought was going to be the main source of federal revenue. They're going to tax goods as they're imported in the U.S.

Someone has to collect those. So they create a Customs Service at the same time. And we still have a customs bureau today. And the Customs Service, along with the Coast Guard, is enforcing these new tariffs, these new taxes on imported goods. You need a Coast Guard because all this stuff is coming in by sea. And so you got to prevent smugglers. You got to have somebody out there in ships to stop them. But they also have these customs collectors in major cities, and that's our revenue service for a long time.

So then we skip ahead a few years. And someone like Alexander Hamilton and the other Federalist Party members, they think, "Well, we've got to make good use of all these powers to tax that the Constitution gave us. And the Constitution says we can use tariffs, and everyone thinks that's going to be mostly what we use, but they also say we can impose internal taxes." And this would be like an excise tax on a product, like, say, whiskey, or it might be a federal property tax because you can, in fact, at the federal level, tax land and houses the same way our localities do right now.

They give the federal government this power, and then they say, "OK, let's make good use of this. We're going to impose a few of these excise taxes." They impose one on whiskey famously, which eventually ends up irritating some farmers in western Pennsylvania. But they also impose a federal property tax in 1798. And they create a department of the federal government to collect this stuff. It's in the Treasury Department, but they appoint a commissioner of the revenue. The first one's name is Tench Coxe. And these people all work for him, and they're all around the country. They create revenue districts. I think there are like 14 originally. They create a whole bunch of revenue districts, and they're staffed with people who are out there assessing things, assessing how much tax people owe, and then collecting that tax. And this is the first real tax agency in the way we think of it. If the Internal Revenue Service is the organization that now collects internal taxes, this was the 1790s version of that.

And that lasted up until 1802 when the opponents of the federalists — they were called Democratic Republicans at the time, and it was really Thomas Jefferson and his friends — they hated the internal taxes. So when Jefferson gets elected president, a very quick thing that they do is they get rid of all the internal taxes. And while they're at it, they fire the commissioner of revenue and all the people who work for them.

Robert Goulder: It sounds like the external taxes, the tariffs, are tolerated a lot better than the internal ones.

Joseph Thorndike: I think that's right.

Robert Goulder: Is it because they were used to paying them when they were a colony?

Joseph Thorndike: Yeah. I mean, there are precedent for all these taxes, but I think that tariffs were generally regarded as sort of a reasonable, traditional, normal way for governments to raise their money. And so when they decided, how are we going to raise money for the new government? That was where they looked first because this was the stuff that was best tolerated. You wouldn't necessarily think that because throwing the tea into Boston Harbor, that was about an import tax, but it is just a fact that these tariffs were better tolerated.

They were also really easy to collect because there are only a certain number of places where things get imported into the colonies.

Robert Goulder: Right.

Joseph Thorndike: There are only a certain number of ports. So all you got to do is put a big customs house in that port, staff it up with a bunch of people. That's a lot easier than trying to collect things around the country. Now when they impose the whiskey tax, they got to go to the whiskey producers, right?

Robert Goulder: Yeah, everywhere.

Joseph Thorndike: There were a lot of them in western Pennsylvania. That was the frontier in those days. And they go out there, and they got to find the guys with the stills. They got to say, "How much are you producing? Show me your records. Show me your vats of whiskey," whatever it is, barrels, I guess, of whiskey. "And show me your books. Show me how much you've sold and everything." That's very invasive. People don't like that kind of thing. And for a lot of reasons, they felt that this was unfair. Why whiskey? Why not something else? Why are you picking on us? That sort of thing.

Robert Goulder: And it sounds like internal taxes could be subject to concealment, right? If you hide it out in the shed or the barn, maybe they won't find it. Whereas how are you going to conceal an import?

Joseph Thorndike: It's a lot harder to conceal an import because it's just coming into those few places. So you can get the tax police on the job pretty easily. It's a lot harder to do when you're walking through farms looking for people who are distilling whiskey.

Robert Goulder: All right.

Joseph Thorndike: It wasn't really all that well tolerated. And in fact, those farmers in western Pennsylvania revolted in the Whiskey Rebellion famously, lasts for a few years, and the federal government crushes it with an army.

Robert Goulder: Oppression.

Joseph Thorndike: Because that's what it takes. Ultimately, taxes are collected at the point of a gun at some level.

Robert Goulder: Get used to it.

Joseph Thorndike: Somewhere back there, there's a threat. We're going to make good on this. And that's what the federal government did. And we ended up still having a whiskey tax.

Robert Goulder: I guess it did wonders for compliance, having an army to come in and crush you.
Speaking of being crushed, we had the War of 1812. And if I understand history, the British troops sailed right up the Potomac, got off in Washington, D.C., burned down the Treasury Building. Did that have an effect on our revenue capacity or not so much?

Joseph Thorndike: Well, not so much burning all the buildings in D.C., but wars have a way of changing taxes, right?

Robert Goulder: How so?

Joseph Thorndike: So especially when you're relying heavily on a tariff, and at this point, at the start of the War of 1812, we're relying solely on tariffs, really, tariffs don't do very well in a war because, at least in those days, everything's coming in by sea. There are navies out there who are trying to intercept that stuff that's being shipped around. It disrupts trade and it reduces trade. And so tariffs, which are dependent on the volume of trade, they do less well during a war.

Then the second thing that happens in a war is you need a lot more money because you got to buy guns, you got to buy bullets, you got to buy cannons, all that sort of stuff. You got to pay soldiers. And they need a lot of extra money, and the tariff's underperforming at the same time. So what do they do?

Robert Goulder: So you need more revenue, and you're getting less of it from your traditional course.

Joseph Thorndike: Right. And just raising tariffs, well, that can sort of make it better, but it can't really fix the whole problem. So what happens? They look back at those internal taxes, the ones that were so unpopular before, and they're like, "Well, they may be unpopular, but we need them again." So in the War of 1812, they recreate a lot of those internal taxes. They impose a bunch of excises on things again. And they hire all those tax collectors back. There's a new commissioner of revenue. He hires new supervisors and new assessors of revenue all around the country.

So they rebuild this, this bureaucracy of tax collection, and that bureaucracy successfully collects these wartime taxes. Meanwhile, those tariffs are still coming in, right? The customs guys are getting them. But they rebuild the internal tax system, and they hold onto it until the end of the war. And then what do you think they do?

Robert Goulder: Then they fire them all.

Joseph Thorndike: They fire them all.

Robert Goulder: There is a pattern here.

Joseph Thorndike: Coming out of the War of 1812, we get rid of all the internal taxes, and we go back to just relying on tariffs. And that's all we do until the Civil War. We have no federal tax collection agency, internal tax collection agency, until the Civil War.

Robert Goulder: All right. Civil War, you got to pay for the Union Army. A lot depends on it. And the same scenario, you've got tariffs. But what?

Joseph Thorndike: And the tariffs are not doing so well. And there's been a lot of argument about tariffs all these years anyway.

Robert Goulder: Yeah. The intervening years.

Joseph Thorndike: Should they be high? Should they be low? And they're at a point where they're kind of low going into these war years. They raise tariffs quickly as soon as the war begins, but it's not enough. And so Congress does what Congress does in these situations. They say, "Well, what are we going to do? We got to raise more money."

Robert Goulder: And what do they come up with?

Joseph Thorndike: They come up with a whole bunch of things. So they come up with excise taxes again. And not just whiskey and tobacco this time, but a whole range of things. They have hundreds of excise levies. There's so many, it's almost like a national sales tax, but it's levied one-on-one.

Robert Goulder: Wow.

Joseph Thorndike: And so they try that. And then they say, "And hey, what about that old property tax that we used a few times in the beginning of this century? We could use one of those again. We're going to tax land and buildings on the land, that sort of thing." So that's another internal tax.

And then in a really bold move of tax innovation, they say, "And let's try an income tax. We've never had one of those, but those British have been using them for a while. So let's us try an income tax."

Robert Goulder: So that's when it started?

Joseph Thorndike: It starts —

Robert Goulder: 1860s?

Joseph Thorndike: 1861, Congress passes a law and says, "Let's have an income tax." And then what happens? Nobody does anything to collect it.

Robert Goulder: Do they know how?

Joseph Thorndike: Well, we don't really have a tax collection agency. Congress has authorized that kind of an agency. Treasury doesn't hire anybody. So it goes for a whole year and nobody collects anything. And then in 1862, they actually make it happen for real. They pass another income tax law. This time they say, "We're going to create a Bureau of Internal Revenue that's going to collect this stuff. It's going to have a commissioner of internal revenue." His name was George Boutwell, the very first one. He was a politician from Massachusetts, and he takes over this new agency. He has like four people working for him originally.

They hire — Again, you got, to collect these internal taxes, you got to have people all around the country. So they hire a whole bunch of people in revenue districts around the nation. They hire assessors. These are the people who go and look at your books and say, "Oh, here's how much money I think you made and how much I think you're going to owe in income taxes," for instance. Then they hire collectors who knock on your door and say, "OK, well, here's what we're planning on collecting from you," and then actually do the collecting. This is a lot of people. It's a lot of staff.

And so this Bureau of Internal Revenue scales up really quickly. And under the press of a war, you make that stuff happen, right? They very quickly build this Bureau of Internal Revenue nationwide in the Union, all throughout the Union.

Robert Goulder: And nobody's calling it the IRS.

Joseph Thorndike: No one's calling it —

Robert Goulder: The Bureau —

Joseph Thorndike: They're calling it the Bureau of Internal Revenue, not the Internal Revenue Service. It's just a little tweak.

Robert Goulder: Yeah.

Joseph Thorndike: And they won't get to that service thing until the 1950s.

Robert Goulder: All right. So then the war ends, what happens to this income tax? I'm guessing, based on our prior discussion, that as soon as the war is over, people are realizing those internal taxes are kind of unpopular. Do they want to go back to tariffs?

Joseph Thorndike: So they do go back to tariffs, and they do get rid of the income tax. In 1872 Congress just lets it expire. It's true. The excise taxes to some degree, but the income tax in particular, are really not popular. It's because of all that intrusive, they used to call it inquisitorial enforcement, where if you think it's bad enforcing a whiskey tax where you got to go look through people's barns to look for stills, it's much worse with an income tax because you're looking for dollars and cents, you're looking for numbers on a page to figure out how much people earned. And that can be really hard to find and really involves tearing apart their financial records. And people don't like that. That was considered inquisitorial. Worse — to help with that task of trying to figure out what people's income was, they basically enlist everyone in the country to be a tax collector, and they say, "What we're going to do is we're going to take what you owe, Bob, you owe $1,000 and we're going to put that up on the courthouse door. We're going to nail a list of everybody's taxes due."

Robert Goulder: In public.

Joseph Thorndike: In public. "And we're going to ask the newspapers to publish it." So all your neighbors will be able to look at it, and then your neighbors will be like, "Wait a minute, Bob makes way more money than that. I know. He's got a new plow. There's no way he could have that if he was only making this money."

Robert Goulder: Neighbor versus neighbor.

Joseph Thorndike: So the idea was that your neighbors would rat you out to the IRS — or the Bureau of Internal Revenue, excuse me — and say, "This can't be right. You should take a harder look at this person."

So this is making everybody into a tattletale. Well, you can imagine that's a little irritating for a lot of people. That made the tax unpopular, along with just having to actually pay it made it unpopular. And so in 1872, they let it go. But they don't, this time, get rid of the federal tax agency. They say, "You guys are still going to need to collect a couple of internal taxes, mostly just taxes on alcohol and tobacco," because these are the excise taxes that make the most money. They're also well tolerated. Now, by that point, they're sort of quasi-traditional, and people think, "We can justify this as a tax on something that's optional, maybe even a little sinful." And so they hold onto the whiskey, to the alcohol and the tobacco taxes. And then the IRS — the Bureau of Internal Revenue, I make this mistake all the time too — tends to focus their efforts on that, and especially on collecting alcohol taxes, which are still burdensome. They're still tracking people through the mountains of West Virginia looking for stills, and that occupies a lot of time.

Robert Goulder: Better than looking at [Form] 1040s, I guess.

Joseph Thorndike: Yeah, at least for the time being.

Robert Goulder: So the income tax goes away, but it sort of comes back, right? You get the progressive era, you get the 1890s, and they bring back the income tax. And it's declared unconstitutional the following year?

Joseph Thorndike: Yeah. So a lot of people sort of miss the income tax when it's gone, especially the people who weren't paying it. And they are agitating to bring it back in the 1880s and the 1890s. In 1894 they actually convinced Congress, it's like a tax swap, essentially. What they're hoping for is that we'll raise some more money from income taxes. That'll allow us to reduce the tariff, which a lot of people didn't like because they saw it as a consumption tax that was regressive. So let's have this progressive income tax.

They bring it back in 1894, and then the Supreme Court says, "No, no, unconstitutional. Even though we have previously said that the Civil War income tax was fine, we now find that this one is not fine. It's unconstitutional; you're going to throw out the whole thing." So they throw out the income tax in 1895. They've already received a lot of tax returns to pay that tax. They burn the tax returns, which I love.

Robert Goulder: Yeah.

Joseph Thorndike: And it takes a while before the income tax reappears again, because the movement doesn't end, right? There's still all those agitated people and they want to —

Robert Goulder: But they have to amend the Constitution, right?

Joseph Thorndike: Well, they might've been able to do it without that, but that was the clearest way to shut the Supreme Court up, right?

Robert Goulder: Yeah.

Joseph Thorndike: It's just pass a constitutional amendment that says, "Nope, income taxes are good." And they do that. Even before that, they managed to impose a corporate income tax with some fancy legal framing that passes constitutional muster. And the Bureau of Internal Revenue creates a new division to collect this corporate income tax, the returns of which incidentally were public records that could be inspected. Four years later, they create the individual income tax, and the Bureau of Internal Revenue creates a new division to take care of that. And we have now returned. We've opened a new chapter in the fiscal history of the country with this income tax at its center, still collecting tariffs, still collecting excises, but the Bureau of Internal Revenue has a whole new job collecting all of these new income taxes of various kinds.

Robert Goulder: All right. Now I wanted to talk to you about the popular mythology of the tax collectors in, say, the 1920s. There's this idea that they're running around carrying machine guns. They're almost like a quasi-military force, because they're going after bootleggers and folks. But why were tax collectors involved in enforcing prohibition in the first place? Why isn't that for the FBI?

Joseph Thorndike: Yeah. Well, so I think the answer to that is that it was a kind of obvious role for the Bureau of Internal Revenue because they were already doing a lot of enforcement around excise taxes on alcohol. So literally they were used to carrying guns around looking for illegal alcohol operations, and that's essentially what the prohibition enforcement is about. So when it comes time to pass the enabling laws, which actually enforce the prohibition amendment, they're going to turn to the Bureau of Internal Revenue and say, "You guys do the enforcement on this." And they did have help. It was not exclusive to them, but they had primary responsibility for enforcing prohibition. And it became famous, as you say. The most famous tax avoider of all times was Al Capone.

Robert Goulder: Yeah.

Joseph Thorndike: And he was convicted of tax evasion because he was selling all of this alcohol and yet not paying taxes on the income that he made from it. Because that's the deal about the income tax, even if the source of your income is illegal, you still owe taxes on it.

Robert Goulder: So Al Capone could go around and murder people in Chicago, but he went to jail for not paying taxes.

Joseph Thorndike: Right. Now the FBI might come for you if you're murdering people, but if you're not paying taxes on your illegal income, the Bureau of Internal Revenue will come for you. And that's how he actually went to jail because it was easier to prove at the time, and also a federal crime. So you could have the feds enforce it clearly instead of a lot of murders, which are enforced at the local level. So it's a weird role for the Bureau of Internal Revenue for these tax collectors, but really one of their most famous.

Robert Goulder: Well, part of it is just that it sounds like they knew who the perpetrators were.

Joseph Thorndike: Yes.

Robert Goulder: There you go. Familiarity. By the time of the 1930s, we get the Great Depression. By the 1940s, we get FDR in the White House. During that time, it seemed like there was an expansion in both the scope and the mandate of the federal government. It was being asked to do more and more. How did the Bureau of Internal Revenue keep pace with that?

Joseph Thorndike: All right. So there is actually an important innovation in the 1930s when we introduced Social —

Robert Goulder: A tax innovation?

Joseph Thorndike: A tax innovation or a tax collection innovation even more. Congress creates the Social Security system, these old-age pensions, and they fund it with a special tax. That special tax, they decide, is going to be collected directly from paychecks. We're not going to file — You and I don't file Social Security tax returns. Nobody does. Instead, our employers take the money right out of our paycheck and send it to the federal government on our behalf. That's called withholding.

Robert Goulder: Yeah.

Joseph Thorndike: Now, there had been withholding before. They had done some withholding during the Civil War even. And again, in the early years of the income tax and the 19-teens, they did a little bit of withholding. Didn't really stick because those financial intermediaries, the banks or the employers doing — they don't really like that job. They don't like being deputized as tax collectors. But in 1935, they establish a Social Security tax, and they need to collect tax from a huge number of new people very quickly. And they're like, the only really decent way to do this is with a withholding mechanism. So that's the big thing that happens in 1935.

Fast-forward a few years to World War II. The income tax has expanded dramatically. Again, a war, need lots of revenue. Are the old ways holding up? Not really. So what they do that time is instead of creating a new tax, they say, "We're going to take the income tax," which used to be paid by like 5 to 10 percent of the population —

Robert Goulder: A small percentage of the population.

Joseph Thorndike: Rich people.

Robert Goulder: Yeah.

Joseph Thorndike: And they say, "For the first time, we're going to ask nonrich people to pay income taxes as well. We're going to add millions of new people to the role." The number of income tax payers went from like 7 million just before World War II started to like 42 million at the end of the war. So dramatic increase.

Robert Goulder: Huge expansion.

Joseph Thorndike: And how are we going to get these people to pay? Same sort of thinking as went into Social Security. "If we ask them all to file returns about this and figure it all out on their own, there are going to be logistical problems with that, so we're going to withhold this money from them." Now, they are going to have to file returns anyway, but they're not going to have to save up all their money over the course of the year and then send in a great big check. Instead, we're going to use those employers, again, to keep that money out of their paychecks and then just send the money directly to the federal government for us. So they introduce withholding in 1943 for these millions of new taxpayers, and the income tax meanwhile is being transformed, as they often say, from a class tax to a mass tax.

Robert Goulder: The class being the affluent.

Joseph Thorndike: Yeah.

Robert Goulder: And again, mass being everyone.

Joseph Thorndike: There were a couple of tax professors who at the time actually described the change saying the income tax changed its morning coat for overalls.

Robert Goulder: Well, that's an incredible efficiency when you think about just the number of taxpayers they have to deal with.

Joseph Thorndike: Probably couldn't have been done without it, in fact.

Robert Goulder: Yes.

Joseph Thorndike: Because the fear was this: They're going to have all these millions of new people have to pay, and they're not going to remember to save money over the course of the year. And at the end of the year, they're going to file a return, and they'll be like, "Oh my goodness."

Robert Goulder: Yeah, "I haven't got it."

Joseph Thorndike: "I haven't got this money. What am I going to do?" And the Treasury Department thought this was a real problem too. And the Treasury secretary, Henry Morgenthau, looks at his aides at one meeting and he says, "What are we going to do if we have to arrest 5 million people for not paying their taxes?"

Robert Goulder: Can't do it.

Joseph Thorndike: So withholding was the answer to that.

Robert Goulder: Finally, after World War II, we get to the 1950s, the transition from Roosevelt to Truman to Eisenhower. And finally — what was it, 1952, '53, thereabouts — we finally get the IRS. And was it just a rebranding or?

Joseph Thorndike: No, it's a restructuring. So there is a great big scandal at the IRS in the early 1950s.

Robert Goulder: Oh, I love a good scandal.

Joseph Thorndike: Yeah.

Robert Goulder: Tell me.

Joseph Thorndike: Sort of comes to light in California, and then the investigation moves to Congress, and Congress starts looking hard at the agency. And what they discover is that there's a ton of corruption, that people are taking payoffs from taxpayers. So it's like blatant corruption. And they put a bunch of people on trial. They fire a bunch. They get some resignations, but they also put a few people on trial, including a former commissioner.

Robert Goulder: Ooh.

Joseph Thorndike: And so this is a big deal, as you might imagine, finding wholesale corruption. Those collectors of internal revenue scattered all around the country turn out to be a big problem. They're presidentially appointed. They think they're pretty cool and consequently don't really feel like they're answerable to many people. They turn out to be a problem. So they say, "We got to rebuild the agency. We got to reconstruct it in some way where this won't happen again." They get rid of all those collectors, and a lot of those field staff who are presidentially appointed, they get rid of them. And they simultaneously sort of like centralize and decentralize the agency, but they try to just professionalize the whole thing and reorganize it in a way that make it less prone to sort of like petty political corruption in these collectors' offices.

And so this big reconstruction, this redesign of the agency gets pushed through in the 1950s. It's really bipartisan. And as a part of that, they decide to rename the agency the Internal Revenue Service. So we finally get the name. But it's worth recalling that that is the same agency that Congress established in 1862. So the IRS, as we know it today, does not date from the '50s. It dates from the 1860s. It's just that we decided to change its name partway through.

Robert Goulder: That explains a lot. So moving on to the 1970s, the Nixon era. There was a whole lot going on with Nixon, but part of it that I'm interested [in] here is what he was doing with the IRS.

Joseph Thorndike: Yeah.

Robert Goulder: Not for his personal taxes — that's his own shenanigan.

Joseph Thorndike: Personal taxes are part of the story, actually.

Robert Goulder: All right, explain.

Joseph Thorndike: All right. So he's doing a lot of things with the IRS. Most famously, he created an "enemies list," or his aides created an enemies list. And the idea was that they were going to use the Internal Revenue Service to harass and make life difficult, and possibly prosecute, enemies of the administration.

Robert Goulder: He just picked people he doesn't like and have them audited?

Joseph Thorndike: Take a hard look at them, see if we can find problems. And if we find problems, then we can go after them, and we can at least threaten them. This is a weaponization of the tax collection process in the U.S. And it's quite self-conscious; it's quite brazen within the White House. The IRS does resist this to some degree, but it raises the specter of corruption and of misuse and malfeasance at the IRS.

And it does come to light during the Watergate investigation. When they're off investigating this break-in, they discover all this stuff about ways in which the president and his aides had tried to misuse the IRS. So that gets Congress worried.

At the same time — this is his personal stuff — Nixon is not really following the letter of the law or the spirit of law, or really any part of the law when it comes to his own taxes. He makes a bunch of claims on tax returns that are really unsupportable and in some cases involve forged documents, and he gets in a lot of trouble for his personal tax returns. Turns out he ends up owing like half a million dollars in back taxes to the federal government, which is a lot of money in 1973.

Robert Goulder: Yeah.

Joseph Thorndike: Nixon has a lot of these problems. And one of the things that they discover is that the IRS audits the president in the middle of this and they're like, "Oh, all good. Thumbs up. Your returns look terrific." Then somebody who doesn't work for the IRS actually takes a look, the Joint Committee on Internal Revenue Taxation; these experts take a look, and they're like, "No, actually he owes half a million dollars."

Robert Goulder: Wow.

Joseph Thorndike: So the IRS clearly dropped the ball when it came to auditing their own boss. What this really spawns is certain changes in the way the IRS operates, but even more, a change to the way we handle taxpayer information. They decide to crank down on the protection of taxpayer information and they won't let — And this is to prevent that weaponization of the IRS against enemies. So if you can't share tax return data with the White House, well, then the White House won't really be able to weaponize very well. They do make those changes in '76.

Robert Goulder: Now there's another, I don't know if scandal is the right word, but when we get to the 1990s — I'm old enough to remember this — we had the Roth hearings. There was all sorts of congressional oversight. There were these allegations about IRS agents engaging in tactics that maybe weren't legitimate. There were stories of witnesses coming in to testify with voice-altering equipment and masks over their face, or black curtains and stuff to shield their anonymity. What was going on in the '90s? Was there really a scandal?

Joseph Thorndike: So I think the answer is yes, but maybe not everything it was made out to be, right? That the IRS was overzealous in certain instances and running roughshod over taxpayers and their rights at times. There were real complaints to be made, and they were certainly being made. It's also true that attacking tax collectors is generally pretty good politics; especially, this is becoming more and more true in the 1990s. And so there were a number of politicians who, I think, realized they could score political points if they went after the agency hard. So they did. They convened a lot of hearings. They were very theatrical, like you say, like voice-changing devices and whatever to —

Robert Goulder: It was quite dramatic.

Joseph Thorndike: Yeah, to make it look exciting. So they do hold these hearings, and they do develop another reconstruction plan for the IRS, and the first one since the 1950s. And these changes are the biggest since the 1950s. So we have certain signposts in the development of the agency, and one would be its creation in 1862. Another would probably be that reform in the 1950s when they rebuild it. And then this is the next one. And it is a pretty significant change. They put a lot more emphasis on taxpayer rights, and they create institutions to try to protect those rights, including the Office of the Taxpayer Advocate, which is a new position at the IRS dedicated to looking at the rights of individual taxpayers.

Robert Goulder: An important office.

Joseph Thorndike: So I think that this is an important moment, and it is, I think, partly a real one and partly a manufactured one. It's both.

Robert Goulder: OK, Joe, final question. I want to ask you about today, 21st century. When I think about the IRS, the words that come to mind are resource constrained. Frankly, for decades now, it seems that Congress has been intentionally defunding the IRS. It just doesn't have the budgetary resources that it arguably needs to do its job. How is the agency coping with that?

Joseph Thorndike: Well, I think not well. So since the '90s, the IRS has been struggling to, I think, get the funding that it needs. And it's had intermittent moments where it's had its funding boosted, but more generally, its funding has been on decline, especially when you're accounting for the fact that the U.S. population is increasing over this period and inflation is rising. So the budgets are not being increased to keep pace with that. That means that the IRS has fewer and fewer employees trying to do the same job, trying to do new jobs. A lot of the time, we pass new benefits in Congress, and we ask the IRS to administer them. And the agency has not had the resources to do that while also keeping up with the regular tax enforcement that they're supposed to do, looking over people's tax returns and making sure that they're not cheating or making innocent mistakes, but mistakes all the same. The agency hasn't had the resources to do that. That was a long process of like two decades, at least two decades.

And then there's an effort in the Inflation Reduction Act in the Biden administration to breathe new life into the agency by giving it more adequate funding. And they come up with something like $80 billion —

Robert Goulder: $80 billion, yeah.

Joseph Thorndike: — of new funding for the agency, phased in over a bunch of years. And the idea is we're going to let the agency catch up so it can do a better job with its core mission, especially of assisting taxpayers in filing their returns and enforcing the tax laws that are on the books. And that was a pretty reasonable idea for a resource-constrained agency, but you can't constrain the works of a future Congress. And so the problem was is that that $80 billion sounded like a great idea to the Democrats who were running Congress at the time. Sounds like a much less good idea to the Republicans who are running Congress now. And so a lot of that $80 billion has been clawed back, has been reduced. And the agency's also had just mass firings along with a lot of other parts of the federal government. So it's even —

Robert Goulder: Are you talking about DOGE [the Department of Government Efficiency]?

Joseph Thorndike: Yes. And whatever replaces DOGE. But in general, the second Trump administration's effort to shrink the federal workforce has definitely hit the IRS. Meanwhile, they're not going to get the money that they were hoping for, that they thought that they would get from the Inflation —

Robert Goulder: But what if they can't do their job?

Joseph Thorndike: They will find a way to do some portion of their job, but if they don't do their job well, then we don't enforce our taxes well. And the question is, do people hate the IRS so much that they don't care if the tax laws are enforced well? Or do people get angry when they say, "Hey, the people most likely to cheat, who are going to make the most money out of this, are going to be rich people who are going to try to avoid taxes, who can afford to hire all those expensive lawyers and accountants to help them legally avoid taxes, or at least not transparently illegally avoid taxes, but that wouldn't pass muster if the IRS were really looking hard at it." And I think that the politics of this are not clear yet. They haven't played out, because I do think it's true, nobody likes a tax collector.

Robert Goulder: No.

Joseph Thorndike: That's a millennia of truth behind that.

Robert Goulder: Nobody likes paying taxes.

Joseph Thorndike: But nobody likes somebody else failing to pay their taxes. And we rely on the IRS to stop that from happening. So I think both those are powerful political things, and we might well — But we're still waiting to see how that plays out in contemporary politics.

Robert Goulder: So you can't really predict what the IRS is going to look like, say, in another 25 years.

Joseph Thorndike: I can't. I can predict that we will not be done arguing about it.

Robert Goulder: There you go. And there you have it. Thank you so much for tuning in. And I wanted to add that this series is being produced by Jordan Parrish. It is edited by Chris Trigo and Carolyn Kuimelis. And the executive producer is Paige Jones. Thank you and see you next time.