Who Gets to See Your Taxes? The Evolution of Taxpayer Privacy

Robert Goulder and Joseph Thorndike of Tax Notes discuss how public opinion has shaped taxpayer privacy laws from the early days of the income tax to immigration enforcement today.

**

Credits
Hosts: Robert Goulder, Joseph Thorndike
Executive Producer: Paige Jones
Producer: Jordan Parrish
Editors: Christopher Trigo, Carolyn Kuimelis

Transcript

SPEAKER_00 0:00

If you start sharing this information, people are going to get squirrely about that, and they're not going to be so willing to come forward with the accurate information. So, again, like every time you compromise privacy at all, every anytime you start sharing tax data inside the government or outside the government, people, taxpayers get a little bit nervous.

SPEAKER_01 0:24

Hello, and welcome to the podcast. I'm your host, Bob Goulder, contributing editor with Tax Notes. I'm joined by Joe Thorndike, the tax history guy at Tax Notes, and he runs our tax history project, where among other things you can find a vast inventory of publicly available presidential tax returns. Joe, I want to talk about taxpayer privacy, which is an issue that I know you've written about a lot. And when I go through your past catalogs of things you've written about this, you talk about an implied bargain between the individual in the state. Can you elaborate on that? What is this bargain?

SPEAKER_00 1:06

All right. So you're right. I do love taxpayer privacy as an issue. And it is a bargain. And like most bargains, it gets renegotiated once in a while. But the basic question is how much information does the government need about you in order to deal with your taxes, to get you to, you know, fill out to collect taxes from you. And what kind of privacy do they owe you in exchange for that? What can they not reveal about that? So the government's going to take in all this information because you actually need a lot of information for an income tax. It's just a necessity. But the government has to make some sort of promise. If they don't make that kind of promise, then taxpayers aren't going to be so forthcoming with all their information, right? Because we sometimes call the system of voluntary assessment. All the time. It's like quasi-voluntary these days, you know. But it you do need to provide the government information that they really can't get any other place.

SPEAKER_01 1:56

It's intrusive.

SPEAKER_00 1:56

Right. So there's the deal. It's like, okay, we're going to ask you for a lot, but we're going to promise you that we will protect that from public disclosure. That's the bargain. And it's been, as I said, renegotiated at different times.

SPEAKER_01 2:07

Okay. So they concede that it is necessarily intrusive.

SPEAKER_00 2:11

Yes. I think I think that's just a known fact about the income tax and that people feel like it's worth it because the income tax is arguably more fair than most alternatives and that sort of thing. And it's effective. It raises a lot of money. So, you know, necessary evil, that sort of thing.

SPEAKER_01 2:27

All right. What is the origin story of this bargain? I mean, how long has it been going on for?

SPEAKER_00 2:32

All right. So it really begins with the income tax itself, with the federal income tax, which first appears in the 1860s. We've never tried it out in this country before. And they don't have like an IRS to collect it yet. Like they're going to build the whole thing from the from the ground up. And in 1862 is when it really gets rolling, and they and they implement this income tax. And uh they, you know, they require people, like today, people to had to file returns. And the idea was, and this was a this was a novel concept and not one that we we still use today, but the idea was we were gonna make these returns public record. We were gonna allow everyone to see the returns filed by everyone else.

SPEAKER_01 3:12

Public as in full disclosure.

SPEAKER_00 3:13

Right. Now I call this sort of uh a crowdsourced auditing technique, right? So the idea is it's hard for this, and and back then it's called the Bureau of Internal Revenue, not the IRS. It's hard for the Bureau to actually figure out accurately how much um people are earning in a given year. They're relying on people to give them this personal information. That's the deal, that's the bargain, right? But we didn't, we don't really know. Is are they being honest? Are they being truthful? Now, our promise, oh, we'll keep it secret. Maybe that means that they'll be inclined to be more truthful, and that is the thought. But they might lie anyway, just because they don't want to pay taxes.

SPEAKER_01 3:49

Self-interest.

SPEAKER_00 3:50

So the idea is we're gonna actually not protect this information so well. We're gonna actually make it public so that your neighbors can rat you out if you're cheating.

SPEAKER_01 3:59

You're encouraging people to tattle.

SPEAKER_00 4:01

Yes. So for the sake of enforcement. The idea is that uh Bob is gonna report$1,000 in income and his next door neighbor is gonna say, that's crazy. He made he definitely made more than that. You should check up on it. What we're doing is sort of appointing everybody as a tax collector themselves.

SPEAKER_01 4:15

We are all deputized tax collectors.

SPEAKER_00 4:17

So that's that's this crowdsourced audit. This is a moment when taxpayer privacy is not highly valued, right? We're asking for all of this information and we're making it publicly available. We're putting it in the newspaper. It was published in local newspapers. Uh, we're also having it posted on like courthouse doors or you know, prominent places around town. And and that's just an enforcement technique.

SPEAKER_01 4:38

It was not super popular. Imagine that.

SPEAKER_00 4:42

So people obviously don't like paying taxes, and and some people didn't like the income tax because of that. But other people really objected to all of this public disclosure. Now, this tax is small in the 1860s. It's only paid by, you know, not more than like five or 10% of the population. But those people, those taxpayers, resent the tax itself, and they resent this public disclosure. Now there's a war on it, and it's actually a war for the survival of the union. This is a high deals. So it's pretty well tolerated while there's while there's a war, this privacy invasion that's going on. But as soon as the war is over, people are complaining about it all the time. By the end of the 1860s, tax is still in place. So's the publicity, but people are getting really restless about it. And Congress listens. It starts to scale back the publicity, says, oh, can't use newspapers anymore, don't want to see it in the newspaper. And then a couple years after that, they say, ah, let's just get rid of the whole tax. So I I do think that the blowback from that publicity played an important role in ending the income tax after a sort of decade-long trial.

SPEAKER_01 5:44

So then the Civil War ends and the income tax goes away, but it doesn't go away for long because eventually in the late 19th century, you get the progressive era and you actually have the income tax come back at some point in 1894, and then later after after the uh 16th Amendment. When the income tax comes back, is that implied bargain more or less the same, or has it been renegotiated as you said before?

SPEAKER_00 6:10

Well, they kind of picked up where the argument left off in 1872, which was with a lot of interest in privacy and like bolstering the privacy. So when they bring it back in 1894, things are pretty locked down. They're going to make it private. And then that tax never takes effect, right? The Supreme Court finds it to be unconstitutional, so it's never, uh, they never do anything with it. Notably, there had already been returns filed for that tax, and the treasury burned them after the Supreme Court decision. So again, that's like a statement about privacy, right? Like we're these things are private and we're not going to need them anymore.

SPEAKER_01 6:42

So let's they didn't want them to fall into the wrong hands. That's a pretty big swing from we're going to post it on the door of the merit document.

SPEAKER_00 6:51

Because the advocates of disclosure are still around. Now, the other reason you might want to disclose information is that you think it's just good for the country, right? It's good information. People should know how rich rich people are. Transparency. Yeah, I mean, you know, not everyone feels like that's transparency, but transparency is a positive spin on it. But yes, so the the advocates of uh the income tax, many of them think let's disclose some of this information about how much individuals are making or more to the point, how much corporations are making. So the first income tax that really sticks in the 20th century is a corporate income tax in 1909. And what they decide at that point is that these returns filed by corporations will be public records, open to inspection. That's interesting, right? Like they're they're thinking like we can regulate corporations because at that time, kind of like today, a lot of people are running around saying corporations are out of control. We got too powerful. Yeah. And they think, oh, I all right, we're gonna try to regulate them. And one way to do that is to get some information out about how much money they're making and what their operations look like. And the tax returns provide a glimpse into that. Not by any means a complete picture, but a little bit of information. So they think, let's do this uh and let's make them public record. Interestingly, they don't give the Treasury Department any money to make those things public, right? You got to have some like clerk somewhere stamping on like things, yeah, letting people through the door. They don't pay for any of that, Congress. So they don't actually have any public returns because no one can look at them. And the next year, they actually rethink it a little. So again, see, we're like we're renegotiating as we go. And the next year they say, okay, well, here's some money to make them public. However, you can only make them public if the president says it's okay. So they delegate some power to the president and they could just say, as they had the year before, all these things are public record, open to look at. And then they say, actually, we're gonna let the president regulate that. So they they do this partial handoff. Congress doesn't really want to be in the business of saying what's a good level of disclosure and what's too much. So they ask the president to do it for them. And you end up with like a weird sort of compromise arrangement where you have tax returns which are technically public records, but are in practice not available for public inspection because the president never says okay. The president never allows it. Uh, there is a limited amount of that, but not much. So I I talk about this as being like a a weird part of the bargain where, again, technically public, practically speaking, private. That's what the compromise they agree on in 1910. Three years later, we get the regular income tax, the one we think of as an income tax that we all pay. And they go with the same bargain. These things are public records. And that's that's an important phrase because it remains in the law until the mid-1970s that these are public records, at least theoretically available for public inspection, but then practically speaking, locked down by the executive branch so that no one can look at them. That's the compromise they go with in 1913, also.

SPEAKER_01 9:49

There's a bit of nuance there, Joe. There are public documents, but no one can see them. They're public nuance. Public and private. Now, so far you've been talking about public access. There's another dimension to this that I want to explore. There's this term that you've used called a generalized governmental asset. This is about other parts of the government looking at your tax returns as opposed to your neighbors and so forth.

SPEAKER_00 10:12

What are the issues there? As far as I know, that term arises in sort of the 1970s, but it's used to describe these bargains that are made in the 1920s. So, yeah, here's the thing tax returns, kind of useful, right? We might be able to use them for things other than collecting taxes or enforcing the tax laws. We might want to use them to track down criminals because it has information about where they might be, you know, where their businesses are located, and that sort of thing. So the Justice Department might want to have access to these. Lots of places in the government want uh access to tax returns because the data is helpful for them. Sometimes just for collecting statistics, it's useful. So there is this thought, hey, well, as long as the Bureau of Internal Revenue has all this stuff, the Treasury Department, which is part of the Treasury Department, has all this stuff. How about you share it with us over at Justice or us over in, say, the Agriculture Department or anywhere else in the federal government? And and initially they don't do a lot of sharing with individual returns. They do a lot of sharing with um corporate returns. And then by 1920, it's it's actually sort of evened out and they start to share more and more, including the individual returns around the government. And later, looking back on this period, what uh this this one investigating committee said was that they start to treat tax returns like a generalized governmental asset, something that the government owns that's useful and should therefore be used throughout the government. For more than just collecting tax. More than just collecting tax. Any reason they want. Yeah, or no reason at all. So you might add in certain circumstances you might need, they might need to justify why do you need to see this? But actually, as time goes on and the decades pass, the justifications are more perfunctory and and the sharing is more and more common. And pretty soon it's just, you know, tax returns are going everywhere. They're being shared with state and local governments, and they're also being shared with all sorts of you know agencies within the federal government. So again, is that wrong? I mean, we're all in the same team, right? And and there is a thought that, like, why would we withhold that information? You know, why would one part of the government withhold that from another part of the government when we're all supposed to be working for the people? So there's a, you know, it's a it's a good government and an efficiency argument, but a lot of people think if you start sharing this information, people are going to get squirrely about that and they're not gonna be so willing to come forward with the accurate information. So again, like every time you compromise privacy at all, every anytime you start sharing tax data inside the government or outside the government, people, taxpayers get a little bit nervous and and are maybe less truthful. And we do rely on the the truthfulness of taxpayers in order to make the income tax work less and less as the years go by, but but even today we still depend on like small business people to report their information accurately.

SPEAKER_01 12:57

Yeah, it's a voluntary compliance system nominally.

SPEAKER_00 13:00

So privacy gets in the way of that potentially.

SPEAKER_01 13:03

Okay, now there's got to be pushback, right? As for all the reasons that you just mentioned, there's something called a pink slip that I've read about in your articles. And it's not the kind of one where they give you when they're they're sacking you. In the tax context, what is a pink slip?

SPEAKER_00 13:16

Well, actually, even before that pink slip is in 1934, but a decade before that, they renegotiate that bargain again, right? So we're in this period where it's public yet private, yet shared within the government. That's the deal. In 1924, they're like, hey, let's go back to that Civil War thing. Let's try publicizing this stuff to regular people out there. And they pass as part of the Revenue Act in 24, they say, we're gonna publish information about taxpayers. We're gonna tell you who they are, how much they earned, and what they owed in taxes. And that information is gonna be publicly available and ultimately published in newspapers, among other things, right? It's it's available for people to look at. This is a crazy experiment. And as you can imagine, newspapers thought this was pretty cool. It was made for great copy, right? And so they would run long lists of people who were like local taxpayers and and what they were, what they had paid that year. So I just looked it up the other day. The New York Times runs this thing in 1925. It says Al Jolson, the comedian, right? He paid$33,000 in taxes that year. The widow of Andrew Carnegie paid$61,000. Uh, Rudyard Kipling, the author, uh paid on his royalties almost$5,000. And even had the income for Chief Justice Taft of the Supreme Court, who paid uh$1,723. So you can see, like, there's a voyeuristic quality to that. Oh, it's out there in the open. Right. People like this. I think it was the manager of the giants. They had his in there. Like it's so they would publish these things based on, you know, are they big incomes or really are they just sort of interesting incomes? All right, so that's 1924. Within two years, Congress is like, oh, big mistake. That was a bad idea, and they repeal it and they get rid of the um the publicity element in the 24 law. Ten years after that, in 1934, they try it again. Congress passes a law that says you're gonna file uh a little pink slip, literally a piece of pink paper with your return. It's gonna say who you are, what you paid, how much you earned, that sort of thing. The pink slip never even gets out the door. Before it's actually implemented, uh, there's this huge lobbying campaign against it. People are saying, oh, once that information gets out, kidnappers are gonna use that and thieves will use it to defraud rich people. Uh whatever the reason, Congress rethinks it and actually repeals it before it takes effect. So there are two moments where Congress flirts with the idea of full-on disclosure of taxpayer information and then retreats from it immediately. And so I think that's the beginning of a trend that we see toward privacy, right? That's sort of the last gasp of public disclosure. Um, and then it it kind of peters out. Notably, once World War II happens and the income tax stops being just a tax for rich people and becomes a tax for middle class people too. There's really no talk about public disclosure of tax data if it goes around. I mean, not surprisingly.

SPEAKER_01 16:03

Yeah, when everyone's paying it. Sure. Okay, then we have the post-war period. We've got the the 40s, 50s, 60s. We get through to the the 1970s and the Nixon era, and there was a whole lot of stuff going on with Nixon and the IRS. Up until that point, it was still that same model of public yet private. Yes. Then what happened?

SPEAKER_00 16:23

As you say, there are these scandals, right? And the Nixon's most famous scandals around this are probably around his enemies list, where they put together a list of people who were perceived to be enemies of the president, the White House, the administration, and they said, let's ask the IRS to go hard on these guys. Let's use the IRS to investigate them, see if we can get them up on some tax charges somehow. We want to harass them using the tax system. Within the administration, that's a the transparent goal. They're not shy about stating it. All right, that comes out during the Watergate investigation. It's obviously a problem. There we're using taxpayer information. We're sharing it around the government, we're sharing it with White House operatives, and they're using it for nefarious purposes. Most people would agree that that's a bad idea.

SPEAKER_01 17:05

A bad idea, but maybe not illegal if it was a generalized governmental asset. You could use it for non-revenue.

SPEAKER_00 17:12

It was it was contrary to practices at the IRS, but it was not illegal as it turned out. Maybe. And that was never adjudicated, so it's hard to say. But at the same time, there's like a much wonkier, but probably even more important thing that a scandal that goes down. It's not even really a scandal. The Department of Agriculture decides that they need taxpayer information, access to tax returns, in order to produce better data about American farms. They actually get tripped up because they're trying to make estimates of like, you know, how big is the crop of this and how big is this crop over here? And they really screw up a count for the number of cows in the country in one year. They get it really wrong. And they say, we need a better system. We're going to develop a better statistical system. And what they come up with is like this idea that they're going to use tax data to develop better sampling techniques for agriculture. Yeah, so this is going to be wonky, right? It's about agriculture statistics. It is literally about counting cows. And they get an approval from the Nixon White House that says, okay, sure, here's your permission. They issue an executive order. It says you can share this information with the agriculture department. Because remember, the president's in charge, right? Right, right. These are public records, not actually available to anyone, but the president can make them available. And he does. He says, okay, Ag Department can see this stuff, right? IRS open up the doors. Farmers freak out, right? Comes out in the newspapers and hundreds of articles run. And the farm states are in an uproar. And what they're basically saying is, why are you snooping on farmers? Why us? Why not anyone else? Why, you know, why are no other kinds of workers or or Americans having their tax returns looked at?

SPEAKER_01 18:45

Why not bankers?

SPEAKER_00 18:46

Right. So why are us farmers? And so people are suspicious out of the gate. Meanwhile, Watergate's unfolding, information's coming out about the enemies list, some White House people are being forced out of their jobs, forced to resign. Some people are going to jail already for the break-in. And there's a lot of suspicion about the Nixon White House. And it all sort of snowballs, right? This wonky USDA information sharing thing, which really does appear to have been pretty much innocuous and just about better data about farms, um, starts to take a be seen in a different light. People freak out. Nixon is forced to resign ultimately. And Congress says we are gonna ratchet down this published stuff around taxes because look what Nixon did. And look at that, and this is this is the striking thing. They're like, and look at that agriculture department thing. That was the thing that really motivated them was this was this crazy technical little thing about statistics. And what they said is we need better rules. And the first thing we need to do is take the president out of the driver's seat.

SPEAKER_01 19:50

Yeah.

SPEAKER_00 19:50

Congress needs to take back the power to decide what is permissible disclosure and what is not. So rather than saying technically public, you make the actual call, Mr. President, they're gonna say, all right, technically private, and then in the in the 1976 Tax Reform Act, they define tax returns as private records, as confidential records. And so they are confidential unless otherwise made not confidential, unless otherwise approved for sharing. Uh so they reverse the presumption. And and the idea is they're gonna they're gonna lay out a specific set of circumstances that there are certain exceptions to privacy. It can be used for this, it can be used for that, you know, it can be used by the Justice Department for non-tax criminal cases, it can be used by the agriculture department for statistical purposes, because everyone understood like that was actually kind of a reasonable request. You just don't want the president in charge of making it.

SPEAKER_01 20:40

And this is statutory now. So they're changing federal law.

SPEAKER_00 20:44

They rewrite a section of the Internal Revenue Code, section 6103, which protects taxpayer privacy, uh, and they spell out exactly what these privacy compromises would be, but in the process also establish that privacy is the norm, confidentiality is the norm. And that's a that's a big shift. And it reflects that realization that this generalized governmental asset idea had led to rampant sharing of taxpayer information. Everybody's taxes were being shared throughout the government. I mean, there were millions and millions of returns every year of returns or return information being shared throughout the government. They wanted to rein that in. They didn't get rid of it. And actually, over time, those exceptions have grown in number.

SPEAKER_01 21:28

Well, how many exceptions are we talking about? I don't know.

SPEAKER_00 21:30

You said there are a dozen or so when in the 70s? In the beginning. And there are many times that now. So anyway, that's that's the long and the story of how this has proceeded. That is the regime we live under now, is that 1976 regime in general terms. The question really becomes like, well, once we start carving out 13 exceptions and then maybe 24 and then 34, and sometimes I hear that there are something like 80, although I haven't counted them lately.

SPEAKER_01 21:58

Yeah.

SPEAKER_00 21:59

But once we do that. That is there really a rule left?

SPEAKER_01 22:03

Have the exception swallow the general rule.

SPEAKER_00 22:05

Right.

SPEAKER_01 22:05

And and that's the general rule being confidentiality, the exceptions being numerous. Okay.

SPEAKER_00 22:14

And so I I think that's the question. That's a policy question, right? And we're wrestling with that right now because uh the Trump administration has sort of brokered an agreement between the IRS and the Department of Homeland Security to use some taxpayer information enforcing immigration laws.

SPEAKER_01 22:29

So that's you know, tell me this is ice. We hear about it in the shoes all the time.

SPEAKER_00 22:33

That's you know, that that's still playing out as we speak. Is that the way you want to use taxpayer information? Is that a good use or a bad use of taxpayer information? Reasonable people can disagree on this because that again, that that bargain is being renegotiated. Is is this something that's worth sacrificing in order to enforce the immigration laws? But the tax people, you know, of this world, the tax experts, generally don't like this sort of sharing, any kind of sharing, really, because what they usually say is that's gonna hurt the tax system. People are gonna be less honest, and that and and the tax system is just not gonna function as well. If people lose faith in the security of their information, they are gonna withhold information from the government. They so the government by violating the bargain endangers the tax system itself. That's the argument against any kind of sharing. And and and you know, we can all agree that some sharing is reasonable, but but trying to like scope out exactly what the boundaries are for that, that's a political question, and it doesn't get easily resolved. And that's why we're arguing about it even right now.

SPEAKER_01 23:36

There you have it. Joe, thank you for your insightful takes. I wanted to say that this podcast series is produced by Jordan Parrish. It is edited by Chris Strigo and Carolyn Cumulus, and the series' executive producer is Paige Jones. Thank you for tuning in and see you next time. Tax Analyst Inc. does not provide tax advice or tax preparation services. Nothing in the podcast constitutes legal accounting or tax advice. A full disclaimer is included in the transcript.