Not exactly. Podcasts are built using RSS feeds and because of the nature of an RSS feed, anybody who has the feed address will be able to download the episodes in your feed. However, there are ways to make the podcast harder to find.
To make sure your podcast stays private, you can keep it out of directories (Apple Podcasts, Spotify, Google, etc.) by not submitting to them. Then you can give out the direct link to your audience or embed your episodes
on your personal website.
Note: It would still be possible for people to share the direct link to somebody else, but it is unlikely that somebody would be able to find the podcast on their own.