In the app Swipe right to left on the podcast in the main screen and select delete. That unsubscribes you.
That unsubscribes you from an individual podcast. Trust me when I say I had hundreds of podcasts that I had to unsubscribe from.
Ah, I don’t think that there’s an easy way to delete all, other than to start a new account.