What I have seen. The Podcast doesn't close unless you hit the menu key and choose close, using the back key does not close the pod cast application.
Since the pod cast is still in the background hitting the mute key will start it. (as mentioned aiharkness)
Is it possible the mute key is being hit? If so that would cause the pod cast to start playing (since it may already be running the background.)
I think that's intention, when you unplug anything from the Aux port it pauses the media.. I actually like that