I called the tech support number posted above and figured out a solution. I have to pause the music before I remove the adapter in order for the audio to be routed properly. Not sure if this will be a viable option though, since I cannot manually pause he music when I have an incoming call. I guess I will have to see how it it works out over time.
Might give me a good excuse not to answer my phone