I think I figured it out. Ha ha, of course after I posted this I thought of another way to word my question and found a forum that mentioned proximity sensors.
Turns out I need to hold the phone on my right side. Guess the sensor is only on one side!