Ahhh, I just found my solution... looked for an hour, then found it 2 minutes after posting.

For anyone else with this problem, if you disable Scrolling and Local Input from the menu, vi works well.