I've synced 150+ contacts at one time from outlook. So you shouldn't be running into a limit or memory issue.
Did you check the log?
Goto the Synchonize screen and click on the Configuration tab.
Look for the View log button.
This is the log of the most recent sync. There should be some clue here.