I think this a network conjestion problem. I have found one place where the network conjestion is low (my bro-in-law's house) there it doesnt take long to download any thing and whatsapp connects in well less that 5 minutes. I also suspect having so many contacts takes too long to synchronize whatsapp favorites from your phone.
In a nut shell to avoid the error do any of the following:
a) be patient open whatsapp and let it run for a long long time. hit the red button and stop watching it. Do other things and only come back to check on it after 30min, 1hour etc. Or just leave it connecting for the whole night.
b) open whatsapp in a place you know the network is great! like going to my sister's house too
c) change to a fast network and leave your snail network service provider! Its not fun to have whatsapp on a slow network either.
I hope this helps.