It may harm the memory card if you have one installed, but only if the Mac is somehow interacting with it. I would suggest you eject the card in Finder before pulling the cord, just to be safe.

Incidentally, with the random device removal error, I was able to keep this from happening by going into Energy Saver preferences and unchecking the "put disks to sleep when possible" option.