No, someone who was determined could not do it.
There is no password for them to crack. So I don't think it would be safer to use a traditional password (which you use anyways, as a backup).
Your finger doesn't have to be anywhere near the number or the location to unlock the device. You can be as determined as you want, but unless you know the location of the unlock point and which number the person is using, then the person is SOL.
How would you go about unlocking the phone from the screen below?
