Sunglasses are usually made of shatter-resistant plastic lenses which provide more protection for your eyes.
Prescription eyeglasses are usually made of glass lenses which can shatter/splinter causing severe trauma to your eyes.
But... eyewear such as prescription glasses or goggles are permitted with appropriate head straps.
Try wearing contact lenses instead.
I wear glasses, but my sight isn't that bad, so I just take them off and leave them in them in the car or something when I go to water parks. If you really want to keep your glasses on, maybe you could put a pair of sunglasses over them so then they'll let you keep them on.
That's weird, I've never been told to take mine off.