I guess it would suck to have a punching specialty if you'd been disarmed.
That said, I wouldn't have any more issue with it than someone with the Semi-automatics pistol specialty. 95% of the time you can make sure to be using your specialty. That five percent when you wake up cuffed in the back of a KE cruiser, things might be bit rough.