No, small men can pack a punch, look at Macron, Sarkozy or Putin. but when men, of any size, act as if they are little and insignificant and dress the part, yes they will be ignored and walked over, especially if they start out being slight in stature. Especially by physically big bullies.
People have always been judged by how they present themselves, by their demeanour as well as their clothes and always will be.
Compare Mrs May with Christine Lagarde, the head of the IMF, Mrs May completely fails to project an air of authority and it shows in how she dresses and her demeanour.
Christine Lagarde, is white haired and her face is wrinkled, She dresses quietly and discretely in black, white and grey, but, even in pictures she projects power and influence, because of how, in her carefully chosen and deliberately expensive clothes, she projects an image of a powerful person both by demeanour and dress.
www.google.co.uk/search?q=Christine+Lagarde+images&tbm=isch&tbo=u&source=univ&sa=X&ved=2ahUKEwiNkKv0xdHeAhWJD8AKHdNrDN8QsAR6BAgFEAE&biw=1600&bih=758