Yeah, most of these SF don't stand imposing like what you would expect when we are talking about Special Force.
I remember when I was in Ranger School. That instructor told us what set us and normal soldier apart is that you have the will to last longer, and when everything go against you, your will power is the final frontier to overcome everything.
Most people found that vague as to what they were saying, but when you went thru the training, you know that your biggest enemy is your own body. You get tired, you get hungry, you get thirsty and you get pain, all these is when your body is telling you to stop, which your will power is the only thing that stop you from stopping, thus making that extra mile. Which mean the bigger a person is, you get more tired (Because you are bigger), you get more hungry (again, because you are bigger, you eat more), you get more thirsty and you get more pain, most likely big guy don't last in Special Force selection. Well, some do make but at the end of the day, it's not the body size that make you special, it's your mind.