The problem is that mullahs, extremists or just plain local gundas or mafias work in organized groups, whereas the ordinary good people are unorganized individuals, and that's why they can have control over a larger population in spite of their smaller numbers, that's why handful of bad people can run a large mafia and extortion racket.
In case of religious extremists, just imagine 20-30 of them attacking a minority's house in a locality of 100 ordinary innocent Muslim families, will they dare to come out and protest or drive away the extremists even if they feel really bad about it? When the bloggers were hacked on the road, did anybody dared to fight the killers? Good people are helpless individuals everywhere, so a small extremist group can become deadly. Unfortunately extremism seems to be growing in Bangladesh.