Exactly. The one who had the knowledge was the Brahmins. Not the other way around.
The point of discussion is how and when the system got corrupted. There are plenty of examples in Kerala (since it was untouched by violent islamic invasion) where dalits became as knowledgeable as the Brahmins and rejected the supremacy of the Namboodris. They started their own temple and had their own priests.
In the North, the Bhumiars did it too.
The division of power was well established in Hinduism.
1. The Brahmins got to have knowledge, but was devoid of land, wealth and arms.
2. The kshatriyas got to have arms, but was denied knowledge outside his domain, land and wealth.
3. The Vaishya got to have wealth, but was denied knowledge outside his domain, land and arms.
4. The Shdra got to have land and cows, but was denied denied knowledge outside his domain, arms and wealth.