The source of knowledge of all rituals is Veda, particularly Yajur Veda. Only Brahmins were originally entitled to learn Veda and practice the rituals because only brahmins had the required qualities and qualifications for it.
Those who do the poojas, prayers and rituals were expected to live a pure life, be very dedicated, be free from lust and greed, lead an extremely simple life (funded purely by donations/ gifts/ grants from Kings or the dakshina —voluntary offering paid by the ritual seekers). They were expected to eat less, practice austerities of the highest order, never work for a salary, follow vegetarianism, not consume alcoholic drinks etc. These were indeed the essential qualifications for people who were to be fit enough for conducting rituals.
Of the four varnas, only Brahmins were able to live up to these fundamental expectations. Brahmins had access to Sankrit and they were good in reciting Vedas with proper intonation that is very essential for the mantras to take effect. Naturally, the tradition was followed generations after generations.
Like everything in nature being subjected to degradation, Brahmins too got degraded; Brahmins too became greedy, started demanding money for services, started dominating and demeaning other castes, slackened their grip on sanskrit, diluted vedic practices and so on.
But still, by virtue of the tradition, Brahmins are still seen as best of the lot to continue with the practices of conducting rituals. So it goes on.