| Yoma
"Everybody is eligible to walk it, but the kohanim made a permanent rule not to let an Israelite walk it" (Yoma 66a). The kohen gadol, hands resting on the sa'ir la-azazel, would say vidui, pleading for forgiveness for the sins of the Jewish people, at which point the goat would be led away la-azazel. Leading the goat out of the Temple and into the desert was apparently an act devoid of any special sanctity and hence, could be done by anyone, including non-kohanim. Yet the kohanim, apparently seeing this as an extension of the Yom Kippur service, insisted on having one of their own lead the goat out of the Temple grounds.
While the kohanim considered this a big honour--after all, this is part of the procedure that helps bring atonement to the Jewish people-- not all saw it this way. "And a ramp was made for him because of the Babylonians who were pulling his hair, saying to him, 'take it and leave, take it and leave'". Instead of seeing the goat as effecting atonement and thanking the kohanim for their efforts, they saw it as carrying sin, causing "impurity" in the precincts of the Temple, and displayed total disrespect to the kohanim.
What is unclear is exactly who these Babylonians were. When we think of Babylonians we instinctively think of the Babylonian empire, who destroyed the first Temple, exiled the people and created the first large diaspora in Jewish history. Such, though, would seem to make little sense in our context. Why should the Babylonians care about goats in the Temple? One could argue that they, out of anti- Jewish feelings, wanted to disrupt our religious practices; though their yelling at the priests to quickly leave seem to indicate they were actually bothered by the sair la-azazel staying in the Temple.
The Babylonians understood the significance of the Temple not only for the Jewish people, but for the world at large. That explains their anger at having keruvim in the Temple--a point we have previously discussed (see here). They, like all ancients (and many moderns), knew that G-d had a special relationship with His people and that what happened in the Temple impacted on them, too. Paradoxically, this explains the desire of the Babylonians (and the Romans) to destroy the Temple, thinking that by doing so they could sever the bond between G-d and the Jewish people. But as long as the Temple stood, they understood it was a special place and its presence had universal repercussions.
Just as they could not understand the sexual imagery of the keruvimbeing allowed in the Temple, they could not fathom how a sin-bearing goat could be allowed to remain even for one extra second in the Temple. Where we saw repentance and renewal, they saw sin and despair.
Yet one can also argue that the Babylonians of our Mishnah are actually Jews, those few in number who returned to Israel during the second Temple period. Those who did return were unlearned, and the attitude towards them can be seen in such statements as, "How foolish are these Babylonians! Because they sit in a land of darkness, they say teachings that darken, and we should not listen to them" (Pesachim 34b).
And it seems that the Gemara understood our Mishnah as in fact referring to Jews. "Rabbah the son of the son of Hana, said: These were not Babylonians but Alexandrians, and because they [the Palestinians] hated the Babylonians, called them by their name" (ibid 66b). This statement brought comfort to Rav Yossi, whose family was from Bavel; and the negative statements about Bavel (i.e., its Jews) were upsetting to him. Realizing that Babylonians was 'slang' for the frivolous put his mind at ease.
There is little doubt the Gemara is making fun of these Babylonians. The act of pulling hair is an ironic one if one wants someone to leave, as by doing so, one just makes it harder for that person to do just that. Interestingly, the word for hair and goat come from the same root with only the (unwritten) vowels distinguishing se'ar and sa'ir.
Whoever these Babylonians may be, we are meant to understand that it is not a goat that threatens the holiness of the Temple, but rather the sins of the Jewish people. Perhaps the imagery of hair--with hundreds of thousands of strands joining together--is a message for the Jewish people, that it is only the unity of the people that can lead to the redemption and the building of the Temple.