Symbols change their meaning with time.
When I was growing up, one symbol that had a very clear meaning for me was whether ministers wore scarf or stole. (Scarves are black; stoles have the colour of the liturgical season – green, white, red or purple). If a vicar wore a black scarf, it showed that they understood that their role was primarily as a preacher of God's Word. If they wore a stole, it meant that they saw their ministry as being priests, re-sacrificing Jesus on the altar.
That understanding informed what I wore for my ordination. Lots of evangelical ordinands share that view and want to be given a black scarf at their ordination rather than a white stole, because it symbolises being given authority to preach rather than authority to re-enact the sacrifice of Christ on the cross. The official rules of course say that it makes no doctrinal difference which you wear, but that just prompted a friend of mine to find out what the doctrinal difference was. He wore a scarf.
Years later, I found myself in a different part of the country, in a church where no-one would even dream of thinking that the minister re-enacted Jesus sacrifice of himself at communion, and everyone was clear that a big part of the vicar's role was preaching. When I asked them how they understood the difference between scarves and stoles, the only difference they could find was that stoles were colourful and showed that the minsters valued colour and symbols but that scarves showed the vicar was a bit old-fashioned.
Of course, if people understand the symbolism that way, then I'm not going to be so insistent on wearing a scarf rather than a stole... Symbols are flexible and can mean different things in different contexts. There is nothing inherent about a black scarf that means it's about preaching or about a coloured stole that means it carries a certain understanding of communion – those are labels that some people choose to attach to those items of clothing.
Now it seems that scarves are dying out altogether. Some bishops ban them at ordinations. I don't think that's usually because of theology; I suspect it's because it looks neater if everyone is wearing the same thing. But more evangelicals avoid robes as often as they possibly can, which again comes down to symbolism.
For some people, robes symbolise the church they of their parents stopped going to – the idea of a minister who is boring, old-fashioned and out of touch. (That's not always a bad thing; I wear robes every week for a service where it's a positively good thing.) For others, robes symbolise that the people wearing them are different from everyone else. Ironically, that's how robes came about, but in not the way that you'd expect.
In the 400s AD, some clergy had started dressing in a way that was designed to look impressive. Pope Celestine I objected strongly and wrote this:
We bishops must be distinguished from the people and others by our learning, not our dress, by our life not by our robes, by purity of heart not by elegance.
Quoted in Dix, The Shape of the Liturgy, p.401
Shortly afterwards, to stop the clergy wearing fancy clothes that set them apart, the church introduced some rules about what clergy should wear. Ironically, it was those very rules that then stayed the same for centuries and resulted in clergy wearing different clothes from everyone else as fashion changed!
In the late Roman Empire, people who held an office (magistrates, etc) would wear a special scarf to identify themselves and to show the authority that had been given them to do their role. It's that scarf that is the ancestor of both the scarf and the stole.
People who think that robes make an important statement, and that clergy are more about preaching than presiding at communion are also likely to think that robes themselves communicate the wrong message to people, and so are more likely to avoid wearing them, except on special occasions.