Political colors have a much older history than the modern, American iteration. The oldest case of colors being adopted by political parties is about 1,500 years old. In the sixth century, the organized sport of chariot racing inspired the same passions in the Byzantine Empire that soccer inspires today, as Smithsonian explains. The racing teams each had a color, and each attracted fans from a particular class or area. The Blues, for instance, drew fans from the aristocracy; their hated rivals, the Greens, were supported by commoners. The sporting rivalry and the class, religious, and political enmity between the fans was never quite distinguishable. In 520, a fight that broke out in the hippodrome of Constantinople between the Blues and the Greens — ostensibly over the race, but inflamed by tensions around a new tax law — caused a riot that had to be put down by the army. The Emperor Justinian felt, perhaps with reason, that his life was at risk.
Political colors faded in the Middle Ages, when the arms of a particular lord or monarch functioned as political badges, but occasionally factions or parties would adopt a color. This was especially true in the Italian republics, like Florence, where the ruling Guelph faction broke into two camps: the Whites and Blacks (via Britannica). Dante Alighieri, a White, would be exiled by his Black rivals on pain of death, inspiring his great poem of exile and salvation, the Divine Comedy.