It becomes even more confusing when you realise actual olives aren't even that colour. Olives are just _green_ green (or very dark brown to blue/black).
The plant parts which will eventually become green, when they will contain enough chlorophyll, pass through an initial stage that is either light green when they were initially white or yellow-green when they were initially yellowish (due to carotenoids).
Immature olives pass usually through a stage when they are yellow-green, but that stage may not last long.
I agree that using as a color term the name of something that may have a variety of colors, is not a good choice.
Nevertheless, the use of "olive" as yellow-green is entrenched in various domains, e.g. the yellowish green mineral that was called "topaz" by the Ancient Greeks (now topaz means something else) and which is called now "peridot" when in gem form, is also called "olivine", due to its color.
However on that page I see indeed green, not olive.
Another poster explained the likely reason for this choice of word, but the discrepancy between the shown color and its name is slightly confusing.