I've accidentally wandered onto a film set before. When it's in a public space it's not always obvious what's going on or that you shouldn't be there. I had no idea why a generally quite busy part of town was almost entirely devoid of traffic that day. There were no clear signs or ways for me to know what was going on until I talked to a random guy and he told me.
Perhaps your ability to detect and understand context is simply vastly superior to mine.
Especially when they're shooting on one side of a street near dusk (quite common for the good light) in downtown... yea, it's easy to assume you can walk on the other side of the street before realizing that the crew has shut down the entire block since they're capturing shots across the open street and all those randos walking on the other side of the street are extras.
That said, they'll often just let you wander across as long as you're not wearing distracting clothing.
Accidents happen, and I am not perfect too. I live in a city where film sets in public spaces are an (almost) everyday thing, so I also wandered into a film set few times. But I'd never do it on purpose - when you meet them at a bar you'll see they really do work hard, and that random strangers at the set are a really big problem they face daily.
Perhaps your ability to detect and understand context is simply vastly superior to mine.