I was in High School with 300 other kids and there were at least two dozen of us who at the very least toyed with assembly on either the TRS-80, a VIC-20 or an Apple II, and knew enough that we wanted to know more. I grew up in a farming community in Illinois.
I doubt that there were millions in any single country, but over the entire world, I would be very surprised if "millions" is not accurate. (Precision is another discussion)
I was like dexterdog, the only person in my school that could program. I toyed around with assembly and C and made some progress but nothing significant. I know people who went to other schools with a dozen or more kids toying around together, like you, and they made it a lot further during their school years, and this advantage has left them still very far ahead to this day, 30 years later.
People learn much faster and better in groups than as individuals.
I had a number of kids in my school (very few in my grade) who were into programming, but nobody was interested in assembly. This was the late 80s. We had an AP class which was not common at the time and I was able to skip intro comp sci in college because of it. Pascal was the main language.
I doubt that there were millions in any single country, but over the entire world, I would be very surprised if "millions" is not accurate. (Precision is another discussion)