You can make gates out of more than just electronics (mechanical, electromechanical, pneumatic, hydraulic computers all exist), hence the suggestion to start there --- the abstraction doesn't really leak.
More seriously, it is possible to fit quite a lot of this stuff into an undergrad program - my Cambridge degree included logic gates (including drawing silicon layout with coloured pens), computer architecture (ARM assembler), a lab on FPGAs, Java, Standard ML (for type inference and lambda calculus), and had plenty of time left over for higher level topics.
But to understand semiconductors you first need to understand quantum mechanics. And to understand QM you first need to understand classical physics.
Oops, I think we just ran out of time for a masters program. ;)