61A was easy in terms of math, but hard in terms of everything else. Every student I met who had never programmed before taking 61A found it intellectually quite challenging, both before and after the switch to Python; even without math, thinking in terms of computation and execution is not natural.
Given that second-hand experience, I found the excision of all of SICP's math from the curriculum to be quite a good choice, as adding advanced math as another prerequisite would have raised the barrier to entry even higher.
(Side note: your description of EE40 sounds like a professor who didn't schedule well, not intentional difficulty.)
(As you know) most of the lower div EECS courses inherit slide decks and the coverage is set by the department. CS70 has a wonderful set of notes which should be a book. I regularly go through the CS61C, CS152 and CS252 decks; those are great Berkeley classes. CS162 has a good set and then Kubi has his which even better. Hilfinger's CS61A notes were quite good. The deck the EE40 prof used was based on a previous semester and it was pretty good. It's just that the class is an insane amount of material and work.
There was an intent behind the work. All of the lower div classes are hard with a massive amount of content. But you can show up a Berkeley from a substandard high school on the wrong side of the tracks and blow it out of the water; a buddy of mine was like that and went to Columbia for his PhD. It's very simple. You just have to work. Immediately. Day one. I don't think Berkeley EECS rewards creativity. Indeed I think EECS punishes creativity. But EECS teaches its students to work their asses off, regardless of whether they came from Lick Wilmerding or Skyline.
Don't know if the workload was all that hard compared to, say, the experience of my MIT friends, but the level of intellectual rigor and methodical thought was high.
Given that second-hand experience, I found the excision of all of SICP's math from the curriculum to be quite a good choice, as adding advanced math as another prerequisite would have raised the barrier to entry even higher.
(Side note: your description of EE40 sounds like a professor who didn't schedule well, not intentional difficulty.)