It's not so much just the Isp -- rockets can easily get to orbit on LOX/RP-1; LOX-LH2 is not needed -- but also dry mass of the stage. Basically the entire solid rocket motor is a thrust chamber, and must be strong enough to withstand that pressure. In a pump fed liquid propellant rocket, this is not the case: the tanks need only withstand a pressure high enough to avoid cavitation in the pumps.
