That's not the "whole idea" of SR. What he said in his 1905 paper was,
"The introduction of a “luminiferous ether” will prove to be /superfluous/ inasmuch as the view here to be developed will not require an “absolutely stationary space” provided with special properties, nor assign a velocity-vector to a point of the empty space in which electromagnetic processes take place."
In 1920, after developing GR, his views were considerably more nuanced, as expressed in his 1920 lecture at the Leyden University, "Ether and the Theory of Relativity":
"Recapitulating, we may say that according to the general theory of relativity space is endowed with physical qualities; in this sense, therefore, there exists an ether. According to the general theory of relativity space without ether is unthinkable; for in such space there not only would be no propagation of light, but also no possibility of existence for standards of space and time (measuring-rods and clocks), nor therefore any space-time intervals in the physical sense. But this ether may not be thought of as endowed with the quality characteristic of ponderable media, as consisting of parts which may be tracked through time. The idea of motion may not be applied to it."
Einstein, Albert. Sidelights on Relativity . Public Domain Books. Kindle Edition. (0.99$ on Amazon)
That lecture, by the way, is a superb example of presenting the frontier of 1920 physics - before Schrodinger, Dirac, and modern field theory - without using any mathematical notation whatsoever.
"The introduction of a “luminiferous ether” will prove to be /superfluous/ inasmuch as the view here to be developed will not require an “absolutely stationary space” provided with special properties, nor assign a velocity-vector to a point of the empty space in which electromagnetic processes take place."
In 1920, after developing GR, his views were considerably more nuanced, as expressed in his 1920 lecture at the Leyden University, "Ether and the Theory of Relativity":
"Recapitulating, we may say that according to the general theory of relativity space is endowed with physical qualities; in this sense, therefore, there exists an ether. According to the general theory of relativity space without ether is unthinkable; for in such space there not only would be no propagation of light, but also no possibility of existence for standards of space and time (measuring-rods and clocks), nor therefore any space-time intervals in the physical sense. But this ether may not be thought of as endowed with the quality characteristic of ponderable media, as consisting of parts which may be tracked through time. The idea of motion may not be applied to it."
Einstein, Albert. Sidelights on Relativity . Public Domain Books. Kindle Edition. (0.99$ on Amazon)
That lecture, by the way, is a superb example of presenting the frontier of 1920 physics - before Schrodinger, Dirac, and modern field theory - without using any mathematical notation whatsoever.