When you're 67 and basically retired (as Einstein was at this time)?
By contrast, in his 20s, he wrote 4 highly-influential papers in a single year, in his spare time while also working full-time at the Swiss patent office.
Was it? Slacking off doesn't become "work" just because it lead to something more important than the job itself. See also, Feynman, who was quite openly talking about the idea that you need to goof off a bit to do anything interesting.
Nah, this wasn't work, that sounds more like the hobby that's in love-hate relationship with the work (need work to live, but it takes time and energy from the hobby).
It's not a very narrow interpretation of work - it's the one that matters in this context, and the one that matters day-to-day. Sure, you could define work as anything one did that had results you find interesting, but the narrower definition that's important here is things you do because you have to, because you're obliged to do them by others or by circumstances, vs. things you do for fun - because you want to, and which you control. A job vs. a hobby is a good approximation. It's not uncommon to see people being much more effective at their hobbies than at their jobs, even when both of them are in the same domain. It's also not uncommon to see a hobby to be more worthwhile than one's work.
By contrast, in his 20s, he wrote 4 highly-influential papers in a single year, in his spare time while also working full-time at the Swiss patent office.