This approach is what took my math skills to the next level. I would also frequently skip exercises and forget material that I learned previously. Since I started adding exercise books to my study routine I have seen enormous gains in knowledge retention and my ability to build on concepts already learned. It also doesn't have to be Olympiad style, there are also Math Circles and the general Problems in {Area} book, like one of my favorites Sequences, Combinations, Limits.