Also, if something truly serves no purpose, evolution will allow it to go away.
The classic example is the enzyme needed to make vitamin C. In our primate ancestors that lived on a diet rich in vitamin C, there was no penalty to losing this enzyme. Mutations that destroyed its function were not selected against. As a result, we now can't make vitamin C; the remnants of the gene for the enzyme have been so damaged that there's no path back to the working version.
The classic example is the enzyme needed to make vitamin C. In our primate ancestors that lived on a diet rich in vitamin C, there was no penalty to losing this enzyme. Mutations that destroyed its function were not selected against. As a result, we now can't make vitamin C; the remnants of the gene for the enzyme have been so damaged that there's no path back to the working version.