This is true. You also need to take into account how each day is a smaller and smaller percentage of your life which might be why kids think summer vacation lasts forever but it flys by for parents.
This explanation comes up frequently in these discussions. It makes a certain amount of sense on its face, but is there any evidence that this actually affects our perception of time?
See Douwe Draaisma, Why Life Speeds Up as You Get Older, a book that seeks to explain the work done in psychology and neuroscience on this issue to a popular audience.
We don't even know if our brain even has a uniform perception of time (outside of the cultural definition), let alone what that might be. It's all guesswork anyway.
This isn't really evidence at all, but it's well known that as people age, it becomes more difficult to new memories, than it is to keep old ones[1], which is why you regularly get old people mixing up their current loved ones with old, dead people; and why "you can't teach an old dog new tricks".
If the passage of time in the human experiences is largely a function of the memories we make. It's not at all surprising that things seem to go faster as we age.
[1] There's common-sense explaination for this, but I don't think the exact mechanism is understood. It is widely observed though.