Gandalf is around two thousand years old at that point. What he might call "my time" has been very different from most other people's. He has seen kingdoms rise and fall, forests grow and wither, cities built and deserted many times over. Yet he has this peculiar ability to relate to and sympathize with the mere mortals around him, conferring to them bits and pieces of the wisdom he's gained over all those years. When he says "So do I," he is not faking sympathy. He really is down-to-(Middle-)Earth and knows how ordinary people feel.
