You will never be able to tell how you wouldn’t change had you not tried these. But I feel the same as you and I still feel the impact of my only LSD trip I had over 15 years ago like it just happened over the last weekend. However, I know someone who tried LSD for a year here and there every other weekend and I could tell that they incurred some dammage in their brain. They’ve grown in a way but feel fragmented and a bit broken in some other way.
IMO a healthy relationship with LSD is one where, after a certain point (usually single-digit experiences), you can very confidently wish the substance a heartfelt goodbye, and thank it for it's service.
After my third experience, I woke up the next day knowing I'd learned an enormous amount about myself, and learned to think in ways I couldn't previously.
At the same time though, I just... "Knew"... I'd learned everything LSD had to teach me and that any further experiences with it would be like trying to over-optimise my brain, causing more harm than good.
That was ~4 years ago, and I've had literally zero desire to even consider touching the stuff since then.
I'm glad I did (3-4 times), but I'm VERY confident I'll never do it again.
I’m not confident i’ll never do it again ever but after fifteen years I don’t need to trip yet since I’m still drawing from my last one. I feel that it’s useful when at a certain stage in life one needs a stir.