Fully anecdotal, but I got my appendix removed at 19. After 2-3 years i started to have problems and found out that I can no longer process milk(protein), not allergy but intolerance and I started to have issues with "quality". I get bloated and my stomach hurts. Before that I had no issues and had perfect movement. So yeah I didn't "needed it" but quality of my life went down.
In your case I would really doubt if apendix was the cause. As a teen I also loved milk, pancakes, pasta, etc, but around early 20s started to notice that they are bad for me. Now if I eat a lot of milk and glutamine I get digestive issues, skin rashes, etc. My apendix (afaik) is intact.
Milk allergies are different from being lactose intolerant. I developed a mild allergy to dairy in my 30 or so - or so it was discovered then. With lactose intolerance, you can at least take a pill - not so much for an actual allergy. My one child has a much more severe dairy allergy and will get a very upset stomach even with a little dairy.
We both have our appendix.
Though to your point, whether you can develop a milk allergy later in life I am not sure.
