I am familiar with Linear algebra and at most average understanding of graph theory. I know there is lot more math to cover for ML but I find it overwhelming to start
All you need for introductory machine learning is multivariable calculus (for some simple optimization stuff), linear algebra, and probability. If you don't know probability, here's a good course: https://ocw.mit.edu/courses/electrical-engineering-and-compu....
In its current state, even cutting-edge machine learning is pretty accessible if you have a good understanding of linear algebra and calculus. If you want to do have a deeper understanding of machine learning then vector calculus, tensors, graph theory, etc. can only help you.