Sören Laue, creator and first author of this work [1] gave a nice talk [2,3] about it at our reading group not long ago. He also has written about how automatic and symbolic differentiation are essentially equivalent. [4]
I hope that something like his „generic tensor multiplication operator“ [1], a slightly adapted Einstein notation, becomes the standard notation in the future. The traditional linear algebra notation with it‘s ugly transpose formalism is nothing I would ever want to use.
I fully agree with you on this one. It took us ages to figure out that linear algebra notation it not the right way. Using tensor notation and einsum, tensor and matrix derivatives become almost trivial. And no need to use ugly transposes, Kronecker symbols, etc.. Thats why we wrote this article and created the website. (Sören Laue, creator of this website and my personal biased view)
https://en.wikipedia.org/wiki/Tensor_calculus is a term you're much likelier to find as the subject of a book. Matrices are just specific special cases of tensors, after all, and mathematicians like to generalize.
A good book on differential geometry will also probably start with an overview of tensor calculus.
Hubbard and Hubbard's Vector Calculus is a great introduction to multivariable calculus and linear algebra that ends with an introduction to differential forms. And it has a complete solutions manual you can purchase as well.
[1]: https://papers.nips.cc/paper/2018/file/0a1bf96b7165e962e90cb...
[2]: https://www.youtube.com/watch?v=IbTRRlPZwgc
[3]: https://compcalc.github.io/public/laue/tensor_derivatives.pd...
[4]: https://arxiv.org/pdf/1904.02990.pdf