I think that as you have some rough exposure to relativity already, you could first absorb the idea that Minkowski (flat) spacetime is a theory where at every point the Poincaré group is the isometry group. That's a good way to hit on representation theory.
However, Lancaster and Blundell's book has some reviews suggesting that someone good at math should be able to work through it without the background needed by textbooks like Srednicki's https://www.dur.ac.uk/physics/qftgabook/ (I have not read it though).
Thanks again. Your suggestions led me to scan the QC174.45 shelves at a nearby university library. I settled on Maggiore's A Modern Introduction to Quantum Field Theory, which seems to be almost all Math.
Representations of the Poincaré group: http://www2.ph.ed.ac.uk/~s0948358/mysite/Poincare%20Chapters...
and generalizing: https://www.wikiwand.com/en/Particle_physics_and_representat...
Introductions to QFT tend to assume you know a lot of physics. An example is the Preface for Students in Srednicki's prepublication: http://web.physics.ucsb.edu/~mark/ms-qft-DRAFT.pdf
However, Lancaster and Blundell's book has some reviews suggesting that someone good at math should be able to work through it without the background needed by textbooks like Srednicki's https://www.dur.ac.uk/physics/qftgabook/ (I have not read it though).