Category theory might be easier to think about as a different way to think about the same objects we've always studied in mathematics. Since it's a formal theory (just like set theory), we can also study category theory mathematically. In fact, we can study set theory categorically and we can study categories set-theoretically -- they aren't really at odds with each other.
If you wanted to study set theory from a categorical perspective, you would likely be interested in topos theory, which captures essential properties of set theory and generalizes them to a variety of interesting settings. There's also a really good (and IMHO approachable) paper by Tom Leinster [1] on how to see set theory from a categorical perspective, without going all the way out to toposes. I think it's a good way to understand what category theorists try to emphasize.
If you wanted to study categories from a set-theoretic perspective, you're in luck -- that's probably the most common approach to learning category theory in the first place! Studying categories with categories is also possible, but you already need to be fairly comfortable with categories in order to model them internally. (Studying set theory with set theory is much the same -- models of ZFC can get wild.)
(You can also learn category theory from first principles, without thinking in terms of sets. This isn't really any different from set theory, either -- set theory is often peoples' first introductions to abstract mathematics at the undergraduate level, so there isn't really much choice in the matter. For my money, though, I think category theory from first principles is a little less weird.)
If you wanted to study set theory from a categorical perspective, you would likely be interested in topos theory, which captures essential properties of set theory and generalizes them to a variety of interesting settings. There's also a really good (and IMHO approachable) paper by Tom Leinster [1] on how to see set theory from a categorical perspective, without going all the way out to toposes. I think it's a good way to understand what category theorists try to emphasize.
If you wanted to study categories from a set-theoretic perspective, you're in luck -- that's probably the most common approach to learning category theory in the first place! Studying categories with categories is also possible, but you already need to be fairly comfortable with categories in order to model them internally. (Studying set theory with set theory is much the same -- models of ZFC can get wild.)
(You can also learn category theory from first principles, without thinking in terms of sets. This isn't really any different from set theory, either -- set theory is often peoples' first introductions to abstract mathematics at the undergraduate level, so there isn't really much choice in the matter. For my money, though, I think category theory from first principles is a little less weird.)
[1]: https://arxiv.org/abs/1212.6543