The memory numbers reported on OSX by the system are a little weird, and include shared libraries in some kind of odd way. If you want to see the breakdown of how much memory the browser is actually using, go to about:memory and click on "Measure". You can mouse over the various categories to get a description.
Either that or you have an addon installed that is using a huge amount of memory.
Either that or you have an addon installed that is using a huge amount of memory.