Mamá means mom, madre means mother. You use either depending on the context. (I'm only talking about the Mexican use)
Madre is used more often in phrases, such as "Madre de dios!", Or "La madre putria," or "Pinche tu madre", or it's used as a formal to refer to one's mother. It's similar in English, such as "On my mother's side", "Mother fucker", "Mother of God!", etc.
In Mexico, "tu madre" can be a general insult, probably shortened from "pinche tu madre", though in English we have a similar vulgar retort of "your mother". As a weird turn on the phrase, "de puta madre" is slang for a good thing.
I'm not aware of an association with brothels in Mexico, but certainly in some Latin American country it could have.
Madre is used more often in phrases, such as "Madre de dios!", Or "La madre putria," or "Pinche tu madre", or it's used as a formal to refer to one's mother. It's similar in English, such as "On my mother's side", "Mother fucker", "Mother of God!", etc.
In Mexico, "tu madre" can be a general insult, probably shortened from "pinche tu madre", though in English we have a similar vulgar retort of "your mother". As a weird turn on the phrase, "de puta madre" is slang for a good thing.
I'm not aware of an association with brothels in Mexico, but certainly in some Latin American country it could have.