Anti-Semitism is a term that, in the western culture, is used to mean anti-jewish. When you treat it literally, you are ignoring a long and established history associated with it. See this for much more articulate explanation than mine:
<quote>
anti-Semitism is not the optimal term for anti-Jewish sentiment, among other things because there are other Semites than Jews, but it is the one we have, and comes with a long history and a well-established meaning. Rather than critiquing the term or coining a neologism to substitute for it, speakers of English should simply understand the term's history and commonly accepted definition and use it accordingly.
</quote>
http://www.ibishblog.com/blog/hibish/2010/12/16/arabs_and_an...
<quote> anti-Semitism is not the optimal term for anti-Jewish sentiment, among other things because there are other Semites than Jews, but it is the one we have, and comes with a long history and a well-established meaning. Rather than critiquing the term or coining a neologism to substitute for it, speakers of English should simply understand the term's history and commonly accepted definition and use it accordingly. </quote>
edit: corrected typos