Does the size of a popular js library like jquery matter, really? Most browsers (if you're using one of the CDNs) will have it cached already. And the expiration is something like a year.
There are so many versions of jquery, and so many cdns, that most likely 70-90% of visitors don't have your specified jquery file cached. So yes, I believe it does matter. Jquery isn't something understood to be special to a browser, wherein the http requests would go through a special "see if any jq of the required version is cached" process.
Maybe I'm missing something?