This is a false dichotomy. Code calls code calls code; frameworks are libraries are frameworks. The only difference is when a library gets big and complicated enough, we call it a framework.
By your silly standard, passing a function to _.each() makes Underscore.js a "framework", which is probably not the point you were trying to make.
It isn't a false dichotomy. Inside of vs on-top-of. Or, you can say that framework code is lower in the stack frames. Or, you can say that you must make your code comply with the framework's execution model. Or, you could say that the framework supplies the organization and you provide the customization.
From this perspective, underscore.js presents an Enumeration framework. However, it is usually a couple stack frames deep and still way at the leafs of your code paths. Backbone.js is clearly a framework -- it describes your data model plumbing and interactions and provides the bindings and pipelines -- this structure dominates the organization of your code and program flow. Nothing about underscore's utility functions have this "infectious" or "dominating" characteristic.
I don't have a strict definition for library and framework, but I think the distinction comes down to more than just size.
A framework is indeed "just a library," but when one is using a framework one is building on that framework, using that structure to construct some new thing. A library I see as something that is called into for some routine, it's a body of functionalities one can tap into. Your code is-a instance of your chosen framework, and it has-a series of calls out to libraries.
By your silly standard, passing a function to _.each() makes Underscore.js a "framework", which is probably not the point you were trying to make.