It matters. Projects that don't support IE ≤11 could ship ES Modules to the browser today. However, you can't ship most code that's written as "es modules" because this code is typically written for a transpiler rather than a real runtime modules environment.
Some of this is a limitation in browsers, like support for bare module specifiers. That is, you should be able to write code like `import {foldl} from 'lodash';`. Work is underway to support this in the browser without requiring multiple round trips to the server to guess where 'lodash' should be loaded from: https://github.com/domenic/import-maps