You're going to be severely constrained by the fact that idiomatic clojure code uses immutable data structures which cannot theoretically be as fast as mutable ones in sbcl. Even with transience, the Clojure hash map trie is an order of magnitude slower than sbcl's hash table.
Of course. I don't mean to imply that optimized Clojure could match optimized Common Lisp, just that the disparity wouldn't be quite as stark. For truly speed-focused code on the JVM, you have to write speed-focused Java.