Start with _The Little Schemer_ or _Land of Lisp_* . It's really not hard, less difficult than a language like Java. It's different, though, like APL or Erlang are different.
* The former uses Scheme, the latter uses Common Lisp. While you could get caught up in the lengthy debate over which (if either) is superior, it's probably a better to just pick one and learn the basics. Then, you'll be able to make an informed decision later.
I'm definitely not thinking of SICP, but now that I look at the table of contents of _The Little Schemer_ I must have been thinking of another book. It had an innocuous title but very tricky and subtle contents. It wasn't very long. And it wasn't _Simply Scheme_ either.
Okay. If it helps, the formatting for _The Little Schemer_ is very distinctive: a lot of short question-and-response pairs, with (perhaps overly) cutesy drawings of elephants. _The Seasoned Schemer_ is harder, but it's the sequel.
If you like the format, _The Little MLer_ is one of the best intros I've seen to ML-style type systems.
* The former uses Scheme, the latter uses Common Lisp. While you could get caught up in the lengthy debate over which (if either) is superior, it's probably a better to just pick one and learn the basics. Then, you'll be able to make an informed decision later.