S-expressions are not directly an AST: they are a form easily matched to an AST (which could be the Lisp at hands, or some language for which there is some translator/parser/interpreter around, or some language for there is no code in the system to (fully) handle it). This makes the use of s-expressions simple, they can be handled by the development tools without them always having to know what language they represent.
If your development tool knows how to handle a particular language, then it can handle the AST for that language, but not any other. You could instruct the tool about the other languages, but that requires ways to let the tool know; in the end this is similar to the static versus dynamic typing discussion: static typing doesn't allow any potentially unsafe data, and a tool working with an AST directly wouldn't allow any potentially invalid programs, which in principle (or theory) is nice but difficult to achieve practically, it presents overheads for the tool implementer and for the user probably alike. And it probably takes away a lot of interactivity: just like reloading modules in a statically typed language (like ML or Haskell) is not as direct as being able to modify values and functions in a running system and having them be called directly.
So I think S-expressions are a very pragmatic concept that is going to be difficult to replace without the replacement costing more than it benefits.
S-expressions are a syntax for data. It's more related to what a tokenizer does, but with nested lists of tokens - not a flat token stream. It's only loosely related to an AST, since the s-expression represents some form of nesting - but not explicitly what kinds.
S-expressions know nothing about the syntax of the program language Lisp (what is the syntax for the LET operator?). If we have a Lisp expression, an AST for it would need to represent specific language constructs - like knowing that LET is a built-in operator and has parts like a list of bindings (where each binding is a symbol or a list of symbol + value form, optionally declarations (with their specific syntax) and a list of body forms. The s-expression does not represent that: in
(let ((let 'let))
(list let 'let))
the s-expression does not represent that LET is a built-in operator, a variable name, a symbol, a variable name and a symbol. All we see is five LET symbols and no additional information.
Writing Lisp code using s-expressions allows us to use data manipulation functions both on text and on internal Lisp data: reverse, transpose, move, move in, move out, ...
We can then also enrich the editor (or other tools) to understand the syntax of Lisp. Example: know what the structure of a LET form is and then allow for example to manipulate the binding list, knowing where it is and what a variable and what a value form is.
If your development tool knows how to handle a particular language, then it can handle the AST for that language, but not any other. You could instruct the tool about the other languages, but that requires ways to let the tool know; in the end this is similar to the static versus dynamic typing discussion: static typing doesn't allow any potentially unsafe data, and a tool working with an AST directly wouldn't allow any potentially invalid programs, which in principle (or theory) is nice but difficult to achieve practically, it presents overheads for the tool implementer and for the user probably alike. And it probably takes away a lot of interactivity: just like reloading modules in a statically typed language (like ML or Haskell) is not as direct as being able to modify values and functions in a running system and having them be called directly.
So I think S-expressions are a very pragmatic concept that is going to be difficult to replace without the replacement costing more than it benefits.