Regular expression systems are designed for a very wide variety of text processing tasks. Lexing and parsing systems are optimized for a much more limited range of tasks--lexing and parsing. When you are writing software that is intended as a production system, not a one-off quickie, you are almost always better off using a more specialized system optimized for your specific task, if one is available. The unoptimized, general purpose system would be Plan B. I think that was Pike's point.
Unfortunately, it will probably be misinterpreted as a general criticism of regular expression systems. There are so many cases where my knowledge of a powerful, general-purpose system has saved me from having to learn a new software library for each random task that I intend to keep using such things as regular expressions, scripting languages, unix command line pipelines, Excel, etc. Writing my own code or learning to use a new, specialized system is Plan B for simple, non-production tasks.
Only when I can amortize the cost (time/effort) over a lot of use (as in a production system), or when the advantage of the specialized system is noticeable to the user (including me), does the specialized system become my Plan A.
General purpose tools, special purpose tools, and custom coding all have different costs and benefits, and there are circumstances under which each one is the best choice.
One thing I've found is that using a bit of a subset of regular expressions seems to work very nicely for doing many lexing tasks. Things like using ^[a-zA-Z][a-zA-Z0-9]+ (or whatever shorthand your RE system has for those classes) can make a lexer nicer to work with. Now I wouldn't be trying to do lots with most of the other features in them for a lexer but it ends up a compact but simple to understand way of describing the input the lexer is looking for to pull tokens out.
Unfortunately, it will probably be misinterpreted as a general criticism of regular expression systems. There are so many cases where my knowledge of a powerful, general-purpose system has saved me from having to learn a new software library for each random task that I intend to keep using such things as regular expressions, scripting languages, unix command line pipelines, Excel, etc. Writing my own code or learning to use a new, specialized system is Plan B for simple, non-production tasks.
Only when I can amortize the cost (time/effort) over a lot of use (as in a production system), or when the advantage of the specialized system is noticeable to the user (including me), does the specialized system become my Plan A.
General purpose tools, special purpose tools, and custom coding all have different costs and benefits, and there are circumstances under which each one is the best choice.