Ask HN: Modern approach to web application development with Common Lisp

[kung-fu-master]

Recently, we have started developing web application with Django. Everything is wonderful, but Python is not Lisp and I feel uncomfortable with Python. In the sense that with Python & Django I am restricted. I can't build eDSL to describe application workflows, eDSL for DB access, etc.

I want to try rewrite it from scratch in Common Lisp. What web technologies stack to use?

SproutCore + Hunchentoot + MongoDB is it right choice?

[vog]

Personally, I'd leverage on PostgreSQL rather than MongoDB, because I think relational databases are still the best fit for the majority of modern web applications.

Unless, of course, you have a really simple schema so that you want to use the DB as a mere persistence layer. Or you know from the beginning that you'll have to scale really huge. But then, you'd need a highly customized framework anyway, which would somewhat defeat the purpose of writing a general purpose framework in the spirit of Django.

Also, PostgreSQL is allows to be extended from inside, thus working much in the spirit of Lisp. It can be extended with plain C code, but has also language support for pgSQL, Perl, Python and Tcl:

http://www.postgresql.org/docs/current/static/server-program...

Unfortunately, language support for Lisp languages is missing yet, but this might be only a matter of time (and would be really great).

[escanda → vog]

Following the above, if you're looking for an object store you'd find interesting cl-prevalence which is entirely written in CL.

About web frameworks others have mentioned AllegroServe, which looks quite good with the logic and view separation in templates and actions, similar to Django. Other people prefer to code the HTML inline with cl-who i.e.

If you're developing an application in the web you may find worth a continuation based framework like weblocks or UCW. Cliki contains a fair amount of libraries categorized. Here you have the database libraries: http://www.cliki.net/database

[sedachv → escanda]

bknr-datastore works on the same principle as cl-prevalence, but is actively developed: http://www.cliki.net/bknr-datastore

[Zak]

With any luck, mahmud will be here to talk about real-world CL web development. He's been using CL for web stuff in his startup for a couple years, I think.

Another approach for people who are just trying to get something done rather than write a framework to consider is to use CL to do the interesting bits and something else for the web frontend. I've built a couple webapps in CL that would simply have been easier in Rails.

[silentbicycle → Zak]

Mahmud's traveling and offline for a few weeks (http://twitter.com/#!/BigThingist/status/1947382564519936), but he's answered variants of this question before - try searching the HN archives.

[gracchus]

Hunchentoot is a bit odd, but works well. In particular, the fact that handlers (think, like prefix-matching dispatch functions) are global, rather than per-listener. There is, in general, an apparent tendency to use var globals for expediency's sake, but they tend to infiltrate code, and make things harder to debug.

I concur with other commentators that PostgreSQL is worth looking at. I wrote the postgres bindings for what is now CL-SQL many years ago (hi stringbot!), and postgres is fast and reliable (aside from SBCL having interrupt hander sensitivities during ffi calls), though again, the CL-SQL interface is littered with globals, the connection pooling is a bit .. unusual .. and the reader macro syntax probably a pretty bad idea, all in all. If hunchentoot and cl-sql were mildly rewritten with an eye towards looking more like, for example, Java's JDBC and Servlet interfaces, you would have a simple base environment where you could reap most of the big CL wins immediately (DSLs, CLOS, coding closer to the speed of thought, etc.)

The 'IMHO' sources are kicking around here: they suffered a bit from WebObjects envy (hey, it was the 90's..) and do somewhat suboptimal handling of session state. It's still not clear to me how OO you really want to get with the server-side representation of a page, especially in light of all the nice JS client-side component kits that are around.

Mull.

[macmac]

How about going 3/4 of the way to Common Lisp and try Clojure instead. Great lightweight web + mongo libs + all the java libs.

[pavelludiq → macmac]

Yes, taking a serious look at clojure is worth it, the language is awesome, the libraries are maturing and growing in numbers, and the community is awesome. But i wouldn't bet against common lisp either, im currently learning it, and im amazed, clojure programmers can only gain by learning it.

[stringbot]

Back in the day (around the turn of the century) I worked for a startup building a web application using Common Lisp. We used a framework called IMHO, which was developed in-house by a couple of the lead developers. IMHO seems to have been deprecated, per http://www.cliki.net/IMHO and the source code is nowhere to be found. I am still in touch with the authors, so maybe I can encourage them to make it more available.

Using Lisp for a web dev platform was pretty nice. I was a support engineer in those days and I always found it useful to be able to pull up a console and inspect the actual running sessions to debug problems in production.

[gibsonf1]

We've been using Portable Allegroserve + mysql (running on SBCL on EC2) with very good results.

[mark_l_watson → gibsonf1]

I second the Portable Allegroserve advice. It is an older project, but John Foderaro really put a lot of good functionality in Portable Allegroserve. That said, Edi's Hunchentoot project is very popular.

[vog → gibsonf1]

I wouldn't count on MySQL anymore. Oracle recently attempted to remove InnoDB support (i.e. transactional safety) from the Free Software variant of MySQL:

http://www.h-online.com/open/news/item/Oracle-raises-prices-...

Although they took this back for the moment, allowing InnoDB to stay available in the "Community Edition", they made very clear that we can't count on that in the future.

So if you attach importance to ACID, I recommend to start with PostgreSQL rather than MySQL.

[smanek]

I'd hesitate about MongoDB - I'm not aware of a mature Mongo-backed ORM, for starters. Personally, I like http://marijnhaverbeke.nl/postmodern/.

[sqrt17 → smanek]

Since you directly pull JSON objects out of MongoDB (i.e. usable data structures), MongoDB lends itself to a slightly different programming style than you'd have with a relational database and an ORM. I'd hesitate to call either one more idiomatic in a lisp-ish programming style.

EDIT: That said, MongoDB's reliability model (you have to have a synchronized pair of DB servers to get the reliability you get from a single PostgreSQL instance), PostgreSQL may be the safer choice.

[vii]

How about tpd2? If you're comfortable rewriting everything from scratch, at least you'll be working from a high performance foundation -- http://github.com/vii/teepeedee2/tree/master

Weblocks is a very good concept if you're willing to get your head round it, and work within its worldview (which is about displaying widgets rather than manually generating HTML).

Everybody uses parenscript, which is a great concept.

[sedachv → vii]

Both TPD2 and Weblocks rewrite your code using a continuation-passing style transformer, which makes web apps written in them harder to debug than they need to be. It also makes testing your apps or doing any kind of automated HTTP client scripting much harder than it needs to be. There's no upside to managing app state with continuations if you are able to use AJAX.

[WalterGR]

SproutCore + Hunchentoot + MongoDB is it right choice?

Even then you're not yet off to the races. In particular, Hunchentoot's session management capabilities are anemic out of the box.

I haven't seen any drop-in replacements, so you're looking at reinventing at the very least this extremely core part of any web development stack.

Hunchentoot also doesn't have any built-in capabilities for handling updates to the website's source code.

[sedachv → WalterGR]

"In particular, Hunchentoot's session management capabilities are anemic out of the box."

Do you mean user authentication? What would be really nice to have is a user authentication library that offered a generic protocol that could be used with several web servers and persistence mechanisms.

"Hunchentoot also doesn't have any built-in capabilities for handling updates to the website's source code."

You just load the new code and it works.

[WalterGR → sedachv]

Do you mean user authentication?

No, I mean session management. Example of anemia: it offers no way to back sessions by persistent storage.

You just load the new code and it works.

Really? What about threads that are busy processing requests when you load the new code, which may have different assumptions?

It seems like a better approach would be to queue up incoming requests, wait for existing requests to be handled, reload the code, and then resume processing requests.

[sedachv → WalterGR]

"No, I mean session management. Example of anemia: it offers no way to back sessions by persistent storage."

The session protocol is a bit awkward in that you need to implement all of it if you want to change only a small part, but it's there to be hooked into, which is what the frameworks built on top of Hunchentoot do. The support is there to have persistent session objects, you just need to find something that does that (I agree it would be great to have libraries independent of frameworks but built for specific DBs for this) or do it yourself (all that Hunchentoot requires for this to work is to implement two methods).

A lot of the time what you'd really want to do is subclass the session class to make it persistent through an ORM or prevalence, but the problem is that you can't just drop in arbitrary metaclasses into arbitrary subclasses in CLOS.

"Really? What about threads that are busy processing requests when you load the new code, which may have different assumptions? It seems like a better approach would be to queue up incoming requests, wait for existing requests to be handled, reload the code, and then resume processing requests."

You're talking about atomic change management. Queueing up requests works on a per-request basis, but what happens when your code changes affect existing sessions (for example, people in the middle of a multi-step order confirmation form)? The easiest way to do this is have multiple servers and switch them over one by one. In the more general case you need to think in terms of protocols and protocol versioning (http://carcaddar.blogspot.com/2009/03/p-cos-blew-my-mind.htm...)

[WalterGR → sedachv]

Queueing up requests works on a per-request basis, but what happens when your code changes affect existing sessions

Good point.

The session protocol is a bit awkward in that you need to implement all of it if you want to change only a small part, but it's there to be hooked into, which is what the frameworks built on top of Hunchentoot do.

I notice you used the plural "frameworks" built on top of Hunchentoot. Are there any aside from Weblocks?

[sedachv → WalterGR]

RESTAS (http://restas.lisper.ru/en/) and web4r (http://web4r.org/en/) are both built on top of Hunchentoot.

[sedachv]

Take a look at RESTAS, I think it has some good ideas, and doesn't try to force you to do too many things in certain ways, unlike many other CL web frameworks: http://restas.lisper.ru/en/

[pge]

as someone who has written webapps in both lisp and python, I would recommend sticking with python. you have to choose the right tool for the job and get comfortable working with different languages. I am a huge fan of CL, but if you go down that road for a webapp, you will spend a lot of time filling in gaps in the framework instead of focusing on the functionality of your app.

Django may be the wrong tool for the job in your case. If so, I recommend exploring some of the more flexible python frameworks like pylons before making the switch to common lisp.

[gruseom]

If you're building web apps in CL you should take a close look at Parenscript.