Hacker News new | past | comments | ask | show | jobs | submit login
Growler: Asyncio Micro-Framework in Python (github.com/pygrowler)
116 points by akubera on May 4, 2016 | hide | past | favorite | 30 comments



Awesome work! I just watched your PyOhio talk[0], and seems really promising. I use Flask daily, and while performance has not been an issue yet, I know the key word in that phrase is yet.

Would love to play with the code. Could you add a CONTRIBUTING.md? I could try to figure out how to get started on my own, but having a few pointers would be really helpful.

I see there are no issues on GH. What are things you're hoping to add soon/are not implemented yet?

[0] https://www.youtube.com/watch?v=vs7XOn1TXVQ


Thanks!

The biggest missing piece is simple, well-tested middleware.

My focus has been on very general middleware; a kind of 'meta-middleware' that actually spec out a common behavior to minimize surprises and boilerplate. For example, my Renderer & RenderEngine classes[0] don't do anything on their own, but are intended to be used by other, more specific renderers (eg MakoRenderer, JinjaRenderer). I think some common functionality really need to be standardized and not done ad-hoc, as conflicts could arise. (more description on how Renderers work below)

The next one of these I'd like to solve is authentication: a standard interface for passwords/oauth/etc. I've kinda used http://passportjs.org before, might be a starting point; I don't know what a pythonic version would look like.

## Non-Middleware

- Figure out equivalent of JS EventEmitter interface. Currently I have no good way to do something like res.on('send-headers', cb) ~ (thinking res.events['send-headers'].add(cb)) with events as EventManager object. Maybe decorators here?

- ETAGs

- HTTP/2

- pytest-growler for automated fixtures

- I don't think an admin panel is necessary (can't beat django) but would like some CLI tools for site creation, testing, and deployment.

Renderer Explaination: (too many 'render' words)

The author creates a RenderEngine as the actual middleware: app.use(MakoRenderer("/path/to/view")). All this middleware does is attach a callable Renderer object to res at res.render if it doesn't exist, then add itself to res.render - this allows multiple RenderEngines in the same app (if the author desires) - and none of them will clobber one another. Upon calling the Renderer: res.render('foo', data_obj), the callable scans through the engines until one with 'foo' is found, then the rendered text is sent. It's non-(obvious|trivial) situations like these I want in Growler proper (some batteries included) - the rest can be extensions.

[0] https://github.com/pyGrowler/Growler/blob/dev/growler/middle...


Also pressing is the method for turning "route strings" (I call them "sinatra paths", I'm not sure where they actually originated) into a regex

/foo/{bar} or /foo/:bar => re.compile(r'/foo/(?P<bar>[^/]+)')

https://github.com/pyGrowler/Growler/blob/dev/growler/router...


Impressive stuff. But um, about that project name; ever watched "Bo Selecta" https://en.wikipedia.org/wiki/Bo%27_Selecta! ?


No. But if it's related to the urbandictionary definition...

I'm taking back the word!


a quick request (and something similar was discussed here - https://news.ycombinator.com/item?id=11626454)

could you add a representative example to show how we can use your framework with Postgres ? For example, psycopg vs psycopg2 vs psycogreen vs psycopg2-cffi vs aiopg ... I dont even know which one will work properly using your framework and how (and maintain async characteristics).


(Comment from a bystander). You probably want aiopg in order to maintain the async characteristics and use the "friendly" asyncio syntax.

There is also peewee-async (http://peewee-async.readthedocs.io/en/latest/) if you want an ORM using asyncio. (I've been meaning to try this myself, but haven't had an opportunity).


absolutely.. i am concerned about frameworks not fundamentally thinking about DB. With async, there is that additional problem of "this does not work in async mode". So I would hope the developers include database hookup as part of their unit test suite.


How is this different to Twisted?


As I understand, the asyncio module was inspired by Twisted.

Growler is, at its heart, a super minimal interface: passing (req, res) pairs through a pipeline of middleware (similar to node).

Since this is built with the standard library, all kinds of features, such as async DB access, will become available as the community builds them; and these may be added to your pipeline super cheap.

It's just an experiment.


It's asyncio native. Also I think syntactically it's different.


Like Flask + Express, specifically.


Could you clarify this vs aiohttp (aiohttp.web)?

http://aiohttp.readthedocs.io/en/stable/

Here's a good example of use: http://igordavydenko.com/talks/lvivpy-4/#slide-1


I have not used aiohttp, but it looks like their application must be initialized with a list of middleware_factories. I do not see any logic in choosing middleware - just a straight shot.

Growler apps just add functions to an internal list, and these may be 'mounted' on particular end points/http requests.

Also, in aiohttp it is up to your endpoint to create the response object and return it. While in Growler it is created by the application (or rather the GrowlerHttpResponder) and passed around the middleware - getting enhancements (res.render('foo') is not standard) and having callbacks registered (to, say, log ALL headers sent - something you want done only at the time of sending headers).

I think it would be possible (or at least I would like to get to the point) where you could actually mount other apps from other frameworks into a Growler app. So if there is blog software you like written in aiohttp, there is special middleware to transform growler requests into a aiohttp requests.

  import growler
  import aiohttp.web
  aio_blog_app = aiohttp.web.Application()
  ...
  app = growler.App("main site")
  app.use(GrowlerAiohttpMiddleware(aio_blog_app), path='/blog')
  ...

THAT would be kinda cool.


Does this work with uvloop?


Yes!

An initial test last night of a trivial growler app (i.e. one method that replies with a simple string) saw a median response time drop from ~200ms to ~80ms.

(using apachebenchmark, n=1000 c=100)


Wait, p50 using the built in asyncio loop is 200ms?

That seems incredibly slow.


How well would it work with, say, PostgreSQL? Would you need to use async wrappers like aiopg in your views?


For optimal performance, yes use aiopg or equivalent (it looks like there is an asyncio sqlalchemy project which might be nice.)

Any middleware may be a coroutine object, so it's as easy as `await res.render("whatever")` in an async function



do you have this in a github somewhere ? would love to check it out. Also would be nice to see this same thing with postgres DB access.


Does it blend ?


*Will


If you want this feature, but multithreaded and with hot-deploy + filesystem database clustered you can try: http://github.com/tinspin/rupy


Slightly disingenuous not to point out that's a Java app. I suspect the choice of language is a factor for a fair few people, maybe even the majority.


Yep, languages are like religion. But you can't hotdeploy any other language properly.


I'm not sure what exactly categorizes as "hot deploy" but how about erlang, elixir?


Rupy can switch code on the fly, meaning you can have users hitting your server and they will run the old code one request and new code the next. No other system I know of works like this, hot-deploying on a remote machine over HTTP seamlessly. Maybe erlang can do it but then again I can't write a 3D game in that language so Java still has the upper hand for my requirements.


Off the top of my head, if you write a custom MiddlewareChain class that exposed its own management port, you could inspect and modify the state of the chain at any time, checkboxes to disable middleware, change endpoints, insert debug middleware, etc.

I like this... definitely worth thinking about. Thanks, all!


Also, if you're talking about specifically replacing a source files (i.e. modules), one could hack around with https://docs.python.org/3/library/importlib.html#importlib.r.... I think it would work with enough indirection/factory functions.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: