Show HN: Concurrently Logging HTTP Requests

[PabloSichert]

https://github.com/PabloSichert/concurrency-logger

[mrcactu5]

beautiful ASCII visualization! In theory class we called it "interval arithmetic"

https://en.wikipedia.org/wiki/Interval_arithmetic (2+2)-free posets, ascent sequences and pattern avoiding permutations https://arxiv.org/abs/0806.0666

[PabloSichert → mrcactu5]

Sorry, I couldn't quite follow - could you break it down in simple sentences what the connection between those links you provided and the visualization is?

The coloring depending on response time happens in intervals, that much I can say.

[mrcactu5 → PabloSichert]

I am not saying anything very deep. We can try to count all the different ways 5 (or 100) people can have sessions on a website

  #1 |--------------------|
  #2        |---------------|      |---------|
  #3   |------------|          |---|
  #4      |----|             |------|      |--|
  #5   |---------------------|

Each of these channels have overlapping event. Some sessions happen strictly before the other. Sometimes, one logs on before the other one logs of. Etc. The more channels we have the more complex behavior. That is all the paper is saying.

Sorry about that.

[PabloSichert → mrcactu5]

Thanks for making that example!

Assuming I am looking at your sketch with the x-axis being time, I just want to note that specific to this library there won't be two events (request start, end or log) at the same time. (The "concurrency" here comes from coroutines, not parallel threads)

[mrcactu5 → PabloSichert]

My mistake. I only see a few outcomes then. #1 happens then #2

  #1 |------|
  #2             |--------|

and #2 happens before #1

  #1                  |-------|
  #2   |---------|

or #1 starts then #2 starts:

  #1   |-------------|
  #2           |------------|

or #1 starts before #2 and finishes before #2

  #1  |---------------------|
  #2       |----------|

For two channels are are 8 possibilities in all. https://oeis.org/A079144

[fizzbatter]

This is really cool! I'm tempted to try and write something generic for this same concept, so i can use it in whatever lang and logging framework desired.

Concept: A simple pattern match against each line to link their respective contexts. Eg, if you log each request with an id `[timestamp] [request-id] some module: some message`, then theoretically you could parse a log to show visual relation between all request ids.

Furthermore, you could filter out request ids with generic tools like grep/etc.

There must be logging viewer tools that work generically like this though, so perhaps i'd be wasting my time :)

[PabloSichert → fizzbatter]

I'd be interested in such a solution, also in regard to the input from another commenter to make this work with nginx.

If you have a concrete idea and eventually a project on GitHub, feel free to contact me! Would like to contribute

[stuhood]

Nice. Could imagine this as a realtime renderer for http://zipkin.io/

[mholt]

This is cool! I wonder how it looks for really busy servers (hundreds or thousands of req/sec).

[PabloSichert → mholt]

That's an interesting question!

I wrote this mainly for the need of better debugging in development, and never inspected live traffic on a production server.

If you balance your application across servers/workers and keep the log on a per-worker basis I think there shouldn't be a problem in doing so.

Otherwise - you could make a middleware that collects requests that require attention (response time above some threshold, non-200 status codes, errors).

Then you could "replay" the logger on those requests.

(You can't do it live, since you can't know in advance which requests will be of interest.)

Maybe I find a way to incorporate this into the project, so that you can use it out of the box :-)

[nugator]

Really nice! I wish it could hook up to for example an Nginx as a command line tool.