On: Logs, Tails, Long Tails (2013)

[contravariant]

Other fun stuff you can do:

- Easy weighting for observations (just put a weight in front of the corresponding term in the sum)

- Identify and isolate sufficient statistics (by breaking up the summation until you've got terms that only contain data, no parameters)

- Just add a quadratic term and pretend it's kinda Gaussian now (esp. fun if you pretend one of your parameters is generated with a Gaussian process).

- Take the expected value of it w.r.t. some other distribution (Cross entropy / Variational Bayes)

- Pretend the negative log probability is a Hamiltonian and add some momentum + kinetic energy terms (Hamiltonian MC)

- Same but skip the kinetic bit and pretend it's a thermodynamic ensemble (information field theory)

- Same but perform a Legendre transformation to get the Effective action / Free energy (ends up being the same as the KL-divergence)

[bollu → contravariant]

Shameless plug: I wrote some high quality (I hope) notes on HMC, because I couldn't find an explanation that was (a) rigorous, (b) explained the different points of view of MCMC, and (c) was visual! Here's the notes: https://github.com/bollu/notes/blob/master/mcmc/report.pdf

Feedback is much appreciated.

[amitport → bollu]

Looks cool. Also, I wish that when I started university somebody would have forced me to start a "notes" repo like yours. Looks great and very smart to have.

[contravariant → bollu]

Your code contains a small bug, in the leapfrog HMC function the line pnext = -p should be pnext = -pnext. Also this line is only of theoretical importance, it has no effect on the final result (unless you use a weird asymmetric kinetic energy).

[bollu → contravariant]

Nice, thank you for the bug report. fixed!

[pixelpoet → bollu]

Oh man, these are great notes, very succinct! I'm not much into proofs, but finally picked up the essence of Hamiltonian M-H thanks to this. (Also, I can't imagine what kind of genius it takes to rigorously derive such a method...)

[kgwgk → bollu]

Minor correction: it's "detailed balance" (not "detail balance").

[bollu → kgwgk]

Thank you, Fixed!

[rawoke083600 → contravariant]

I love your real world usage and dare I say "tricks" ?

If I want to learn more about "applied statistics" best source you would recommend that is not dry as bone ?

[contravariant → rawoke083600]

Well, my method so far has been to at least be familiar with some of the theoretical stuff and then try to figure out how to apply it in depth when the opportunity to do so arises. Not sure if there are references where you can directly jump to the applied bit.

[diego898 → contravariant]

Would you mind saying some more about these topics? Or provide some references? I’d like to learn a bit more on all of these!

[contravariant → diego898]

Most of those things are stuff you'll encounter if you read up on Hamiltonian MC and information field theory, but it might take some additional reading to get all the required background knowledge.

The first two are just tricks really. For the weighted estimates you just go from:

    log(P) = \sum log(P_i)

to

    log(P) = \sum w_i log(P_i)

and for the sufficient statistics you get derivations like:

    log(P) = \sum_i (1/2) (x_i - m)^2
           = \sum_i (1/2) x_i^2 - x_i m - (1/2)m^2
           = (\sum_i (1/2) x_i^2) - (\sum_i x_i) m - (1/2)m^2

to show that (\sum_i (1/2) x_i^2) and (\sum_i x_i) fully determine the posterior distribution.

[ravar → diego898]

I wonder how much he knows, interpreting it as a thermodynamic ensemble is just going back to thinking about probabilities really. Information field theory is something very different. The cheesy explanation for it is that it is just quantum field theory in imaginary time.

[contravariant → ravar]

>it is just quantum field theory in imaginary time.

Quantum field theory is just thermodynamics in imaginary time, not sure why you object to calling it a thermodynamic ensemble. Besides, I could hardly introduce a subject as 'quantum field theory in imaginary time' now could I?