Hacker News new | past | comments | ask | show | jobs | submit login

Of course. As others here point out, the hypervolume inside an n-dimensional hypersphere grows as the nth power of a linear increase in radius. In high dimensions, tiny increases in radius cause hypervolume to grow by more than 100%. The concentration of hypervolume is always highest at the edge.[0]

The theoretical tools (and intuitions) we have today for making sense of the distribution of data, developed over the past three centuries, break down in high dimensions. The fact that in high dimensions Gaussian distributions are not "clouds" but actually "soap bubbles" is a perfect example of this breakdown. Can you imagine trying to model a cloud of high-dimensional points lying on or near a lower-dimensional manifold with soap bubbles?

If the data is not only high-dimensional but also non-linearly entangled, we don't yet have "mental tools" for reasoning about it:

* https://medium.com/intuitionmachine/why-probability-theory-s...

* https://news.ycombinator.com/item?id=15620794

[0] See kgwgk's comment below.




> Density is always highest at the edge.

More precisely: it is the mass that is “concentrated” at the edge, not the density. In the Gaussian case the distribution “gets more and more dense in the middle” regardless of the number of dimensions. However, in high dimensions the volume in the middle is so low that essentially all the mass is close to the surface of the hypersphere.


What i found quite surprising in that context, is that the volume of the n-dimensional ball for any finite fixed radius goes to zero as n goes to infinity (see, for example, section "high dimensions" of https://en.m.wikipedia.org/wiki/Volume_of_an_n-ball)


There are multiple good (intuitive) explanations for this here: https://math.stackexchange.com/questions/67039/why-does-volu...


Yes, you're right. For clarity's sake, I've updated my comment.


Thanks for this comment as it manages to explain something non-trivial in a very concise way, which is not something I find often.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: