Looking at it the other way, Black 4.0 reflects 0.05% of the light, and Vantablack 0.035%. So Black 4.0 reflects about 40% more light. The difference is not that small.
On the other hand, Vantablack is highly toxic (similar to asbestos), while Black 4.0 should be quite a lot safer.
Right but 40% of something very small is also very small. If you are experiencing the two blacks in a normal ambient environment around lighting and other non highly light absorbent surfaces, they will look extremely similar.
No, the logarithmic effect bottoms out towards the low part of the scale. If you're somewhere very dark, your eye adjusts so that you get back to the middle of the range. But if you're somewhere light and just looking at black paint, it doesn't.
On the other hand, Vantablack is highly toxic (similar to asbestos), while Black 4.0 should be quite a lot safer.