Sure, but that's the least helpful possible response you could have made. We've got an observation:
> I never needed to scan the directory to find "all users"
> lookup by name, and creating/deleting files, is quick, even if you have millions of files.
And a question: given these observations, where do the benefits of filesystem fanout come from? Is it not true that looking up a file by name is fast no matter how many other files sit in the same directory? Is HN doing something weird?
You can't answer the question "where do the performance benefits come from?" by saying "look, the performance benefits exist".
> You can't answer the question "where do the performance benefits come from?" by saying "look, the performance benefits exist".
I think he is trying to say is that the parent poster's observations must be wrong. After all, we are talking about an unsubstantiated claim ("there's no benefit to fanning out files") that directly contradicts another claim which we have data for ("HN is 5x faster after fanning out files").
Again, when someone asks why they're wrong, it's not useful to tell them "but you're wrong". Parent poster already acknowledged that the combination of his ideas and the facts on the ground didn't make sense. What good does it do anyone to repeat it back to him?
I guess when you're absolutely sure that you're right, but the observation proves you wrong, you have to be prepared to consider the possibility that you're wrong.
The comment I was replying to was saying that the file system takes care of it automatically, so there's no purpose to arranging millions of files into directories. I'm not going to speculate how it all works under the hood.
> I never needed to scan the directory to find "all users"
> lookup by name, and creating/deleting files, is quick, even if you have millions of files.
And a question: given these observations, where do the benefits of filesystem fanout come from? Is it not true that looking up a file by name is fast no matter how many other files sit in the same directory? Is HN doing something weird?
You can't answer the question "where do the performance benefits come from?" by saying "look, the performance benefits exist".