Yoshinori is the engineer who developed MHA, MySQL replication auto failover. It's essentially a perl script that ensures a master is always available and functional. This is so DBAs don't have to spend hours manually fixing a broken master/slave replication.
Derp, yes. It's not an auxiliary index. However, it still seems weird for a full table scan to be implemented this way instead of whatever's convenient given the page layout. Why does it guarantee that the results will be in PK order?
The problem is compounded by the fact that with the small data sets people tend to use to test during development, things are usually returned in insertion order.
Oh, it can often be worse. People sometimes try to do tricky things like running totals by relying on implicit order in SQL queries. Often they are trying to squeeze as much performance out of the database server as possible (usually SQL Server) by avoiding cursors.
My favourite ever SQL Server hack was one done by a guy called Jeff Moden, who came up with a running total update that relied on a specific quirk of SQL Server clustered indexes. The amount of effort he put into it is quite remarkable! [1]
One issue might be that InnoDB by default uses a single idb file where all tables are stored. Further, this file also never shrinks even if you delete rows or drop indecies. So basically if page n contains your data page n + 1 might be useless to you, so InnoDB looks at pages by pk instead to only read pages it needs. (All this is my wild speculation.)
I don't know the ins and outs of InnoDB, but some databases match the physical order to the primary key order. Therefore this would probably make sense in that case.
Unless, of course, the secondary index has all of the columns you need to execute your query. (Sometimes it can help to add an extra column or two to your secondary index if you want to avoid the extra lookup.)
sqlite3 stores rows in an integer primary key index. If you do not specify an "integer primary key" column, it synthesizes one behind the scenes (essentially a rowid) and your primary key lookups end up going through two indexes.
Only because it was the first thing that popped into my head, you can use async file IO in Java 7+ with the AsynchronousFileChannel class[1]
Currently digging through Google to see if I can find how those async calls are implemented on Linux/Windows to see if you can expect similar speedups in performance as Yoshinori saw in this article.
