TIL AWS doesn't initialize any blocks on your EBS volume until you read from it

[jobead]

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html

[jobead]

Basically if you never read a particular block from your volume after you initialize it, AWS never moves it out of S3 into EBS.

This can cause bizarrely slow performance for database or other random-read applications when you fire up a volume from a snapshot.

The official mitigation is to literally read every block once and then your volume will perform the way you expect:

  sudo fio --filename=/dev/xvdf --rw=randread --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize

[ebneter → jobead]

That's if you create the volume from a snapshot. You may want to clarify that.

This process can take a long time; sometimes you're better off actually rsyncing the data than using a snapshot.