I would love to see some longer-term measurements. I get the impression that this benchmark was doing a one time measurement. But the one big problem I have with cloud services is I/O performance over a longer time period.
We read so many times about very unreliable performance. Sometimes it's ok and sometimes it's really, really bad.
Without any kind of time period to continuously run a benchmark in, this doesn't really help. For all we know, the first placed service was just having a good day and the last placed a very bad one.
As a hobby, I've been running long-term benchmarks on a handful of cloud services, for exactly the reason you suggest -- performance varies quite a bit over time. You can browse through some of the data at (http://amistrongeryet.com/dashboard.jsp). The UI on this site is abysmal, and it only shows 30 days of data (I've actually collected almost two years), but it still gives some flavor of how much variability there is. For instance, check out this graph of SimpleDB reads: (http://amistrongeryet.com/op_detail.jsp?op=simpledb_readInco...). I've blogged on the data from time to time; for instance, (http://amistrongeryet.blogspot.com/2010/04/three-latency-ano...).
If there's interest, I'll work on making this data more accessible, adding documentation, and providing access to the full two years. In any case it's limited by the fact that I'm only probing AWS and Google App Engine, and only one or two instances of each. What I'd really like to do is open this up to crowdsourcing -- as I discussed a while back at (http://amistrongeryet.blogspot.com/2011/07/cloudsat.html). If anyone is interested in participating in a project like this, let me know!
We read so many times about very unreliable performance. Sometimes it's ok and sometimes it's really, really bad.
Without any kind of time period to continuously run a benchmark in, this doesn't really help. For all we know, the first placed service was just having a good day and the last placed a very bad one.