Hi - I'm a lead on the Steampipe (https://steampipe.io) project. Rate limits are definitely an interesting challenge, but actually less of an issue for AWS than many of our other plugins.
Because rate limits are separate across accounts, regions and services, the parallel queries work really well in those cases.
We do sub-API queries for tables, but only when the column is requested. So, "select name from aws_s3_bucket" just does a list call while "select * from aws_s3_bucket" does multiple API calls per row. These sub-API calls are the main potential source of rate limits since they hit the same API [1]. BTW, Cloud Control from AWS is actually much more subsceptible to this problem! [2].
We also use a custom backoff algorithm that is fast then slow to give good speed in the usual cases and ensure results if throttling ramps up [3].
Finally, we automatically cache results in memory between requests and can always save results into materialized views or similar to avoid repeated calls in larger cases.
TBH, live querying of APIs has proven so much more effective than we even hoped when we started!
A few years ago I built something similar to this, in that it used the JSON API definitions to automatically scrape the entire AWS API surface area.
But I was doing it every 5 minutes and storing the result in postgres instead of it being on demand. I very quickly received a call from our account rep asking for it to not query particular services as frequently.
Apparently some of the newer services at the time I was scraping were not as optimized in terms of response caching or metadata queries as they should have been.
(Note: apparently it wasn't the overall frequency that was the problem, it was a few services that returned paginated results and fetching every page every 5 minutes was causing issues.)
Most of the AWS SDKs have built in support to retry SDK calls using exponential backoff + jitter. However it is not obvious unless you instrument logging to see the individual attempts, nor is it obvious how to tweak the number of retries (default 3 for everything except dynamodb) or the backoff parameters. Java, Golang, and nodejs have it for sure - PHP does not.
