I am very interested in data ingestion. I develop a desktop data wrangling tool in C++ ( Easy Data Transform ). So far it can import files in various formats (CSV, Excel, JSON, XML etc). But I am interested in being able to import from databases, APIs and other sources. Would I be able to ship your CLI as part of my product on Windows and Mac? Or can someone suggest some other approach to importing from lots of data sources without coding them all individually?
hmm, that's an interesting question, I don't know the answer to be honest. are you able to run external scripts on the device? if so, you might be able to install & run ingestr with a CSV destination (which I released literally 2 mins ago), but that seems like a lot of work as well, and will probably be way slower than your C++ application.
I can start a CLI as a separate process. But ingesting to CSV and then reading the CSV would be slow. Maybe it would be better to ingest into DuckDB or in memory in Arrow memory format. If anyone has any other suggestions, I am all ears.
