Hey cryptoquick, I remember you from a while back. I'm https://twitter.com/aphelionz, one of the maintainers. Nice to see you :)
You're not wrong about the memory usage. We decided early on that the tradeoff would be more RAM than to take the performance hit with the I/O (either to and from IPFS or to and from the filesystem in the case of SnapDB).
That being said, I have been experimenting with removing indexes like the ones you pointed out in favor of generators that emit the values as they are traversed. There are two main efforts here, one is feasible and the other is more difficult.
1. The `entryIndex` inside of ipfs-log can probably go, and the ipfs-log#traverse function can be made into an async generator function that passes the oplog values up to the store
2. The indexes you linked to that hold the calculated STATE are harder to get rid of - they could be persisted to IPFS or the file system as well
Open to ideas on #2. SnapDB might work for all I know, as I haven't attempted it yet.
I can't speak to Electron, React, Redux, Iced, etc, but my guess is there are optimizations one can do there as well.
You're not wrong about the memory usage. We decided early on that the tradeoff would be more RAM than to take the performance hit with the I/O (either to and from IPFS or to and from the filesystem in the case of SnapDB).
That being said, I have been experimenting with removing indexes like the ones you pointed out in favor of generators that emit the values as they are traversed. There are two main efforts here, one is feasible and the other is more difficult.
1. The `entryIndex` inside of ipfs-log can probably go, and the ipfs-log#traverse function can be made into an async generator function that passes the oplog values up to the store 2. The indexes you linked to that hold the calculated STATE are harder to get rid of - they could be persisted to IPFS or the file system as well
Open to ideas on #2. SnapDB might work for all I know, as I haven't attempted it yet.
I can't speak to Electron, React, Redux, Iced, etc, but my guess is there are optimizations one can do there as well.