Great start. Please develop a plugin I can secretly add to my Mother's browser to dial down the outrage she reads on the fly! She'll notice 'redacted' ;)
The main issue is that its very slow and expensive to browse the internet like this. The LLM will only perform well if you have it do chain of thought reasoning, and that has a latency hit because of a longer generation.
Gooey.AI | Full Time | REMOTE / ONSITE | Bengaluru | http://gooey.ai/
We are building a GenAI platform that abstracts all the latest LLMs, animation tools, speech reco, etc from OpenAI, Stability, Google, Meta, etc.
One of our projects - https://farmer.chat - was demoed at the UN General Assembly in April 2023.
The way to solve session management is to just store the state in html forms and send it back to a stateless http endpoint. That way you get a website that behaves the way all websites work, and can scale easily.
The trouble is though that you also have state in Python which can’t necessarily be represented in Javascript, such as a database connection. You could pickle it, but accepting a user-provided pickled value is a security risk. Also, doing so would leak database credentials, and recreating the state on the server every event is expensive anyway (especially when the state is something like a pandas dataframe). It’s better to just have one dedicated machine per application session that holds the state for the duration of the app.
Don't use pickle. Use JSON for stuff you can't represent in HTML forms.
For database connections, large datasets, etc (stuff you can't send off to the browser) use redis.
This is a trade-off between DX and UX. Holding session state means you need sticky session routing, and restarting servers kills your user's sessions. Plus imposing a websocket on your users is a cardinal sin, makes scaling incredibly hard, makes page load times abysmal, makes disconnects a nightmare for both you and your user.
Its fine if you're creating something like stable diffusion web ui though, which is meant to be a single user app.
I find these dogmas to be presumptuous. For the use case, Pickle would be fine. Scaling was essentially a non-concern… it was an internal tool behind a VPN and the usage was below a dozen (and at most would have grown to dozens).
JSON is an alright storage format but it can’t do infinity, nan, or many other values important to Python. Also, you would have to write a reifier and serializer for every value. These simply create a problem rather than solving one.
Not optimizing DX would be a mistake when the whole purpose of the project is to improve the DX by allowing the users to not write JS/React. Once the DX starts becoming that cumbersome you might as well just avoid it by switching back to native React.
ReactPy dev here. We haven't actually landed on how we want to solve this problem yet. We have some ideas though. Would be curious to hear your thoughts on this issue: https://github.com/reactive-python/reactpy/issues/828
Wish I could help, but I never got as far as auth, and it sounds like ReactPy takes a bit of a different direction anyway. Looking forward to digging in but just haven’t gotten to it yet. Thanks for surfacing though!
I can't wait for the morally concerned to rail against Stable diffusion once they find the /b/ threads and AI is going to get locked down... Its going to happen. "This AI tool enables sick PREDATORS!!"
