I have a bit of a tangent question for more experienced back-end developers: where do you fit Redis (or other caching mechanisms) in the traditional MVC model? I haven't had a use-case for Redis yet, but I'd like to know how should I approach the architecture of an app using Redis.
We use it to crash our servers on larger customers, by using it to cache all our user entities, pull all of them out at runtime filter them in PHP then stampede ourselves when we clear the cache
Well, the use for a cache is caching expensive operations. Sorry if this is just stating the obvious, but I'm not sure how else to answer how it fits into traditional MVC operations. It could be a front-end HTTP cache (although you'd probably use a CDN for that instead). It could be caching something expensive to look up or calculate, for which it's fine to not have up-to-the-second-current value.
Many people at least in Rails also use Redis to hold a queue for background processing, where it ends up working well, although hypothetically many other things could too.
You can also use redis for all sorts of other stuff that isn't caching or a queue.
I use it for caching, temporary content, pubsub, and distributed locking.
It's been particularly useful in load balancers / proxies for authorization, managing csrf, and tracking session activity to auto-log out users. I do this with OpenResty.
In async job or internal rpc systems, I use pubsub and streams for distributing work, ensuring exactly once processing, and getting results back to the caller.
Redis is a flexible tool. Some things you could potentially do:
- This DB query is pretty slow. Cache the result in Redis.
- I'm using server-side sessions, but I have multiple servers. Where should I store session data where it'll be super fast, but all my servers can access it? You could use Redis.
- I need to do distributed locking. Use Redis
- Simple job queue? Redis.
- I'm processing a lot of inbound webhooks, and I want to avoid trying to process the same one twice. I'd like to keep a list of all webhooks I've seen in the last week, and quickly check if a new webhook is in the list, but it needs to be really fast. Redis.
Basically, Redis has support for some nifty data structures, and it's very fast. There's a lot of places in most apps where being able share a very fast list, set, map, stream, whatever between servers can be useful. Of course, all the above uses cases can be solved by other more specialised tools too, and in some cases better.
(That being said, it's so useful and generally applicable that you should be careful not to ignore fundamental issues. For example, if you have an unavoidably slow query, then by all means cache it in Redis. But if all your queries are slow because you forgot to add indexes, maybe you should fix that instead of using Redis to cache every query!)
Redis can be a very powerful tool, it can also be a sign that something has gone wrong.
Redis is a high performance key/value store with adjustable consistency. It can be used as a caching layer, and it can also do a solid job of representing a basic message queue. It typically fits in on the data layer to store computed results for fast retrieval, but it can also behave like a read replica (storing data from outside your domain).
That being said, when Redis becomes a critical part of your stack for performance reasons it usually means something has gone wrong in your data layer. I often times see teams use an antipattern of taking data from their database, changing it's shape to what they want, and then storing that new model in redis. If you're the only consumer of your data, then your data model is incorrect and you're using Redis to paper over that fact.
This doesn't sound like the worst way to reduce expensive reads of normalised data. Is the implication that it should have been solved with views, was incorrectly normalised, or that a document/key-value store should have been used other than Redis?
Edit: I suppose a hypothetical can go many ways, it could be a poor data access pattern. What was the root cause in some of your experiences?
If your data model is too slow to work for your use cases, your data model is wrong. If your data model can't be indexed by your data store, your data model or your data store is wrong. If you're storing large amounts of performance sensitive normalized key-value data in a column based SQL database, again something has gone terrible wrong.
Is this that bad? At our company we are using redis for real time recommendations. And this is our primary database for storing accumulated statistics about items and users. It has been important to us to have a quick way to share the data between the workers and we chose redis.
You're fine. The simplistic interface that Redis provides to sets, and therefore empowering pragmatic recommendation engines, is the perfect tool for this job.
There's nothing a priori wrong with using redis as a primary data store if your data is temporal and your durability needs are aligned with it's model.
We use redis for session storage, jobs with celery/redis-rq, view fragment caching, and certain query caching.
It can really give you a good boost in performance especially on frequently accessed pages were the content rarely changes - database queries are often "expensive" - so for frequently accessed data that doesn't change frequently - such as product descriptions - it can be a huge help.
STREAMS, best pubsub solution imho. I used it as a backing store for an MQTT frontend once, and also for generally coordinating worker processes to handle background tasks.
Just to add a bit on that, if you dynamically generate channel names according to a validatable naming convention that any consumer can predict (ideally with a client lib for generating them), you can do pretty complex message passing that doesn't blow up code complexity. Combine that with the locking and consumer groups built in, it's pretty much distributed computing "for free" even if stuck with multiprocessing for runtime scaling (e.g. Python/JS without the builtin concurrency or multithreading of VMs/hosted languages).
We use it for storing user sessions, for caching responses from a third-party API we access, and for imposing per-IP address rate limits on the use of that API. We've also previously used it for lightweight worker queues.
IF you want to use it for caching, THEN you would use it to cache stuff for your controller.
But, Redis is much more than caching. It supports all kinda of fun data-structures like sorted lists, timeouts, sets, pub-sub and more! You can almost think of it like memory that is held by another process. In that way, there are SO many uses cases.
I think a good example of this is session storage - just store the session with a ttl and now redis will automatically "expire the session" when the time is over.
We use it as the main database for low latency (less than 150ms) response time machine learning services. Store a pretty massive amount of data as well - close to 750GB.
However for your specific use case, considering a typical MVC web app with RDBMS data storage, you would add a check at the beginning of your Model method to return cached data if it exists, else go through the method execution and write the data to the cache just before returning it back to the controller. This way the cache would be 'warmed' on first call and data will be served directly from the cache (memory) next time till it is cleared, saving expensive disk I/O.
You need to be careful with caching inside models, because you want your models to reflect the current, completely up-to-date state of the application. Conceptually, the best place to do it is inside controllers, where you know when you can serve data which isn't completely up to date.
With skinny controllers, very often you have some specific places (eg service objects or similar layers) where the controller logic lives, and that is where you can do your caching.
Our main use case in $JOB for Redis is distributed locking. We usually do not need key-value storage and even if we do, we just go with DynamoDB instead.
There's lots of functionality that Redis provides beyond a key-value store. For example the data structures that it supports. These are very powerful on their own. Also I understand after considerable investments vendor freedom is a bit of an illusion, but you know, if you can choose an OS technology under the hood, it's effectively like not nailing shut a door, but still leaving it closed.
I actually use it as primary database on some parts of the mmorpg I’m developing, Redis actually has ACID capabilities so it is actually very suitable to use as primary database in gaming platforms. That being said main game server requires Mongodb due to sheer size of the data.
Well, if you have only a single instance that doesn't support threading it is trivial to get those properties, but what about durability? Do you realize you store all of the data in the RAM?
Redis can dump and fsync data to disk WAL every query, and can run multiple commands in a transaction, so it can provide the highest guarantees possible for a database.
The catch is that the data must fit in RAM, you can only use 1 core for query processing, there is no support for long-lived transactions and no built-in SQL support.
Obviously in an ideal world something like Redis would be useless since proper databases would cover all use cases, but unfortunately the state of database software is disastrous.
This is your second comment in this thread about Redis being in-memory. Redis can function in-memory only but it also has at least 2 persistence stories (rdb snapshots, aof logs), and out of the box is configured use rdb.
Redis got a feature called AOF, I use it for every single command to save the command history to disk.
This does obviously decrease write performance of the Redis quite a bit, but read performance is mostly fine.
It is all about how many writes versus reads you expect per second. My system expects at most 1 write per sec vs 100 reads, so performance is fine and this way it is ACID compliant.
RAM is cheap and lots of realistic data sets never exceed some tens of GBs, even in extreme scenarios. What's wrong with keeping (a copy of) all data in RAM? It's what makes Redis fast yet simple.
Here's a terrifying secret for you: if a dataset is small enough and used intensively enough, SQL databases like PostgreSQL will eventually have all of it in RAM. Better ditch those too!
For PHP applications Redis has been a must for storing sessions, especially when distributing load between multiple servers.
It is also used to collect data for Prometheus exporters, for example, for languages that don't share memory between requests.
As well as caching, we make really heavy use of Redis for Leaderboards for our games. The sorted sets are perfect for storing score along with the userid. Scanning the leaderboard, finding neighboring scores, etc are all really fast operations. This could probably be done with a number of other storage system but we already used Redis and we've never had a problem.
It is many things, but started as more of an architecture for user-data interaction, where the data is in a computer system - and the user wants to interact with it (the user in user-model-view-controller unfortunately fell off at some point).
As for where something like redis fits - I don't think it'd show up on the design that concerns mvc, no (it could be a cache inside model, inside controller - or even inside view.. Caching ui fragments for example?).
I always use it as my default session store. I build out large python backends and I also have been using it as a celery backend, and prefer for it to memcache for those type of tasks also.
Use it to cache certain request payloads that are guaranteed not to change for a certain amount of time (e.g. 1-minute stock market aggregates that only change once a minute).
If you look at single data type you can see how redis takes care about complexity(indicating the Big O notation) of each single data structure and operations.
Many devs use it for caching but in my opinion is super nice for evil-write applications.
I mostly work on solutions used in-house by the client. The most used app that I had created was used by maybe 50 people at the same time, and it was mostly manipulating spreadsheet-like data, so querying the database directly was fast enough.
I know broadly what Redis can be used for, I was just asking for some practical tips.
