I learned a lot by writing "prototypes." If you look at system design and architecture from afar, they're all very simple patterns that can be written up in a self contained python or javascript program:
* RPC = function call
* API design = function signature
* data schema = simple structs
* message queue = list
* load balancer = simple facade object
* database = hash table / in memory SQLite
Once you have a rough sketch in place, you can run your prototype and test different behaviours.
It is a bit "tricky" to tackle asynchronous and concurrency, which you should do from the start. I prefer to do everything based on callbacks, and run my simulation from a static input list of events, ticking every component with a tick() method. This way I can generate different scenarios, save them to JSON, replay them.
What a wonderful comment. I've bookmarked this page specifically to refer back to your list of simplifications and would encourage you to please begin on your book.
I've used probability distributions to generate the incoming events, but the simulation is always deterministic. In the netflix case for example, I would generate 40% of users that would be playing things around 8am with a normal distribution, and 40% that would play around 8pm, and then 20% that play at random times of the day, etc...
I also played with pymc3 where you can just place plain python inside your probabilistic model and MCMC the heck out of it. It's slow but you can use some real world input and see what distributions arise at each point of the graph.
* RPC = function call
* API design = function signature
* data schema = simple structs
* message queue = list
* load balancer = simple facade object
* database = hash table / in memory SQLite
Once you have a rough sketch in place, you can run your prototype and test different behaviours.
It is a bit "tricky" to tackle asynchronous and concurrency, which you should do from the start. I prefer to do everything based on callbacks, and run my simulation from a static input list of events, ticking every component with a tick() method. This way I can generate different scenarios, save them to JSON, replay them.
Here's some examples:
- https://github.com/wesen/summer-pasture-netflix
cell.py is a kubernetes prototype (blackboard system), while app.py is the start of the netflix event notification system.
I am thinking of writing a course / book about the subject, and have a bunch of (wild) notes here: https://publish.obsidian.md/manuel/ZK/Structure+Notes/2+-+So...