A Large System Architecture
Let me continue what I was writing before over designing systems. Though this blog is supposed to be a management related blog, forgive me for this digression. Nothing much - its eating my brain for the last few weeks.
Let's consider a web interface. How do you want to develop it? I am not going into the technicalities of the stuff, this is a workflow how to approach the problem here.
There are obviously some components missing, but the general approach looks something such.
A simple end to end stack - web front end, web backend, services, database. Example: React, Django, Spring, MySQL
Expand the services - each service is now it's own - a path towards microservices
Because we are having multiple services, have a Gateway and a Discovery Service for the web server to communicate with the services. Example: Zuul, Apigee for Gateway and Zookeeper, Consul for Discovery Service
A load balancer to distribute the heavy load - both on server and client side. Examples: Nginix, Microsoft IIS.
You need a centralized logging and tracing to monitor system health. Examples: Jaeger for tracing, Elasticsearch for logging, Prometheus for collecting metrics and Grafana/Kibana to visualize the data.
Introduce caching with, say, Redis, for easy access of frequently used data and asynchronous processing to manage load better by queueing data to be processed using, say, RabbitMQ.
Containerize the solution using Docker and Kubernetes, and if need be, go for a cloud hosting. You can do this from Stage 2 itself.
Now, we have reached the realm of really large systems. Split the database into two parts - OLTP to interact with the application and Data Warehouse to archive old data and access it for analytics.
But then, in rare cases where your OLTP size crosses 1 TB or so, RDBMS starts to see serious performance degradation. At that point, you jump into NoSQL solutions like Cassandra or ScyllaDB.
This means you are looking at the problem from two angles -
What stage you are in, and to what level are you expecting your application will scale to?
How can you reduce the pain of scaling up? For example, splitting of services or containerizing them or replacing Django with Ruby on Rails should be thought about well in advance.