System Design Concepts for Product Managers — Part 1

This is a system design series for Product Managers who should have a high-level understanding of System Design. We will first cover the basic concepts of System Design. This is part 1, and stay tuned for future updates.

1. Load Balancer

What is a Load Balancer?

Load Balancer: Load Balancer is a common term used widely when you are working on an application that serves enormous traffic. Let’s understand with an example:
Assume that Netflix has deployed several servers to handle the number of client requests. When we hit “Netflix.com”, how does Netflix know which server it has routed my request to? This is where the load balancer comes into the picture; load Balancers distribute the incoming traffic from the client efficiently across different servers and make sure that no server is getting overheated or overloaded. Reliability and availability are maintained by redirecting the incoming request to the available servers.

Credits: https://codeburst.io/

Load Balancing Algorithms
The load balancer can be broadly classified into two categories: application-level load balancing and network-level load balancing. Application-level load balancing resides on layer seven which is the application layer and is supported by a protocol like HTTP or HTTPS. Another kind of load balancer is the network-level load balancing, which resides on layer four, the Transport layer (If you cannot understand this, please look at the different network layers) and is supported by TCP/UDP.

Now let’s see the different algorithms for load balancing, in which the load balancer decides which server your request should route to.

1. Round Robin Algorithm: It is a simple algorithm in which the load balancer route the request to the server in a round-robin fashion to the next available server.
2. Least Connection: This is an algorithm in which the load balancer checks out of all the server nodes which node has the least number of connections and then routes the request to that particular node.
3. IP hashing: In this algorithm, we use hashing technique to generate a sequence using the client's IP address. It may also use other factors like source port, destination IP address, protocol etc.

2. Caching

What does it mean by Caching?

The cache is a hardware and software component that helps serve the data, which is either frequently requested or expensive to compute on, so the cache stores the computed response and helps to reduce the cost of computation. Let’s understand by example.

Suppose a client has requested an image from the server; the server will fetch the image from the Database and respond to the client. Suppose millions request the same image of users, so the server won’t fetch multiple times from the Database; in this case, we can cache the image on the server-side or cache the image on the client side.

Cache invalidation

In the Cache, data doesn’t remain there forever; the process of removing the old value from the Cache and updating the new value is called cache invalidation. So now the question arises how do we know that the Cache needs to be updated with the new data and the old one has to be deleted. We can have cache expiry time which decides the life cycle of the cached data, and this cache expiry time is decided depending on the use case.

Cache Eviction

One question always comes, “why can’t we store everything in Cache?” Cache memory is more costly than the other memory; if we overload our cache memory, the computation cost will be as bad as the original memory. So although Cache makes it fast for any request at the same time, it is costly.

So let’s assume we have set the expiry limit to the cache entry and our Cache is operating at its peak limit; now a new request has to be put into Cache; what should we do?
In such cases, any old cache data will be evicted so that the new cache data can go there. This is called cache eviction. There are multiple ways by which we can do cache eviction like “First in, First Out”, “Least Recently Used”, “Least Frequently Used”, etc.

Cache Pattern

When we have to use the Cache in the real-world application, there are different cache patterns available that we can use to leverage the power of Cache.

Cache Aside Pattern: This is the cache pattern in which Cache always talks to the application but never talks to the Database. So whenever a client request comes, the application checks it in the Cache; if it is not available(Cache miss), it will get the data from DB. The advantage is that Cache is not the single point of failure; it acts as a decoupled system. But their problem occurs when there is an update to the DB, and the data which is getting updated is also residing in the Cache. Two update operations have to be performed, one in Cache and one in Database.

Credits: https://codeahoy.com/

Read/Write Through Pattern: This is the cache pattern in which the Cache sits between the application and the Database. In this case, the application server always talks with the case, never with the DB. When the first request comes, there is always a cache miss, and the Cache fetches the data from DB.

Credits: https://codeahoy.com/

Credits: https://codeahoy.com/

Write Around Pattern: In this Cache pattern, the Cache sits between the application and the Database, but the application can talk with the Cache, and the application can talk with the DB.

Credits: Amazon AWS

It also depends on the requirement we will choose the cache pattern like heavy ready operation, a heavy write operation or avoid single point of failure.

-- Technomanagers