Notes of the book System Desgin Interview - An Insider's Guide
Add Unique ID chapter
Gaurav Kumar849401c31ec9
diff --git a/consistent-hashing/Readme.md b/consistent-hashing/Readme.mdindex e267d56..0f55e84 100644--- a/consistent-hashing/Readme.md+++ b/consistent-hashing/Readme.md@@ -24,27 +24,35 @@ Consistent hashing ensures that only a fraction of keys are remapped when server### Key Concepts1. **Hash Space and Ring:** The hash space forms a continuous ring, with hash values distributed from `0` to `2^160-1` (e.g., using hash function like SHA-1). By connecting both ends we get a ring.-+ <p align="center"><img src="./images/hash-ring.png" style="margin-left: 40px; margin-top: 20px; margin-bottom: 20px;" alt="Server hashing" width="400">+ </p>- Using the same hash function f, we map servers based on server IP or name onto the ring.-- <img src="./images/server-ring.png" style="margin-left: 40px; margin-top: 20px; margin-bottom: 20px;" alt="Server hashing" width="400">++ <p align="center">+ <img src="./images/server-ring.png" style="margin-left: 40px; margin-top: 20px; margin-bottom: 20px;" alt="Server hashing" width="400">+ </p>1. **Server Lookup**- A key's server is determined by traversing clockwise on the ring until a server is found.+ <p align="center"><img src="./images/server-lookup.png" style="margin-left: 40px; margin-top: 20px; margin-bottom: 20px;" alt="Server hashing" width="400">+ </p>2. **Adding and Removing Servers**- Adding a server redistributes only nearby keys. Only a fraction of keys are redistributed to the new server.+ <p align="center"><img src="./images/adding-server.png" style="margin-left: 40px; margin-top: 20px; margin-bottom: 20px;" alt="Server hashing" width="400">+ </p>- Removing a server affects only the keys in its range. Only keys from the removed server are reassigned to the next server clockwise.+ <p align="center"><img src="./images/removing-server.png" style="margin-left: 40px; margin-top: 20px; margin-bottom: 20px;" alt="Server hashing" width="400">-+ </p>## Challenges and Solutions### Two Issues in Basic Approach@@ -55,7 +63,9 @@ Consistent hashing ensures that only a fraction of keys are remapped when server- Each server is represented by multiple virtual nodes on the ring uniformly distrubuted on the ring.- Virtual nodes improve key distribution and balance load. As the number of virtual nodes increases, the distribution of keys becomes more balanced. This is because the standard deviation gets smaller with more virtual nodes, leading to balanced data distribution.+ <p align="center"><img src="./images/virtual-nodes.png" style="margin-top: 20px; margin-bottom: 20px;" alt="Server hashing" width="400">+ </p>## Affected KeysWhen servers are added or removed:@@ -64,13 +74,16 @@ When servers are added or removed:added node) and moves anticlockwise around the ring until a server is found (s3). Thus, keyslocated between s3 and s4 need to be redistributed to s4.+ <p align="center"><img src="./images/server-addition.png" style="margin-top: 20px; margin-bottom: 20px;" alt="Server hashing" width="400">+ </p>- **Removed Server:** Affected keys are those between the removed server and its predecessor. In the following example when a server (s1) is removed, the affected range starts from s1(removed node) and moves anticlockwise around the ring until a server is found (s0). Thus, keys located between s0 and s1 must be redistributed to s2.+ <p align="center"><img src="./images/server-removed.png" style="margin-top: 20px; margin-bottom: 20px;" alt="Server hashing" width="400">-+ </p>## Benefits of Consistent Hashing- **Minimized Redistribution:** Only a fraction of keys are reassigned.diff --git a/unique-id-generator/Readme.md b/unique-id-generator/Readme.mdnew file mode 100644index 0000000..be84141--- /dev/null+++ b/unique-id-generator/Readme.md@@ -0,0 +1,97 @@+# Chapter 7: Design a Unique ID Generator in Distributed Systems++## Introduction+This chapter addresses the challenge of designing a **unique ID generator** for distributed systems. Traditional auto-increment keys are unsuitable in distributed environments due to scalability and synchronization challenges. The focus is on creating unique, sortable, 64-bit numerical IDs that meet the following requirements:+- IDs must be **unique** and **ordered by date**.+- IDs must fit within **64 bits**.+- The system should generate **over 10,000 IDs per second**.++---++## Step 1: Understanding the Problem+### Basic Requirements+- IDs must be unique and numerical and should fit in 64 bi.+- IDs increment with time but not strictly by `+1`.+- IDs should be sortable by date.+- System must handle high throughput (10,000 IDs/sec).++---++## Step 2: High-Level Design Options+### 1. Multi-Master Replication+- **Approach:** Use database `auto_increment` with step increments (e.g., `+k` for k servers).++ <p align="left">+ <img src="./images/multi-master.png" alt="Multi Master" width="300">+ </p>++- **Drawbacks:**+ - Hard to scale across data centers.+ - IDs do not always increase with time.+ - Scaling issues when servers are added/removed.++### 2. UUID (Universally Unique Identifier)+- **Approach:**+ - Generate 128-bit unique identifiers independently on each server using UUID.+ - UUIDs can be generated independently without coordination between servers++ <p align="left">+ <img src="./images/uuid.png" alt="UUID generator" width="500">+ </p>++- **Advantages:**+ - No coordination needed between servers.+ - Scales easily with web servers.+- **Drawbacks:**+ - Exceeds 64-bit requirement.+ - IDs are not sortable by time and may be non-numeric.+++### 3. Ticket Server+- **Approach:** Use a centralized database server to increment and assign IDs.++ <p align="left">+ <img src="./images/ticket-server.png" alt="UUID generator" width="500">+ </p>++- **Advantages:**+ - Simple to implement for small-scale systems.+ - Generates numeric IDs.+- **Drawbacks:**+ - Single point of failure.+ - Synchronization challenges in multi-server setups.++### 4. Twitter Snowflake Approach+- **Approach:**+ - Divide IDs into sections to ensure uniqueness and scalability.+ - **Sign Bit (1 bit):** Always `0`, potentially distinguishing signed and unsigned numbers.+ - **Timestamp (41 bits):** Milliseconds since a custom epoch (Twitter's default is `1288834974657`, equivalent to Nov 04, 2010, 01:42:54 UTC). Ensures IDs are time-ordered.+ - **Datacenter ID (5 bits):** Identifies up to `2^5 = 32` datacenters.+ - **Machine ID (5 bits):** Identifies up to `2^5 = 32` machines within each datacenter.+ - **Sequence Number (12 bits):** Tracks IDs generated on a machine within the same millisecond, supporting up to `2^12 = 4096` IDs per millisecond. The sequence resets to `0` every millisecond.++ <p align="left">+ <img src="./images/twitter-snowflake.png" alt="Snowflake approach" width="500">+ </p>+ <p align="left">+ <img src="./images/snowflake-id-breakdown.png" alt="Snowflake ID breakdow" width="500">+ </p>++- **Advantages:**+ - **Scalability:** Handles 10,000+ IDs per second across multiple servers.+ - **Time-Order:** Ensures IDs are sortable by time.+ - **Decentralization:** No single point of failure.+++## Step 4: Additional Considerations+### 1. Clock Synchronization+- **Challenge:** ID generation assumes synchronized clocks across servers.+- **Solution:** Use **Network Time Protocol (NTP)** to minimize drift.++### 2. Section Length Tuning+- Adjust section sizes (e.g., fewer sequence bits, more timestamp bits) based on use case.++### 3. High Availability+- ID generators are mission-critical and must be fault-tolerant.+- Consider redundancy and failover mechanisms.+diff --git a/unique-id-generator/images/multi-master.png b/unique-id-generator/images/multi-master.pngnew file mode 100644index 0000000..4a75299Binary files /dev/null and b/unique-id-generator/images/multi-master.png differdiff --git a/unique-id-generator/images/snowflake-id-breakdown.png b/unique-id-generator/images/snowflake-id-breakdown.pngnew file mode 100644index 0000000..ff4dcdbBinary files /dev/null and b/unique-id-generator/images/snowflake-id-breakdown.png differdiff --git a/unique-id-generator/images/ticket-server.png b/unique-id-generator/images/ticket-server.pngnew file mode 100644index 0000000..da4b825Binary files /dev/null and b/unique-id-generator/images/ticket-server.png differdiff --git a/unique-id-generator/images/twitter-snowflake.png b/unique-id-generator/images/twitter-snowflake.pngnew file mode 100644index 0000000..4d16b85Binary files /dev/null and b/unique-id-generator/images/twitter-snowflake.png differdiff --git a/unique-id-generator/images/uuid.png b/unique-id-generator/images/uuid.pngnew file mode 100644index 0000000..8ca43c9Binary files /dev/null and b/unique-id-generator/images/uuid.png differ