Notes of the book System Desgin Interview - An Insider's Guide
Add Google Drive Chapter
Gaurav Kumar62e2b55557ca
diff --git a/google-drive/Readme.md b/google-drive/Readme.mdnew file mode 100644index 0000000..557da1e--- /dev/null+++ b/google-drive/Readme.md@@ -0,0 +1,230 @@+# Chapter 15: Design Google Drive++## Introduction+Google Drive is a cloud-based file storage and synchronization service that allows users to store, access, and share files from various devices. This chapter discusses designing a scalable system with the following features:+- **File Upload and Download**+- **File Sync Across Devices**+- **File Sharing**+- **File Revision History**+- **Notifications for Edits, Deletes, and Shares**++---++## Step 1: Understanding the Problem++### Key Requirements+#### Functional Requirements:+- Upload and download files.+- Sync files across multiple devices.+- Maintain file revisions.+- Enable file sharing with permissions.+- Send notifications on file edits, deletions, and shares.++#### Non-Functional Requirements:+- **Reliability:** Data loss is unacceptable.+- **Fast Sync Speed:** Avoid user impatience with delayed syncing.+- **Bandwidth Efficiency:** Minimize unnecessary data usage.+- **Scalability:** Handle 10 million daily active users (DAU).+- **High Availability:** Operate seamlessly during server failures or network issues.++### Constraints and Assumptions+- Users get **10 GB free space**.+- Maximum file size: **10 GB**.+- Average file upload size: **500 KB**.+- Upload frequency: **2 files per day per user**.+- Total storage required: **500 PB**.++---++## Step 2: High-Level Design+### Single-Server Setup+A basic setup includes:+1. **Web Server:** Handles uploads and downloads.+2. **Metadata Database:** to keep track of metadata like user data, login info, files info/+3. **Storage Directory:** Holds files organized by namespaces.+++<div style="margin-left:3rem">+ <img src="./images/namespaces.png" alt="Namespaces" width="300" />+</div>++- A web server and a directory called drive/ is set up as the root directory to store uploaded files.+- Under drive/ directory, there is a list of directories called namespaces.+- Each namespace contains all the uploaded files for that user.+- Each file or folder can be uniquely identified by joining the namespace and the relative path.+++This design serves as a starting point but is inadequate for scaling.++#### APIs+1. **Upload a file to Google Drive:** Two types of uploads are supported+ - Simple upload: Used when file size is small.+ - Resumable upload:+ - Endpoint: https://api.example.com/files/upload?uploadType=resumable+ - Send the initial request to retrieve the resumable URL.+ - Upload the data and monitor upload state+ - If upload is disturbed, resume the upload.+2. **Download a file from Google Drive:** To download a file+ - Endpoint: https://api.example.com/files/download+3. **Get file revisions:**+ - Endpoint: https://api.example.com/files/list_revisions++### Moving to Distributed Systems++#### Improvements:+1. **Sharding:** Split storage across servers based on `user_id`.+2. **Amazon S3:** Use S3 for scalable and redundant file storage with cross-region replication.++ <img src="./images/replication.png" alt="Replication" width="400" />++3. **Load Balancer:** Distribute traffic across multiple web servers.+4. **Metadata Database Replication:** Ensure availability through database sharding and replication.+++#### Sync Conflicts:+For a large storage system like Google Drive, sync conflicts happen from time to time.+When two users modify the same file or folder at the same time, a conflict happens.++<div style="margin-left:5rem">+<img src="./images/sync-conflicts.png" alt="Sync Conflicts" width="500" />+</div>++- In the example user 1 and user 2 tries to update the same file at the same time, but user 1’s file is processed by our system first.+- User 1’s update operation goes through, but, user 2 gets a sync conflict.+- The system presents both copies of the same file: user 2’s local copy and the latest version from the server.+- User 2 has the option to merge both files or override one version with the other.++### Improved design+<div style="margin-left:5rem">+<img src="./images/high-level-design.png" alt="High Level Design" width="400" />+</div>++1. **User Interaction:**: Users access the application via browser or mobile app.++2. **Block Servers:**+ - Files are split into **4 MB blocks** (maximum size) and assigned unique hash values.+ - Blocks are stored independently in cloud storage (e.g., Amazon S3).+ - File reconstruction involves joining blocks in a specific order.++3. **Cloud Storage:** Blocks are stored in cloud storage for scalability and redundancy.++4. **Cold Storage:** Inactive files are moved to cold storage to reduce costs.++5. **Load Balancer:** Distributes requests evenly among API servers to ensure efficient operation.++6. **API Servers:**+ - Handle user authentication, profile management, and file metadata updates.+ - Manage all non-uploading workflows.++7. **Metadata Database and Cache:**+ - Stores metadata for users, files, blocks, and versions.+ - Frequently accessed metadata is cached for faster retrieval.++8. **Notification Service:**+ - A **publisher/subscriber system** that notifies clients about file changes (add, edit, delete).+ - Ensures clients can pull the latest updates.++9. **Offline Backup Queue:** Temporarily stores file change information for offline clients to sync when back online.++---++## Step 3: Design Deep Dive++### Metadata Database+A highly simplified is shown below version as it only includes the most important tables and fields.+#### Schema Design:+- **User Table:** Stores user profiles and preferences.+- **File Table:** Maintains file metadata (e.g., size, name, path).+- **Block Table:** Tracks file blocks for reconstructing files.+- **File Version Table:** Stores file revision history.++<div style="margin-left:5rem">+<img src="./images/metadata-database.png" alt="Metadata Database " width="400" />+</div>++---++### File Upload Flow++1. **File Upload:**+ - File is split into blocks, compressed, and encrypted by the block server.+ - Blocks are uploaded to block servers and stored in S3.+2. **Metadata Upload:**+ - Client sends metadata to the API server.+ - Metadata is stored in the database with status `pending`.+3. **Completion:**+ - S3 triggers a callback to update the file status to `uploaded`.+ - Notification service informs relevant users.+++<div style="margin-left:5rem">+<img src="./images/upload-flow.png" alt="Upload Flow " width="500" />+</div>+++---++### File Sync+1. **Delta Sync:** Transfer only modified blocks instead of the entire file.++ <div style="margin-left:2rem">+ <img src="./images/delta-sync.png" alt="Delta Sync" width="200" />+ </div>++2. **Compression:** Blocks are compressed using compression algorithms depending on file types.+3. **Conflict Resolution:**+ - First processed version wins.+ - Conflicting versions are saved separately for user resolution.++<div style="margin-left:5rem">+<img src="./images/file-sync.png" alt="File Synce " width="400" />+</div>++---++### File Download Flow+Download flow is triggered when a file is added or edited elsewhere. There are two ways a client can know:+- If client A is online while a file is changed by another client, notification service will inform client A.+- If client A is offline while a file is changed by another client, data will be saved to the cache. When the offline client is online again, it pulls the latest changes.++Once a client knows a file is changed, it first requests metadata via API servers, then+downloads blocks to construct the file.++1. **Trigger:** Notification service informs the client of file updates.+2. **Metadata Fetch:** Client retrieves updated metadata via API.+3. **Block Download:** Client downloads updated blocks from block servers and reconstructs the file.+++<div style="margin-left:3rem">+<img src="./images/download-flow.png" alt="Upload Flow " width="500" />+</div>+++---++### Notification Service+1. **Purpose:** Keeps clients updated about file changes.+2. **Mechanism:** Implements **long polling** for asynchronous notifications.+3. **Example:** When a file is added, edited, or deleted, notifications are pushed to all relevant clients.+++---++### Storage Optimization+1. **De-duplication:** Remove duplicate blocks at the account level using hash-based comparisons.+2. **Versioning Strategy:**+ - Limit the number of saved revisions.+ - Prioritize recent versions for frequently edited files.+3. **Cold Storage:** Move rarely accessed files to cheaper storage solutions (e.g., Amazon S3 Glacier).++---++### Failure Handling+1. **Load Balancer Failure:** Secondary load balancer becomes active.+2. **Block Server Failure:** Pending tasks are reassigned to other servers.+3. **Metadata Database Failure:**+ - Promote a slave node to master.+ - Redirect traffic to remaining replicas.+4. **Cloud Storage Failure:** Use cross-region replication to fetch unavailable files.+5. **Notification Service Failure:** Clients reconnect to alternative servers.+diff --git a/google-drive/images/delta-sync.png b/google-drive/images/delta-sync.pngnew file mode 100644index 0000000..2d2a534Binary files /dev/null and b/google-drive/images/delta-sync.png differdiff --git a/google-drive/images/download-flow.png b/google-drive/images/download-flow.pngnew file mode 100644index 0000000..dd26cd1Binary files /dev/null and b/google-drive/images/download-flow.png differdiff --git a/google-drive/images/file-sync.png b/google-drive/images/file-sync.pngnew file mode 100644index 0000000..8c06ba7Binary files /dev/null and b/google-drive/images/file-sync.png differdiff --git a/google-drive/images/high-level-design.png b/google-drive/images/high-level-design.pngnew file mode 100644index 0000000..77cb9ddBinary files /dev/null and b/google-drive/images/high-level-design.png differdiff --git a/google-drive/images/metadata-database.png b/google-drive/images/metadata-database.pngnew file mode 100644index 0000000..1bcf0d6Binary files /dev/null and b/google-drive/images/metadata-database.png differdiff --git a/google-drive/images/namespaces.png b/google-drive/images/namespaces.pngnew file mode 100644index 0000000..fcb9cfdBinary files /dev/null and b/google-drive/images/namespaces.png differdiff --git a/google-drive/images/replication.png b/google-drive/images/replication.pngnew file mode 100644index 0000000..de9de3cBinary files /dev/null and b/google-drive/images/replication.png differdiff --git a/google-drive/images/sync-conflicts.png b/google-drive/images/sync-conflicts.pngnew file mode 100644index 0000000..a2f1578Binary files /dev/null and b/google-drive/images/sync-conflicts.png differdiff --git a/google-drive/images/upload-flow.png b/google-drive/images/upload-flow.pngnew file mode 100644index 0000000..5476f34Binary files /dev/null and b/google-drive/images/upload-flow.png differdiff --git a/youtube/Readme.md b/youtube/Readme.mdindex a11c7d0..5ddd3c0 100644--- a/youtube/Readme.md+++ b/youtube/Readme.md@@ -38,6 +38,9 @@ YouTube is a massive video streaming platform supporting video uploads, playback## Step 2: High-Level Design### Components++ <img src="./images/high-level-design.png" alt="High Level Design" width="300">+1. **Client:** Devices like smartphones, computers, and TVs.2. **CDN (Content Delivery Network):** Stores and streams videos.3. **API Servers:** Handles all user interactions except video streaming (e.g., uploads, metadata updates).@@ -46,7 +49,6 @@ YouTube is a massive video streaming platform supporting video uploads, playback6. **Transcoding Servers:** Convert videos into multiple resolutions and formats.7. **Transcoded Storage:** Blob storage for transcoded videos.- <img src="./images/high-level-design.png" alt="High Level Design" width="300">---@@ -72,18 +74,25 @@ YouTube is a massive video streaming platform supporting video uploads, playback- **Metadata Upload (Steps):**++ <img src="./images/metadata-upload.png" alt="Metadata Upload" height="300">+- The client in parallel sends a request to update the video metadata- The request contains video metadata, including file name, size, format, etc.- <img src="./images/metadata-upload.png" alt="Metadata Upload" height="300">+#### 2. Video Streaming Flow++<div style="margin-left: 3em;">+ <img src="./images/video-streaming-flow.png" alt="Video Streaming Flow" height="250">+</div>+- Videos are streamed directly from the CDN using edge servers to minimize latency.- Some of te popular streaming protocols are MPEG_DASH, Apple HLS, Adobe HDS.- *Different streaming protocols support different video encodings and playback players.*- <img src="./images/video-streaming-flow.png" alt="Video Streaming Flow" width="200">---