Notes of the book System Desgin Interview - An Insider's Guide
Add Design Youtube Chapter
Gaurav Kumard490145eee83
diff --git a/youtube/Readme.md b/youtube/Readme.mdnew file mode 100644index 0000000..df501eb--- /dev/null+++ b/youtube/Readme.md@@ -0,0 +1,191 @@+# Chapter 14: Design YouTube++## Introduction+YouTube is a massive video streaming platform supporting video uploads, playback, and various interactions. This chapter focuses on designing a scalable video streaming system with the following core features:+- **Fast video uploads**+- **Smooth video streaming**+- **Ability to change video quality**+- **Low infrastructure cost**+- **High availability and reliability**++### Key Statistics (2020)+- **2 billion monthly active users**+- **5 billion videos watched per day**+- **37% of mobile internet traffic comes from YouTube**+- Available in **80 languages**+- **$15.1 billion ad revenue** in 2019++---++## Step 1: Understand the Problem and Scope++### Core Functionalities+1. Upload videos+2. Watch videos++### Supported Platforms+- Mobile apps, web browsers, and smart TVs++### Assumptions+- **Daily Active Users (DAU):** 5 million+- **Average Video Size:** 300 MB+- **Upload Limits:** Max 1 GB per video+- **Daily Storage Need:** 150 TB+- **CDN Costs:** 5 million * 5 videos * 0.3GB * $0.02 = $150,000/day (using Amazon CloudFront)++---++## Step 2: High-Level Design++### Components+1. **Client:** Devices like smartphones, computers, and TVs.+2. **CDN (Content Delivery Network):** Stores and streams videos.+3. **API Servers:** Handles all user interactions except video streaming (e.g., uploads, metadata updates).+4. **Metadata Database:** Stores video metadata (e.g., title, description, size).+5. **Original Storage:** Blob storage for uploaded videos.+6. **Transcoding Servers:** Convert videos into multiple resolutions and formats.+7. **Transcoded Storage:** Blob storage for transcoded videos.++ <img src="./images/high-level-design.png" alt="High Level Design" width="300">++---++### Core Workflows+#### 1. Video Uploading Flow+- **Parallel Processes:**+ 1. Upload video to original storage.+ 2. Update video metadata in the database.++- **Video Upload (Steps):**+ - [1] Videos are uploaded to blob storage.+ - [2] Transcoding servers convert videos to multiple formats.+ - [3] One trasncoding is complete, following two steps are exectued in parallel.+ - [3a] Transcoded videos are sent to transcoded storage.+ - [3b] Transcoding completion events are queued in the completion queue.+ - [3a.1] Videos are distributed to the CDN.+ - [3b.1] Completion handlers update metadata and inform users.+++ <img src="./images/video-uploading-flow.png" alt="Video Upload Flow" width="400">+++- **Metadata Upload (Steps):**+ - The client in parallel sends a request to update the video metadata+ - The request contains video metadata, including file name, size, format, etc.++ <img src="./images/metadata-upload.png" alt="Metadata Upload" height="300">+++#### 2. Video Streaming Flow+- Videos are streamed directly from the CDN using edge servers to minimize latency.+- Some of te popular streaming protocols are MPEG_DASH, Apple HLS, Adobe HDS.+- *Different streaming protocols support different video encodings and playback players.*++ <img src="./images/video-streaming-flow.png" alt="Video Streaming Flow" width="200">++---++## Step 3: Design Deep Dive++### Video Transcoding+#### Importance+1. Raw video consumes large amounts of storage space. It Reduces storage space.+2. Ensures compatibility across devices and browsers.+3. Adapts video quality to network conditions.++#### Components+- **Container:** Encapsulates video, audio, and metadata (e.g., MP4, AVI).+- **Codecs:** Compression and Decompression algorithms (e.g., H.264, VP9).++#### Directed Acyclic Graph (DAG) Model+- Transcoding a video is computationally expensive and time-consuming.+- DAG Model defines tasks like encoding, thumbnail generation, and watermarking.+- Allows high parallelism in video processing.++ <img src="./images/dag-video-transcoding.png" alt="DAG Video Transcoding" width="400">++- The original video is split into video, audio, and metadata.+ - Video encodings: Videos are converted to support different resolutions, codec, bitrates.+ - Thumbnail: It can either be uploaded by a user or automatically generated bythe system.+ - Watermark: Image overlay on top of your video contains identifying information about the video.++---++### Video Transcoding Architecture++ <img src="./images/video-transcoding-architecture.png" alt="Video Transcoding" width="500">++1. **Preprocessor:** Splits videos into smaller chunks (GOP alignment). It has 4 responsibilities.+ - Video splitting: Video stream is split or further split into smaller Group of Pictures (GOP) alignment.+ - It split videos by GOP alignment for old clients.+ - It generates DAG based on configuration files client programmers write.+ - It stores GOPs and metadata in temporary storage in case the encoding fails, the system could use persisted data for retry operations.++ <img src="./images/dag-config.png" alt="DAG Config" width="400">++2. **DAG Scheduler:** Organizes tasks into sequential or parallel stages.+ - It splits a DAG graph into stages of tasks and puts them in the task queue in the resource manager.+ - Stage 1: video, audio, and metadata.+ - The video file is further split into two tasks in stage 2: video encoding and thumbnail.++ <img src="./images/dag-scheduler.png" alt="DAG Scheduler" width="300">++3. **Resource Manager:** Responsible for managing the efficiency of resource allocation.It+contains 3 queues and a task scheduler.+ - Task queue: priority queue that contains tasks to be executed.+ - Worker queue: priority queue that contains worker utilization info.+ - Running queue: contains currently running tasks and workers running the tasks.+ - Task scheduler: picks the optimal task/worker, and instructs the chosen task worker to execute the job.++ <img src="./images/resource-manager.png" alt="Resource Manager" width="400">++4. **Task Workers:** Perform transcoding and other operations.+ - Different task workers may run different tasks++ <img src="./images/task-worker.png" alt="Task Worker" width="150">++5. **Temporary Storage:** Stores intermediate data for retries.+ - The choice of storage system depends on factors like data type, data size, access frequency, data life span, etc.+6. **Output:** Transcoded videos ready for distribution.+++---++## System Optimizations++### Speed Optimizations+1. **Parallel Video Uploads:** Split videos into smaller chunks for faster, resumable uploads.++ <img src="./images/video-split.png" alt="Video Split" width="400">++2. **Distributed Upload Centers:** Use CDNs as upload hubs close to users.+3. **Parallel Processing:** Decouple modules using message queues for high parallelism.++ <img src="./images/message-queue1.png" alt="Message Queue" width="400">+ <img src="./images/message-queue2.png" alt="Message Queue" height="115" width="400">++### Safety Optimizations+1. **Pre-Signed URLs:** Restrict video uploads to authorized users.++ <img src="./images/pres-signed-urls.png" alt="Pre Signed" width="300">++2. **Protect Videos:**+ - **DRM Systems** (e.g., Apple FairPlay, Google Widevine).+ - **AES Encryption.**+ - **Watermarking.**++### Cost-Saving Optimizations+1. Serve only popular videos via CDN; less popular ones from high-capacity servers.+2. Encode on-demand for rarely accessed videos.+3. Regionalize video distribution based on popularity.+4. Build custom CDNs and partner with ISPs to reduce bandwidth costs.++---++## Error Handling+### Recoverable Errors+- Retry failed uploads, transcoding, or resource allocation tasks.++### Non-Recoverable Errors+- Stop malformed video processing and return error codes.+diff --git a/youtube/images/dag-config.png b/youtube/images/dag-config.pngnew file mode 100644index 0000000..ea23016Binary files /dev/null and b/youtube/images/dag-config.png differdiff --git a/youtube/images/dag-scheduler.png b/youtube/images/dag-scheduler.pngnew file mode 100644index 0000000..1a3f89aBinary files /dev/null and b/youtube/images/dag-scheduler.png differdiff --git a/youtube/images/dag-video-transcoding.png b/youtube/images/dag-video-transcoding.pngnew file mode 100644index 0000000..952aaceBinary files /dev/null and b/youtube/images/dag-video-transcoding.png differdiff --git a/youtube/images/high-level-design.png b/youtube/images/high-level-design.pngnew file mode 100644index 0000000..9d56536Binary files /dev/null and b/youtube/images/high-level-design.png differdiff --git a/youtube/images/message-queue1.png b/youtube/images/message-queue1.pngnew file mode 100644index 0000000..1037803Binary files /dev/null and b/youtube/images/message-queue1.png differdiff --git a/youtube/images/message-queue2.png b/youtube/images/message-queue2.pngnew file mode 100644index 0000000..e084347Binary files /dev/null and b/youtube/images/message-queue2.png differdiff --git a/youtube/images/metadata-upload.png b/youtube/images/metadata-upload.pngnew file mode 100644index 0000000..466a7c2Binary files /dev/null and b/youtube/images/metadata-upload.png differdiff --git a/youtube/images/pres-signed-urls.png b/youtube/images/pres-signed-urls.pngnew file mode 100644index 0000000..387d43cBinary files /dev/null and b/youtube/images/pres-signed-urls.png differdiff --git a/youtube/images/resource-manager.png b/youtube/images/resource-manager.pngnew file mode 100644index 0000000..b175d27Binary files /dev/null and b/youtube/images/resource-manager.png differdiff --git a/youtube/images/task-worker.png b/youtube/images/task-worker.pngnew file mode 100644index 0000000..bd26417Binary files /dev/null and b/youtube/images/task-worker.png differdiff --git a/youtube/images/video-split.png b/youtube/images/video-split.pngnew file mode 100644index 0000000..50d30a4Binary files /dev/null and b/youtube/images/video-split.png differdiff --git a/youtube/images/video-streaming-flow.png b/youtube/images/video-streaming-flow.pngnew file mode 100644index 0000000..9df73d8Binary files /dev/null and b/youtube/images/video-streaming-flow.png differdiff --git a/youtube/images/video-transcoding-architecture.png b/youtube/images/video-transcoding-architecture.pngnew file mode 100644index 0000000..ad220e4Binary files /dev/null and b/youtube/images/video-transcoding-architecture.png differdiff --git a/youtube/images/video-uploading-flow.png b/youtube/images/video-uploading-flow.pngnew file mode 100644index 0000000..7bc42f2Binary files /dev/null and b/youtube/images/video-uploading-flow.png differ