Skip to content

Notes of the book System Desgin Interview - An Insider's Guide

Add Design Youtube Chapter

Gaurav Kumard490145eee83
diff --git a/youtube/Readme.md b/youtube/Readme.md
new file mode 100644
index 0000000..df501eb
--- /dev/null
+++ b/youtube/Readme.md
@@ -0,0 +1,191 @@
+# Chapter 14: Design YouTube
+
+## Introduction
+YouTube is a massive video streaming platform supporting video uploads, playback, and various interactions. This chapter focuses on designing a scalable video streaming system with the following core features:
+- **Fast video uploads**
+- **Smooth video streaming**
+- **Ability to change video quality**
+- **Low infrastructure cost**
+- **High availability and reliability**
+
+### Key Statistics (2020)
+- **2 billion monthly active users**
+- **5 billion videos watched per day**
+- **37% of mobile internet traffic comes from YouTube**
+- Available in **80 languages**
+- **$15.1 billion ad revenue** in 2019
+
+---
+
+## Step 1: Understand the Problem and Scope
+
+### Core Functionalities
+1. Upload videos
+2. Watch videos
+
+### Supported Platforms
+- Mobile apps, web browsers, and smart TVs
+
+### Assumptions
+- **Daily Active Users (DAU):** 5 million
+- **Average Video Size:** 300 MB
+- **Upload Limits:** Max 1 GB per video
+- **Daily Storage Need:** 150 TB
+- **CDN Costs:** 5 million * 5 videos * 0.3GB * $0.02 = $150,000/day (using Amazon CloudFront)
+
+---
+
+## Step 2: High-Level Design
+
+### Components
+1. **Client:** Devices like smartphones, computers, and TVs.
+2. **CDN (Content Delivery Network):** Stores and streams videos.
+3. **API Servers:** Handles all user interactions except video streaming (e.g., uploads, metadata updates).
+4. **Metadata Database:** Stores video metadata (e.g., title, description, size).
+5. **Original Storage:** Blob storage for uploaded videos.
+6. **Transcoding Servers:** Convert videos into multiple resolutions and formats.
+7. **Transcoded Storage:** Blob storage for transcoded videos.
+
+ <img src="./images/high-level-design.png" alt="High Level Design" width="300">
+
+---
+
+### Core Workflows
+#### 1. Video Uploading Flow
+- **Parallel Processes:**
+ 1. Upload video to original storage.
+ 2. Update video metadata in the database.
+
+- **Video Upload (Steps):**
+ - [1] Videos are uploaded to blob storage.
+ - [2] Transcoding servers convert videos to multiple formats.
+ - [3] One trasncoding is complete, following two steps are exectued in parallel.
+ - [3a] Transcoded videos are sent to transcoded storage.
+ - [3b] Transcoding completion events are queued in the completion queue.
+ - [3a.1] Videos are distributed to the CDN.
+ - [3b.1] Completion handlers update metadata and inform users.
+
+
+ <img src="./images/video-uploading-flow.png" alt="Video Upload Flow" width="400">
+
+
+- **Metadata Upload (Steps):**
+ - The client in parallel sends a request to update the video metadata
+ - The request contains video metadata, including file name, size, format, etc.
+
+ <img src="./images/metadata-upload.png" alt="Metadata Upload" height="300">
+
+
+#### 2. Video Streaming Flow
+- Videos are streamed directly from the CDN using edge servers to minimize latency.
+- Some of te popular streaming protocols are MPEG_DASH, Apple HLS, Adobe HDS.
+- *Different streaming protocols support different video encodings and playback players.*
+
+ <img src="./images/video-streaming-flow.png" alt="Video Streaming Flow" width="200">
+
+---
+
+## Step 3: Design Deep Dive
+
+### Video Transcoding
+#### Importance
+1. Raw video consumes large amounts of storage space. It Reduces storage space.
+2. Ensures compatibility across devices and browsers.
+3. Adapts video quality to network conditions.
+
+#### Components
+- **Container:** Encapsulates video, audio, and metadata (e.g., MP4, AVI).
+- **Codecs:** Compression and Decompression algorithms (e.g., H.264, VP9).
+
+#### Directed Acyclic Graph (DAG) Model
+- Transcoding a video is computationally expensive and time-consuming.
+- DAG Model defines tasks like encoding, thumbnail generation, and watermarking.
+- Allows high parallelism in video processing.
+
+ <img src="./images/dag-video-transcoding.png" alt="DAG Video Transcoding" width="400">
+
+- The original video is split into video, audio, and metadata.
+ - Video encodings: Videos are converted to support different resolutions, codec, bitrates.
+ - Thumbnail: It can either be uploaded by a user or automatically generated bythe system.
+ - Watermark: Image overlay on top of your video contains identifying information about the video.
+
+---
+
+### Video Transcoding Architecture
+
+&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;<img src="./images/video-transcoding-architecture.png" alt="Video Transcoding" width="500">
+
+1. **Preprocessor:** Splits videos into smaller chunks (GOP alignment). It has 4 responsibilities.
+ - Video splitting: Video stream is split or further split into smaller Group of Pictures (GOP) alignment.
+ - It split videos by GOP alignment for old clients.
+ - It generates DAG based on configuration files client programmers write.
+ - It stores GOPs and metadata in temporary storage in case the encoding fails, the system could use persisted data for retry operations.
+
+ <img src="./images/dag-config.png" alt="DAG Config" width="400">
+
+2. **DAG Scheduler:** Organizes tasks into sequential or parallel stages.
+ - It splits a DAG graph into stages of tasks and puts them in the task queue in the resource manager.
+ - Stage 1: video, audio, and metadata.
+ - The video file is further split into two tasks in stage 2: video encoding and thumbnail.
+
+ <img src="./images/dag-scheduler.png" alt="DAG Scheduler" width="300">
+
+3. **Resource Manager:** Responsible for managing the efficiency of resource allocation.It
+contains 3 queues and a task scheduler.
+ - Task queue: priority queue that contains tasks to be executed.
+ - Worker queue: priority queue that contains worker utilization info.
+ - Running queue: contains currently running tasks and workers running the tasks.
+ - Task scheduler: picks the optimal task/worker, and instructs the chosen task worker to execute the job.
+
+ <img src="./images/resource-manager.png" alt="Resource Manager" width="400">
+
+4. **Task Workers:** Perform transcoding and other operations.
+ - Different task workers may run different tasks
+
+ <img src="./images/task-worker.png" alt="Task Worker" width="150">
+
+5. **Temporary Storage:** Stores intermediate data for retries.
+ - The choice of storage system depends on factors like data type, data size, access frequency, data life span, etc.
+6. **Output:** Transcoded videos ready for distribution.
+
+
+---
+
+## System Optimizations
+
+### Speed Optimizations
+1. **Parallel Video Uploads:** Split videos into smaller chunks for faster, resumable uploads.
+
+ <img src="./images/video-split.png" alt="Video Split" width="400">
+
+2. **Distributed Upload Centers:** Use CDNs as upload hubs close to users.
+3. **Parallel Processing:** Decouple modules using message queues for high parallelism.
+
+ <img src="./images/message-queue1.png" alt="Message Queue" width="400">
+ <img src="./images/message-queue2.png" alt="Message Queue" height="115" width="400">
+
+### Safety Optimizations
+1. **Pre-Signed URLs:** Restrict video uploads to authorized users.
+
+ <img src="./images/pres-signed-urls.png" alt="Pre Signed" width="300">
+
+2. **Protect Videos:**
+ - **DRM Systems** (e.g., Apple FairPlay, Google Widevine).
+ - **AES Encryption.**
+ - **Watermarking.**
+
+### Cost-Saving Optimizations
+1. Serve only popular videos via CDN; less popular ones from high-capacity servers.
+2. Encode on-demand for rarely accessed videos.
+3. Regionalize video distribution based on popularity.
+4. Build custom CDNs and partner with ISPs to reduce bandwidth costs.
+
+---
+
+## Error Handling
+### Recoverable Errors
+- Retry failed uploads, transcoding, or resource allocation tasks.
+
+### Non-Recoverable Errors
+- Stop malformed video processing and return error codes.
+
diff --git a/youtube/images/dag-config.png b/youtube/images/dag-config.png
new file mode 100644
index 0000000..ea23016
Binary files /dev/null and b/youtube/images/dag-config.png differ
diff --git a/youtube/images/dag-scheduler.png b/youtube/images/dag-scheduler.png
new file mode 100644
index 0000000..1a3f89a
Binary files /dev/null and b/youtube/images/dag-scheduler.png differ
diff --git a/youtube/images/dag-video-transcoding.png b/youtube/images/dag-video-transcoding.png
new file mode 100644
index 0000000..952aace
Binary files /dev/null and b/youtube/images/dag-video-transcoding.png differ
diff --git a/youtube/images/high-level-design.png b/youtube/images/high-level-design.png
new file mode 100644
index 0000000..9d56536
Binary files /dev/null and b/youtube/images/high-level-design.png differ
diff --git a/youtube/images/message-queue1.png b/youtube/images/message-queue1.png
new file mode 100644
index 0000000..1037803
Binary files /dev/null and b/youtube/images/message-queue1.png differ
diff --git a/youtube/images/message-queue2.png b/youtube/images/message-queue2.png
new file mode 100644
index 0000000..e084347
Binary files /dev/null and b/youtube/images/message-queue2.png differ
diff --git a/youtube/images/metadata-upload.png b/youtube/images/metadata-upload.png
new file mode 100644
index 0000000..466a7c2
Binary files /dev/null and b/youtube/images/metadata-upload.png differ
diff --git a/youtube/images/pres-signed-urls.png b/youtube/images/pres-signed-urls.png
new file mode 100644
index 0000000..387d43c
Binary files /dev/null and b/youtube/images/pres-signed-urls.png differ
diff --git a/youtube/images/resource-manager.png b/youtube/images/resource-manager.png
new file mode 100644
index 0000000..b175d27
Binary files /dev/null and b/youtube/images/resource-manager.png differ
diff --git a/youtube/images/task-worker.png b/youtube/images/task-worker.png
new file mode 100644
index 0000000..bd26417
Binary files /dev/null and b/youtube/images/task-worker.png differ
diff --git a/youtube/images/video-split.png b/youtube/images/video-split.png
new file mode 100644
index 0000000..50d30a4
Binary files /dev/null and b/youtube/images/video-split.png differ
diff --git a/youtube/images/video-streaming-flow.png b/youtube/images/video-streaming-flow.png
new file mode 100644
index 0000000..9df73d8
Binary files /dev/null and b/youtube/images/video-streaming-flow.png differ
diff --git a/youtube/images/video-transcoding-architecture.png b/youtube/images/video-transcoding-architecture.png
new file mode 100644
index 0000000..ad220e4
Binary files /dev/null and b/youtube/images/video-transcoding-architecture.png differ
diff --git a/youtube/images/video-uploading-flow.png b/youtube/images/video-uploading-flow.png
new file mode 100644
index 0000000..7bc42f2
Binary files /dev/null and b/youtube/images/video-uploading-flow.png differ