Notes of the book System Desgin Interview - An Insider's Guide
Add Chapter 24 S3-like Object Storage
Gaurav Kumardeb4edebdc8f
diff --git a/24. S3-like Object Storage/README.md b/24. S3-like Object Storage/README.mdnew file mode 100644index 0000000..20ab041--- /dev/null+++ b/24. S3-like Object Storage/README.md@@ -0,0 +1,447 @@+# Chapter 24: S3-like Object Storage++## Introduction++In this chapter, we'll be designing an **object storage** service, similar to **Amazon S3**.++Storage systems fall into three broad categories:+- **Block storage**+- **File storage**+- **Object storage**++**Block storage** are devices, which came out in 1960s. HDDs and SSDs are such examples.+These devices are typically physically attached to a server, although they can also be network-attached via high-speed network protocols.+Servers can format the raw blocks and use them as a file system or it can hand control of them to servers directly.++**File storage** is built on top of block storage. It provides a higher level of abstraction, making it easier to manage folders and files.++**Object storage** sacrifices performance for high durability, vast scale and low cost.+It targets "cold" data and is mainly used for archival and backup.+There is no hierarchical directory structure, all data is stored as objects in a flat structure.+It is relatively slow compared to other storage types. Most cloud providers have an object storage offering - Amazon S3, Google GCS, etc.++<div style="margin-left:3rem">+ <img src="./images/storage-comparison.png" alt="storage-comparison" width="500" />+</div>++| | Block Storage | File Storage | Object Storage |+|-----------------|----------------------------------|-----------------------------------------|--------------------------------|+| Mutable Content | Y | Y | N (has object versioning) |+| Cost | High | Medium to high | Low |+| Performance | Medium to high, very high | Medium to high | Low to medium |+| Consistency | Strong consistency | Strong consistency | Strong consistency [5] |+| Data access | SAS/iSCSI/FC | Standard file access, CIFS/SMB, and NFS | RESTful API |+| Scalability | Medium scalability | High scalability | Vast scalability |+| Good for | Virtual machines (VM), databases | General-purpose file system access | Binary data, unstructured data |++Some terminology, related to object storage:+- **Bucket** - logical container for objects. Name is globally unique.+- **Object** - An individual piece of data, stored in a bucket. Contains object data and metadata.+- **Versioning** - A feature keeping multiple variants of an object in the same bucket.+- **Uniform Resource Identifier (URI)** - each resource is uniquely identified by a URI.+- **Service-level Agreement (SLA)** - contract between service provider and client.++Amazon S3 Standard-Infrequent Access storage class SLAs:+- Durability of 99.999999999% across multiple Availability Zones+- Data is resilient in the event of entire Availability Zone being destroyed+- Designed for 99.9% availability++---++## Step 1: Understand the Problem and Establish Design Scope++- C: Which features should be included?+- I: Bucket creation, Object upload/download, versioning, Listing objects in a bucket+- C: What is the typical data size?+- I: We need to store both massive objects and small objects efficiently+- C: How much data do we store in a year?+- I: 100 petabytes+- C: Can we assume 6 nines of data durbility (99.9999%) and service availability of 4 nines (99.99%)?+- I: Yes, sounds reasonable++### **Non-functional requirements**++- **100 PB of data**+- **6 nines of data durability**+- **4 nines of service availability**+- Storage efficiency. Reduce storage cost while maintaining high reliability and performance++### **Back-of-the-envelope estimation**++Object storage is likely to have bottlenecks in disk capacity or IO per second (IOPS).++Assumptions:+- we have 20% small (less than 1mb), 60% mid-size (1-64mb) and 20% large objects (greater than 64mb),+- One hard disk (SATA, 7200rpm) is capable of doing 100-150 random seeks per second (100-150 IOPS)++Given the assumptions, we can estimate the total number of objects the system can persist.+- Let's use median size per object type to simplify calculation - 0.5mb for small, 32mb for medium, 200mb for large.+- Given 100PB of storage (10^11 MB) and 40% of storage usage results in 0.68bil objects+- If we assume metadata is 1kb, then we need 0.68tb space to store metadata info++---++## Step 2: Propose High-Level Design and Get Buy-In++Let's explore some interesting properties of object storage before diving into the design:+- **Object immutability** - objects in object storage are immutable (not the case in other storage systems). We may delete them or replace them, but no update.+- **Key-value store** - an object URI is its key and we can get its contents by making an HTTP call+- **Write once, read many times** - data access pattern is writing once and reading many times. According to some Linkedin research, 95% of operations are reads+- Support both small and large objects++Design philosophy of object storage is similar to UNIX - when we save a file, it creates the filename in a data structure, called inode and file data is stored in different disk locations.+The inode contains a list of file block pointers, which point to different locations on disk.++When accessing a file, we first fetch its metadata from the inode, prior to fetching the file contents.++Object storage works similarly - metadata store is used for file information, but contents are stored on disk:++<div style="margin-left:3rem">+ <img src="./images/object-store-vs-unix.png" alt="object-store-vs-unix" width="500" />+</div>++By separating metadata from file contents, we can scale the different stores independently:++<div style="margin-left:3rem">+ <img src="./images/bucket-and-object.png" alt="bucket-and-object" width="500" />+</div>++### **High-level design**++<div style="margin-left:3rem">+ <img src="./images/high-level-design.png" alt="high-level-design" width="500" />+</div>++- **Load balancer** - distributes API requests across service replicas+- **API service** - Stateless server, orchestrating calls to metadata and object store, as well as IAM service.+- **Identity and access management (IAM)** - central place for auth, authz, access control.+- **Data store** - stores and retrieves actual data. Operations are based on object ID (UUID).+- **Metadata store** - stores object metadata++### **Uploading an object**++<div style="margin-left:3rem">+ <img src="./images/uploading-object.png" alt="uploading-object" width="500" />+</div>++- Create a bucket named "bucket-to-share" via HTTP PUT request+- API service calls IAM to ensure user is authorized and has write permissions+- API service calls metadata store to create a bucket entry. Once created, success response is returned.+- After bucket is created, HTTP PUT is sent to create an object named "script.txt"+- API service verifies user identity and ensures user has write permissions+- Once validation passes, object payload is sent via HTTP PUT to the data store. Data store persists it and returns a UUID.+- API service calls metadata store to create a new entry with object_id, bucket_id and bucket_name, among other metadata.++Example object upload request:++```+PUT /bucket-to-share/script.txt HTTP/1.1+Host: foo.s3example.org+Date: Sun, 12 Sept 2021 17:51:00 GMT+Authorization: authorization string+Content-Type: text/plain+Content-Length: 4567+x-amz-meta-author: Alex++[4567 bytes of object data]+```++### **Downloading an object**++Buckets have no directory hierarchy, buy we can create a logical hierarchy by concatenating bucket name and object name to simulate a folder structure.++Example GET request for fetching an object:++```+GET /bucket-to-share/script.txt HTTP/1.1+Host: foo.s3example.org+Date: Sun, 12 Sept 2021 18:30:01 GMT+Authorization: authorization string+```++<div style="margin-left:3rem">+ <img src="./images/download-object.png" alt="download-object" width="500" />+</div>++- Client sends an HTTP GET request to the load balancer, ie `GET /bucket-to-share/script.txt`+- API service queries IAM to verify the user has correct permissions to read the bucket+- Once validated, UUID of object is retrieved from metadata store+- Object payload is retrieved from data store based on UUID and returned to the client++---++// sprint 1++## Step 3: Design Deep Dive++### **Data store**++Here's how the API service interacts with the data store:++<div style="margin-left:3rem">+ <img src="./images/data-store-interactions.png" alt="data-store-interactions" width="500" />+</div>++The data store's main components:++<div style="margin-left:3rem">+ <img src="./images/data-store-main-components.png" alt="data-store-main-components" width="500" />+</div>++The data routing service provides a RESTful or gRPC API to access the data node cluster.+It is a stateless service, which scales by adding more servers.++It's main responsibilities are:+- querying the placement service to get the best data node to store data+- reading data from data nodes and returning it to the API service+- Writing data to data nodes++The placement service determines which data nodes should store an object.+It maintains a virtual cluster map, which determines the physical topology of a cluster.++<div style="margin-left:3rem">+ <img src="./images/virtual-cluster-map.png" alt="virtual-cluster-map" width="500" />+</div>++The service also sends heartbeats to all data nodes to determine if they should be removed from the virtual cluster.++Since this is a critical service, it is recommended to maintain a cluster of 5 or 7 replicas, synchronized via Paxos or Raft consensus algorithms.+Eg a 7 node cluster can tolerate 3 nodes failing.++Data nodes store the actual object data.+Reliability and durability is ensured by replicating data to multiple data nodes.++Each data node has a daemon running, which sends heartbeats to the placement service.++The heartbeat includes:+- How many disk drives (HDD or SSD) does the data node manage?+- How much data is stored on each drive?++#### Data persistence flow++<div style="margin-left:3rem">+ <img src="./images/data-persistence-flow.png" alt="data-persistence-flow" width="500" />+</div>++- API service forwards the object data to data store+- Data routing service sends the data to the primary data node+- Primary data node saves the data locally and replicates it to two secondary data nodes. Response is sent after successful replication.+- The UUID of the object is returned to the API service.++Caveats:+- Given an object UUID, it's replication group is deterministically chosen by using consistent hashing+- In step 4, the primary data node replicates the object data before returning a response. This favors strong consistency over higher latency.++<div style="margin-left:3rem">+ <img src="./images/consistency-vs-latency.png" alt="consistency-vs-latency" width="500" />+</div>++#### How data is organized++One simple approach to managing data is to store each object in a separate file.++This works, but is not performant with many small files in a file system:+- Data blocks on HDD are wasted, because every file uses the whole block size. Typical block size is 4kb.+- Many files means many inodes. Operating systems don't deal well with too many inodes and there is also a max inode limit.++These issues can be addressed by merging many small files into bigger ones via a write-ahead log (WAL). Once the file reaches its capacity (typically a few GB), a new file is created:++<div style="margin-left:3rem">+ <img src="./images/wal-optimization.png" alt="wal-optimization" width="500" />+</div>++The downside of this approach is that write access to the file needs to be serialized. Multiple cores accessing the same file must wait for each other.+To fix this, we can confine files to specific cores to avoid lock contention.++#### Object lookup++To support storing multiple objects in the same file, we need to maintain a table, which tells the data node:+- `object_id`+- `filename` where object is stored+- `file_offset` where object starts+- `object_size`++We can deploy this table in a file-based db like RocksDB or a traditional relational database.+Since the access pattern is low write+high read, a relational database works better.++How should we deploy it?+We could deploy the db and scale it separately in a cluster, accessed by all data nodes.++Downsides:+- we'd need to aggressively scale the cluster to serve all requests+- there's additional network latency between data node and db cluster++An alternative is to take advantage of the fact that data nodes are only interested to data related to them,+so we can deploy the relational db within the data node itself.++SQLite is a good option as it's a lightweight file-based relational database.++#### Updated data persistence flow++<div style="margin-left:3rem">+ <img src="./images/updated-data-persistence-flow.png" alt="updated-data-persistence-flow" width="500" />+</div>++- API Service sends a request to save a new object+- Data node service appends the new object at the end of a file, named "/data/c"+- A new record for the object is inserted into the object mapping table++#### Durability++Data durability is an important requirement in our design. In order to achieve 6 nines of durability, every failure case needs to be properly examined.++First problem to address is hardware failures. We can achieve that by replicating data nodes to minimize probability of failure.+But in addition to that, we also ought to replicate across different failure domains (cross-rack, cross-dc, separate networks, etc).+A critical event can cause multiple hardware failures within the same domain:++<div style="margin-left:3rem">+ <img src="./images/failure-domain-isolation.png" alt="failure-domain-isolation" width="500" />+</div>++Assuming annual failure rate of a typical HDD is 0.81%, making three copies gives us 6 nines of durability.++Replicating the data nodes like that grants us the durability we want, but we could also leverage erasure coding to reduce storage costs.++Erasure coding enables us to use parity bits, which allow us to reconstruct lost bits in the event of a failure:++<div style="margin-left:3rem">+ <img src="./images/erasure-coding.png" alt="erasure-coding" width="500" />+</div>++Imagine those bits are data nodes. If two of them go down, they can be recovered using the remaining four ones.++There are different erasure coding schemes. In our case, we could use 8+4 erasure coding, split across different failure domains to maximize reliability:++<div style="margin-left:3rem">+ <img src="./images/erasure-coding-across-failure-domains.png" alt="erasure-coding-across-failure-domains" width="500" />+</div>++Erasure coding enables us to achieve a much lower storage cost (50% improvement) at the expense of access speed due to the data routing service having to collect data from multiple locations:++<div style="margin-left:3rem">+ <img src="./images/erasure-coding-vs-replication.png" alt="erasure-coding-vs-replication" width="500" />+</div>++Other caveats:+- Replication requires 200% storage overhead (in case of 3 replicas) vs. 50% via erasure coding+- Erasure coding [gives us 11 nines of durability](https://github.com/Backblaze/erasure-coding-durability) vs 6 nines via replication+- Erasure coding requires more computation to calculate and store parities++In sum, replication is more useful for latency-sensitive applications, whereas erasure coding is attractive for storage cost efficiency and durability.+Erasure coding is also much harder to implement.++#### Correctness verification++If a disk fails entirely, then the failure is easy to detect. This is less straightforward in the event part of the disk memory gets corrupted.++To detect this, we can use checksums - a hash of the file contents, which can be used to verify the file's integrity.++In our case, we'll store checksums for each file and each object:++<div style="margin-left:3rem">+ <img src="./images/checksums-for-correctness.png" alt="checksums-for-correctness" width="500" />+</div>++In the case of erasure coding (8+4), we'll need to fetch each of the 8 pieces of data separately and verify each of their checksums.++// sprint 2++### **Metadata data model**++Table schemas:++<div style="margin-left:3rem">+ <img src="./images/metadata-data-model.png" alt="metadata-data-model" width="500" />+</div>++Queries we need to support:+- Find an object ID by name+- Insert/delete object based on name+- List objects in a bucket sharing the same prefix++There is usually a limit on the number of buckets a user can create, hence, the size of the buckets table is small and can fit into a single db server.+But we still need to scale the server for read throughput.++The object table will probably not fit into a single database server, though. Hence, we can scale the table via sharding:+- Sharding by bucket_id will lead to hotspot issues as a bucket can have billions of objects+- Sharding by bucket_id makes the load more evenly distributed, but our queries will be slow+- We choose sharding by `hash(bucket_name, object_name)` since most queries are based on the object/bucket name.++Even with this sharding scheme, though, listing objects in a bucket will be slow.++### **Listing objects in a bucket**++In a single database, listing an object based on its prefix (looks like a directory) works like this:++```+SELECT * FROM object WHERE bucket_id = "123" AND object_name LIKE `abc/%`+```++This is challenging to fulfill when the database is sharded. To achieve it, we can run the query on every shard and aggregate the results in-memory.+This makes pagination challenging though, since different shards contain a different result size and we need to maintain separate limit/offset for each.++We can leverage the fact that typically object stores are not optimized for listing objects, so we can sacrifice listing performance.+We can also create a denormalized table for listing objects, sharded by bucket ID.+That would make our listing query sufficiently fast as it's isolated to a single database instance.++### **Object versioning**++Versioning works by having another `object_version` column which is of type TIMEUUID, enabling us to sort records based on it.++Each new version produces a new `object_id`:++<div style="margin-left:3rem">+ <img src="./images/object-versioning.png" alt="object-versioning" width="500" />+</div>++Deleting an object creates a new version with a special `object_id` indicating that the object was deleted. Queries for it return 404:++<div style="margin-left:3rem">+ <img src="./images/deleting-versioned-object.png" alt="deleting-versioned-object" width="500" />+</div>++### **Optimizing uploads of large files**++Uploading large files can be optimized by using multipart uploads - splitting a big file into several chunks, uploaded independently:++<div style="margin-left:3rem">+ <img src="./images/multipart-upload.png" alt="multipart-upload" width="500" />+</div>++- Client calls service to initiate a multipart upload+- Data store returns an upload ID which uniquely identifies the upload+- Client splits the large file into several chunks, uploaded independently using the upload id+- When a chunk is uploaded, the data store returns an etag, which is a md5 checksum, identifying that upload chunk+- After all parts are uploaded, client sends a complete multipart upload request, which includes upload_id, part numbers and all etags+- Data store reassembles the object from its parts. The process can take a few minutes. After that, success response is returned to the client.++Old parts, which are no longer useful can be removed at this point. We can introduce a garbage collector to deal with it.++### **Garbage collection**++Garbage collection is the process of reclaiming storage space, which is no longer used. There are a few ways data becomes garbage:+- **lazy object deletion** - object is marked as deleted without actually getting deleted+- **orphan data** - eg an upload failed mid-flight and old parts need to be deleted+- **corrupted data** - data which failed checksum verification++The garbage collector is also responsible for reclaiming unused space in replicas.+With replication, data is deleted from both primaries and replicas. With erasure coding (8+4), data is deleted from all 12 nodes.++To facilitate the deletion, we'll use a process called compaction:+- Garbage collector copies objects which are not deleted from "data/b" to "data/d"+- `object_mapping` table is updated once copying is complete using a database transaction+- To avoid making too many small files, compaction is done on files which grow beyond a certain threshold++<div style="margin-left:3rem">+ <img src="./images/compaction.png" alt="compaction" width="500" />+</div>++---++## Step 4: Wrap Up++Things we covered:+- Designing an S3-like object storage+- Comparing differences between object, block and file storages+- Covered uploading, downloading, listing, versioning of objects in a bucket+- Deep dived in the design - data store and metadata store, replication and erasure coding, multipart uploads, shardingdiff --git a/24. S3-like Object Storage/images/bucket-and-object.png b/24. S3-like Object Storage/images/bucket-and-object.pngnew file mode 100644index 0000000..f8c2ec9Binary files /dev/null and b/24. S3-like Object Storage/images/bucket-and-object.png differdiff --git a/24. S3-like Object Storage/images/checksums-for-correctness.png b/24. S3-like Object Storage/images/checksums-for-correctness.pngnew file mode 100644index 0000000..98ed39eBinary files /dev/null and b/24. S3-like Object Storage/images/checksums-for-correctness.png differdiff --git a/24. S3-like Object Storage/images/compaction.png b/24. S3-like Object Storage/images/compaction.pngnew file mode 100644index 0000000..381155dBinary files /dev/null and b/24. S3-like Object Storage/images/compaction.png differdiff --git a/24. S3-like Object Storage/images/consistency-vs-latency.png b/24. S3-like Object Storage/images/consistency-vs-latency.pngnew file mode 100644index 0000000..a99abb4Binary files /dev/null and b/24. S3-like Object Storage/images/consistency-vs-latency.png differdiff --git a/24. S3-like Object Storage/images/data-persistence-flow.png b/24. S3-like Object Storage/images/data-persistence-flow.pngnew file mode 100644index 0000000..fd17912Binary files /dev/null and b/24. S3-like Object Storage/images/data-persistence-flow.png differdiff --git a/24. S3-like Object Storage/images/data-store-interactions.png b/24. S3-like Object Storage/images/data-store-interactions.pngnew file mode 100644index 0000000..2cbdc45Binary files /dev/null and b/24. S3-like Object Storage/images/data-store-interactions.png differdiff --git a/24. S3-like Object Storage/images/data-store-main-components.png b/24. S3-like Object Storage/images/data-store-main-components.pngnew file mode 100644index 0000000..82aac1aBinary files /dev/null and b/24. S3-like Object Storage/images/data-store-main-components.png differdiff --git a/24. S3-like Object Storage/images/deleting-versioned-object.png b/24. S3-like Object Storage/images/deleting-versioned-object.pngnew file mode 100644index 0000000..4e2e27bBinary files /dev/null and b/24. S3-like Object Storage/images/deleting-versioned-object.png differdiff --git a/24. S3-like Object Storage/images/download-object.png b/24. S3-like Object Storage/images/download-object.pngnew file mode 100644index 0000000..66fd33aBinary files /dev/null and b/24. S3-like Object Storage/images/download-object.png differdiff --git a/24. S3-like Object Storage/images/erasure-coding-across-failure-domains.png b/24. S3-like Object Storage/images/erasure-coding-across-failure-domains.pngnew file mode 100644index 0000000..0b5424dBinary files /dev/null and b/24. S3-like Object Storage/images/erasure-coding-across-failure-domains.png differdiff --git a/24. S3-like Object Storage/images/erasure-coding-vs-replication.png b/24. S3-like Object Storage/images/erasure-coding-vs-replication.pngnew file mode 100644index 0000000..4d50ed3Binary files /dev/null and b/24. S3-like Object Storage/images/erasure-coding-vs-replication.png differdiff --git a/24. S3-like Object Storage/images/erasure-coding.png b/24. S3-like Object Storage/images/erasure-coding.pngnew file mode 100644index 0000000..d92fa60Binary files /dev/null and b/24. S3-like Object Storage/images/erasure-coding.png differdiff --git a/24. S3-like Object Storage/images/failure-domain-isolation.png b/24. S3-like Object Storage/images/failure-domain-isolation.pngnew file mode 100644index 0000000..9ca9f00Binary files /dev/null and b/24. S3-like Object Storage/images/failure-domain-isolation.png differdiff --git a/24. S3-like Object Storage/images/high-level-design.png b/24. S3-like Object Storage/images/high-level-design.pngnew file mode 100644index 0000000..cf067daBinary files /dev/null and b/24. S3-like Object Storage/images/high-level-design.png differdiff --git a/24. S3-like Object Storage/images/metadata-data-model.png b/24. S3-like Object Storage/images/metadata-data-model.pngnew file mode 100644index 0000000..6bbbcfbBinary files /dev/null and b/24. S3-like Object Storage/images/metadata-data-model.png differdiff --git a/24. S3-like Object Storage/images/multipart-upload.png b/24. S3-like Object Storage/images/multipart-upload.pngnew file mode 100644index 0000000..5b9c056Binary files /dev/null and b/24. S3-like Object Storage/images/multipart-upload.png differdiff --git a/24. S3-like Object Storage/images/object-store-vs-unix.png b/24. S3-like Object Storage/images/object-store-vs-unix.pngnew file mode 100644index 0000000..2a880c2Binary files /dev/null and b/24. S3-like Object Storage/images/object-store-vs-unix.png differdiff --git a/24. S3-like Object Storage/images/object-versioning.png b/24. S3-like Object Storage/images/object-versioning.pngnew file mode 100644index 0000000..4e5ed1cBinary files /dev/null and b/24. S3-like Object Storage/images/object-versioning.png differdiff --git a/24. S3-like Object Storage/images/storage-comparison.png b/24. S3-like Object Storage/images/storage-comparison.pngnew file mode 100644index 0000000..ea70333Binary files /dev/null and b/24. S3-like Object Storage/images/storage-comparison.png differdiff --git a/24. S3-like Object Storage/images/updated-data-persistence-flow.png b/24. S3-like Object Storage/images/updated-data-persistence-flow.pngnew file mode 100644index 0000000..8ca25fbBinary files /dev/null and b/24. S3-like Object Storage/images/updated-data-persistence-flow.png differdiff --git a/24. S3-like Object Storage/images/uploading-object.png b/24. S3-like Object Storage/images/uploading-object.pngnew file mode 100644index 0000000..d950ac5Binary files /dev/null and b/24. S3-like Object Storage/images/uploading-object.png differdiff --git a/24. S3-like Object Storage/images/virtual-cluster-map.png b/24. S3-like Object Storage/images/virtual-cluster-map.pngnew file mode 100644index 0000000..272340aBinary files /dev/null and b/24. S3-like Object Storage/images/virtual-cluster-map.png differdiff --git a/24. S3-like Object Storage/images/wal-optimization.png b/24. S3-like Object Storage/images/wal-optimization.pngnew file mode 100644index 0000000..57a65d6Binary files /dev/null and b/24. S3-like Object Storage/images/wal-optimization.png differ