I recently designed an object store from scratch in Elixir. It has been serving me well as a backend for an app which needs to store all kinds of files: images, videos, documents. I wanted something simple to avoid dealing with off-the-shelf object stores which require complex configurations and to avoid cloud storage which is dead simple to use but can get very expensive, very quickly. For this project, simplicity was the key to make sure I can debug any failures quickly.
Elixir naturally allows a layered approach to designing systems with each layer providing a well-defined subset of overall functionality. This approach simplifies implementation, testing and makes it easier to rewrite parts of the stack if so desired. An object store has to take care of many things:
- Access control: depending on the application specific notion of “user” vs “admin”, or any other roles.
- Generate thumbnail for images, videos, PDFs: some object stores allow setting up webhooks or some other means of notification to let you take actions (such as generating thumbnails) when you PUT a file. It would be nice we can avoid this dance and if the object store can itself take care of this common requirement.
- Store per-file system metadata: such as file creation date, owner, size, format, tags.
- Store per-file user metadata: any application specific metadata as key-value pairs.
Taking care of all these concerns in a single monolith is surely not a good idea.
In the blog series, I will describe the process of designing an object store from scratch, going layer-by-layer starting from the bottom.
The layers in blue are the ones we will design while greyed-out layers are our major system dependencies.
Update: All six parts are now published:
- Part I - Introduction
- Part II - The FileStore layer
- Part III - ImageStore and VideoStore
- Part IV - The API layer
- Part V - The Web layer
- Part VI - Summary