A layered object store design in Elixir (Part VI)

Summary

We built an object store from scratch in Elixir using a layered design approach. The overall theme has been to avoid generalizing the design too much which kept implementation of each layer/module simple. We were also careful when adding any third-party dependencies which has multiple advantages: deeper understanding of your codebase, easier debugging (I hate unknown code-paths in backtraces).

For reference, here are links for all five parts along with their summaries:

  • Part I introduces the overall design of our object store.

  • Part II introduces the FileStore module which is responsible for actually storing input files to object store.

  • Part III introduces file format-specific modules: ImageStore for storing normalizing input images together with thumbnails, VideoStore for storing input video as is together with its thumbnail. These modules each deal with file format specific details while using the FileStore layer to finally store resulting artifacts.

  • Part IV introduces the API layer where we store per-file system and user-defined metadata. This metadata can be used to implement application specific security policies or other things needed for your business logic (say, per-file user specified tags). We focused on a particular submodule Api.File which acts as a thin wrapper around format-specific modules (ImageStore, VideoStore, DocStore and the FileStore).

  • Part V introduces the Web layer which exposes our object store to the web though endpoints: /upload and /file/:id. We also peeked into what other elements a typical Web layer may have: REST/GraphQL endpoints, setting context with UserID obtained from authorization header. We used simple and elegant plug abstraction for providing an HTTP interface with routing, avoiding use of any web frameworks.


See also