We built an object store from scratch in Elixir using a layered design approach. The overall theme has been to avoid generalizing the design too much which kept implementation of each layer/module simple. We were also careful when adding any third-party dependencies which has multiple advantages: deeper understanding of your codebase, easier debugging (I hate unknown code-paths in backtraces).
For reference, here are links for all five parts along with their summaries:
-
Part I introduces the overall design of our object store.
-
Part II introduces the
FileStoremodule which is responsible for actually storing input files to object store. -
Part III introduces file format-specific modules:
ImageStorefor storing normalizing input images together with thumbnails,VideoStorefor storing input video as is together with its thumbnail. These modules each deal with file format specific details while using theFileStorelayer to finally store resulting artifacts. -
Part IV introduces the API layer where we store per-file system and user-defined metadata. This metadata can be used to implement application specific security policies or other things needed for your business logic (say, per-file user specified tags). We focused on a particular submodule
Api.Filewhich acts as a thin wrapper around format-specific modules (ImageStore,VideoStore,DocStoreand theFileStore). -
Part V introduces the Web layer which exposes our object store to the web though endpoints:
/uploadand/file/:id. We also peeked into what other elements a typical Web layer may have: REST/GraphQL endpoints, setting context with UserID obtained from authorization header. We used simple and elegant plug abstraction for providing an HTTP interface with routing, avoiding use of any web frameworks.