Part I, introduces the overall design of our object store. In this post we focus on the Web layer. This is the final layer for our object store responsible for exposing it over the web. It will expose endpoints: /upload for uploading a file and /file/:file_id for getting a file by ID. A typical GraphQL application with also expose endpoint /graphql which directly plugs into your API layer, however I will not discuss this part and stay focused on the object store side of things.

Lets start by creating a new mix project:

mix new --sup web

This project depends on the Api module we developed in part 3 and on plug_cowboy which we use for providing HTTP server and a simple router. Note that there is an excellent Phoenix Web Framework for Elixir but I tend to avoid using it since plug, cowboy together have always been sufficient for me and in-fact, simpler and more transparent to work with. A well designed Web layer should be just a dumb web interface for your API layer; a web framework of any kind generally seems like an overkill to me.

For our Web layer, lets begin with routing.

Routing

Lets define a Web.Endpoint module in lib/web/endpoint.ex to define our routing structure:

/upload for uploading files. Sends incoming file to Api.File.put_file.
/file/:file_id for getting files by ID. Note that in Plug.Router DSL, the :file_id part gives us part of URL after /file/ which we assume as id to file file to be fetched. Send incoming file request to Api.File.get_file.
/graphql and /graphiql to handle GraphQL queries and mutations (as may be defined in your API layer). These are not required for our object store but I have included them here to show a better overall picture of what the Web layer may involve in a real application.

defmodule Web.Endpoint do
  @moduledoc """
  A Plug responsible for logging request info, parsing request body's as JSON,
  matching routes, and dispatching responses.
  """

  use Plug.Router

  # This module is a Plug, that also implements it's own plug pipeline, below:

  # Using Plug.Logger for logging request information
  plug(Plug.Logger)
  # responsible for matching routes
  plug(:match)

  plug(Web.Context)

  # Using Jason for JSON decoding
  # Note, order of plugs is important, by placing this _after_ the 'match' plug,
  # we will only parse the request AFTER there is a route match.
  plug(Plug.Parsers,
    parsers: [:multipart, :json, Absinthe.Plug.Parser],
    pass: ["*/*"],
    json_decoder: Jason
  )

  # responsible for dispatching responses
  plug(:dispatch)

  forward("/graphql",
    to: Absinthe.Plug,
    init_opts: [
      schema: Api.Schema
    ]
  )

  forward("/graphiql",
    to: Absinthe.Plug.GraphiQL,
    init_opts: [
      schema: Api.Schema
    ]
  )

  post "/upload" do
    file_param = conn.body_params["file"]
    tags_param = conn.body_params["tags"]
    context = conn.private[:web][:context]
    user_id = Map.get(context, :user_id)

    if user_id === nil do
      send_resp(conn, 401, "You must be signed in for file uploads.")
    else
      case Api.File.put_file(user_id, file_param.content_type, file_param.path, tags_param) do
        {:ok, file_id} ->
          send_resp(conn, 200, Jason.encode!(%{file_id: file_id}))

        {:error, reason} ->
          send_resp(conn, 401, reason)
      end
    end
  end

  get "/file/:file_id" do
    # strip "extension" (like .pdf) from file_id (if any)
    file_id = file_id |> String.split(".") |> List.first()

    {res_code, res_data} =
      case Api.File.get_file(file_id) do
        {:error, _} -> {404, "File not found"}
        data -> {200, data}
      end

    send_resp(conn, res_code, res_data)
  end

  # A catchall route, 'match' will match no matter the request method,
  # so a response is always returned, even if there is no route to match.
  match _ do
    send_resp(conn, 404, "oops... Nothing here :(")
  end
end

Build your context

A typical responsibility of the Web layer is to parse authorization header to get UserID of the user making the request. This UserID is expected by the API layer to apply application-specific security policies.

Here is an example module which sets up the context which we used above in routes like post "/upload".

defmodule Web.Context do
  @moduledoc false

  @behaviour Plug

  import Plug.Conn

  def init(opts), do: opts

  def call(conn, _) do
    context = build_context(conn)

    conn
    |> Plug.Conn.put_private(:web, %{context: context})
  end

  @doc """
  Return the current user context based on the authorization header
  """
  def build_context(conn) do
    with ["Bearer " <> token] <- get_req_header(conn, "authorization"),
         {:ok, user_id} <- authorize(token) do
      %{user_id: user_id}
    else
      _ -> %{}
    end
  end

  defp authorize(token) do
    case Api.Mutations.verify_auth_token(token) do
      {:ok, claims} -> {:ok, claims["user_id"]}
      {:error, error_reason} -> {:error, error_reason}
    end
  end
end

Here, Api.Token.verify_auth_token would be your implementation of Web Token verification: the Joken package can be used for this purpose. I will not include those details here.

Curl it out

This the Web layer in place, we can finally see our object store in action. Lets start by downloading a sample image:

curl "https://upload.wikimedia.org/wikipedia/commons/c/cc/Resolution_of_SD%2C_Full_HD%2C_4K_Ultra_HD_%26_8K_Ultra_HD.svg" -o /tmp/sample.svg

Here is some image information obtained from identify /tmp/sample.svg:

/tmp/sample.svg SVG 7680x4320 7680x4320+0+0 16-bit sRGB 22518B 0.000u 0:00.006

Lets curl this image to our object store:

curl -F 'file=@/tmp/sample.svg' http://localhost:4001/upload

{"file_id":"t42fzzdypcsr47lvn3vkgxxs6sktl3axye4c2tsbbtm2rusvs7ea"}

This curl command is like submitting a <Form> with the file field set to input file contents. If the file is successfully stored, we get back its file_id.

Note: Our demo endpoint is over HTTP and I have removed check against UserID which we get from the context (which in turn gets it from authentication token in request’s authorization header as: “Bearer <token>”). This was done to keep this demo simple and to keep this post focused on object-store specific details only. In a real scenario, this would be a big NO. You almost always require communication over HTTPS together with some UserID encoded in authentication token, so you can apply application-specific security policies before accepting any incoming file.

Now, lets fetch the file with this file_id:

curl -O http://localhost:4001/file/t42fzzdypcsr47lvn3vkgxxs6sktl3axye4c2tsbbtm2rusvs7ea.jpg

Note that we added the .jpg extension which is technically not part of file_id. Our GET /file/:file_id endpoint strips out any extension added after the file_id part. Allowing users to add an arbitrary file extension like this is helpful in some contexts. For example, when a browser fetches an asset with a .pdf extension, it can open up in-browser viewer instead of just downloading binary data.

identify t42fzzdypcsr47lvn3vkgxxs6sktl3axye4c2tsbbtm2rusvs7ea.jpg

t42fzzdypcsr47lvn3vkgxxs6sktl3axye4c2tsbbtm2rusvs7ea.jpg JPEG 1920x1080 1920x1080+0+0 8-bit sRGB 88374B 0.000u 0:00.000

Note that the original file’s format was SVG with a resolution of 7680x4320. What we got back from our object store was its “normalized” version with a JPG file format and 1920x1080 resolution. If you remember from Part 3, our ImageStore had baked in application-specific requirement of always storing files in the JPG format and that full-size images should not be larger than 1920x1080 (resize-to-fit as needed). Your requirements may of course be different and thus would bake in different set of parameters. Don’t generalize unnecessarily.

Also note that there is no way to fetch thumbnails through this curl interface – you would typically expose that through some other REST/GraphQL interface where, say for a GraphQL schema, a User type’s ProfilePicture field (of graphql type Image) would expose thumbnailFileId as one of its sub-fields.

Overall, this is the code flow triggered by above curl commands:

When storing image:

post "/upload" route defined in Web.Endpoint
    -> Api.File.put_file
        -> ImageStore.put_image_lossy
            -> normalize full-size image |> FileStore.put_file
            -> generate thumbnail |> FileStore.put_file
        -> Store files metadata in `files` SQL table
    -> return file_id as HTTP response, on success

When fetching file by ID:

get "/file/:file_id" route defined in Web.Endpoint
    -> Strip out any "extension" (like ".pdf") after the the file_id part
    -> Api.File.get_file(file_id)
        -> FileStore.get_file(file_id)

Summary

The Web layer exposes our object store over the web though endpoints: /upload and /file/:id. We also peeked into what other elements a typical Web layer may have: REST/GraphQL endpoints, setting context with UserID obtained from authorization header, and so on. We used simple and elegant plug abstraction for providing an HTTP interface with routing, avoiding use of any web frameworks: though Phoenix is quite modular, it may not be needed for thin Web layers (as they should be).

elixir

A layered object store design in Elixir (Part V)

The Web layer

Routing

Build your context

Curl it out

Summary

See also