Part I layer.
ImageStore
The ImageStore module is responsible for storing images along with their thumbnail. It will use the FileStore layer to actually store files on disk. Before we define module interfaces, lets see our application requirements:
- All images must be stored in the
jpgformat. - Images cannot be larger than 1920x1080. We do not want to store user provided version at all.
- Thumbnails should use the same
jpgformat. - All thumbnails must have the same size of 256x256.
Note that we are going for highly application specific requirements rather than a more general, configurable design. I have seen most of the complexity in software stacks is due to the temptation of making them “reusable”. As you will see, the implementation is going to be so simple, with clearly defined interfaces, that it would be much easier for you to create such a module for each of your applications, with its specific requirements baked in.
Now, with the requirements laid out, we need to define module interface:
-
def put_image_lossy(datadir, input_path): Normalizes image to the format and size we set in our requirements, generates thumbnail of the size defined, and finally stores both of them on disk using the FileStore module. Returns {:ok, %Image{}} if all of these steps are successful, or {:error, reason} on failure. For the blog, I have skipped proper error checking and assert successful completion of individual steps, which will cause the calling context to “crash”. A more complete error handling exhaustive pattern matching at each step which I leave up to you but you should understand that this kind of “error handling” may not be okay for your application. -
def get_image_dimensions(input_path): Returns [width, height] pair for the input image.
Now with interfaces laid out, let’s define each one of them. We will start with creating a new module:
mix new image_store
Note that we don’t need a supervision tree or any stateful application: it’s a pure library component. Also, it does not have any external dependencies apart from the FileStore module we created in Part II and the FileUtil module (see below). For normalizing the input image and for creating thumbnails, we use an external program called ImageMagick. There are great Elixir wrappers for it: Mogrify for image manipulation and Thumbnex for thumbnail generation. However, I try not to introduce any dependencies unless I really have to. For ImageStore, all we have to do is invoke a few external commands, so I don’t see a need to use any of these. Adding dependencies should never be taken lightly.
The function definition will in the ImageStore module defined in lib/image_store.ex.
Constants and Structs
Here we bake in our application specific requirements. %Image{}% is the struct we return from put_image_lossy.
@fullsize %{width: 1920, height: 1080}
@thumb %{width: 256, height: 256}
@format "jpg"
defmodule Image do
defstruct [:file_id, :width, :height, :thumb_file_id]
end
Put Image Lossy
@doc """
Resizes the input image to @fullsize size with the @format format. Also, creates
a thumbnail of dimension @thumbnail.
Returns:
On success: {:ok, %Image{}}
On error: throws
"""
def put_image_lossy(datadir, input_path) do
input_path = String.replace(input_path, " ", "_")
fullsize_path = create_fullsize(input_path)
thumb_path = create_thumbnail(input_path)
[width, height] = get_image_dimensions(fullsize_path)
{:ok, fullsize_file_id} = FileStore.put_file(datadir, fullsize_path)
{:ok, thumb_file_id} = FileStore.put_file(datadir, thumb_path)
File.rm!(fullsize_path)
File.rm!(thumb_path)
{:ok,
%Image{file_id: fullsize_file_id, width: width, height: height, thumb_file_id: thumb_file_id}}
end
Lets fill in the blanks in this implementation: create_fullsize and create_thumbnail:
Create Fullsize Image
Here we are doing ‘resize to fit’ operation on the input image. This operation does not crop the image while retaining the aspect ratio. This means we may not get exactly the image size we set in @fullsize global. Also, we only resize images larger than @fullsize (note the ‘>’ character in the command); smaller images are not enlarged.
defp create_fullsize(input_path) do
fullsize_path = FileUtil.tmp_filename()
fullsize_dim = "#{@fullsize.width}x#{@fullsize.height}"
[cmd | args] =
String.split("convert -resize #{fullsize_dim}> -strip #{input_path} #{fullsize_path}")
{_, 0} = System.cmd(cmd, args)
fullsize_path
end
Create Thumbnail
Here we are doing ‘resize to fill’ operation on the input image. This operation retains the aspect ratio while cropping overflowing parts of image in the larger dimension. This guarantees that all thumbnails have exactly @thumb dimensions which is useful when displaying a gallery of thumbnails on your frontend (web/app).
defp create_thumbnail(input_path) do
thumb_path = FileUtil.tmp_filename()
thumb_dim = "#{@thumb.width}x#{@thumb.height}"
input_path = String.replace(input_path, " ", "_")
[cmd | args] =
String.split(
"convert -resize #{thumb_dim}^ -strip -gravity center -extent #{thumb_dim} #{input_path} #{thumb_path}"
)
{_, 0} = System.cmd(cmd, args)
thumb_path
end
Get Image Dimensions
This interface may not be useful in your application but ImageStore seems like a logical place for it. Here we are using ImageMagick’s identify command to get image dimensions.
@doc """
Returns input image dimensions as [width, height] in pixels.
"""
def get_image_dimensions(input_path) do
input_path = String.replace(input_path, " ", "_")
[cmd | args] = String.split("identify -format %w,%h #{input_path}")
{output, 0} = System.cmd(cmd, args)
[width, height] =
output
|> String.split(",")
|> Enum.map(fn s ->
{x, _} = Integer.parse(s)
x
end)
[width, height]
end
Creating Temporary Files
You many have noticed we are using FileUtil.tmp_filename() to generate a temporary filename. There are many ways to generate a temp filename but here is the one I use:
mix new file_util
lib/file_util.ex:
defmodule FileUtil do
@moduledoc false
def tmp_filename(extension \\ "") do
Path.join(System.tmp_dir!(), [UUID.uuid1(), ".", extension])
end
end
This module has one dependency: uuid.
Security
Directly feeding user uploaded images to this module can expose your website/app to security attacks. I’m not a security expert but we can surely think of some steps to reduce the risks:
- Run ImageMagick commands as an unprivileged:
su - nobody .... - Run it under docker:
docker exec .... - By default ImageMagick tries to detect input image format and in case it gets that wrong, it can end up using a wrong decoder which is like randomized fuzz testing in production environment! You can reduce the risk by verifying image headers yourself and explicitly tell
convertthe input image format you detected. - Check image file size: processing a large image can get your application OOM killed or bring system to a halt, depending on your system configuration.
These steps can reduce the risks but you can never be sure. I hope such encoders/decoders are eventually written in safer languages like Rust. I have some hopes from the image-rs but I don’t think it’s there yet.
VideoStore
We now come to the VideoStore module which is responsible for storing video files. The details are mostly the same as ImageStore module, however, keeping them separate helps maintain a smaller interface for each module which helps with maintainability. In case we decide to do much elaborate business for videos, like transcoding all uploaded video into device-specific formats etc., the extra logic will not clutter our simplistic ImageStore module. Separate modules also help keep dependencies separate.
Lets start with creating a new module:
mix new video_store
Now lets define the module interface:
def put_video(datadir, input_path): Stores input video as-is along with its thumbnail. Returns{:ok, %Video{}}on success, throws on failure.def video_info(input_path): On success returns{:ok, [width, height, duration]}where width, height is in pixels and the duration is rounded down to seconds. On error,{:error, reason}is returned.
The %Video{} struct is defined as:
defmodule Video do
defstruct [:file_id, :width, :height, :duration, :thumb_file_id]
end
With interfaces defined, lets move to the implementation. Note that the module is again a pure library component. Also, it depends only on our internal modules: FileStore (for actual file storage), ImageStore (for thumbnail generation), and FileUtil (for generating temp filenames). Any decoding of the input video is delegated to ffmpeg. Again, a good Elixir wrapper exist for it, like ffmpex but we don’t use it here since all we have to do here is invoke a few ffmpeg commands and we want precise control over the flags we pass to this command – wrappers often make it unclear.
Put Video
@doc """
This function stores the input video file as-is together with its thumbnail (more on this later).
Retuns {:ok, %Video{}} on success, {:error, reason} on failure.
"""
def put_video(datadir, input_path) do
input_path = String.replace(input_path, " ", "_")
case extract_frame(input_path) do
{:ok, frame_file_path} ->
thumb_path = ImageStore.create_thumbnail(frame_file_path)
{:ok, video_file_id} = FileStore.put_file(datadir, input_path)
{:ok, thumb_file_id} = FileStore.put_file(datadir, thumb_path)
File.rm!(thumb_path)
{:ok, [width, height, duration]} = video_info(input_path)
{:ok,
%VideoStore.Video{
file_id: video_file_id,
width: width,
height: height,
duration: duration,
thumb_file_id: thumb_file_id
}}
{:error, error} ->
{:error, error}
end
end
Video Info
Extracts video info using the ffprobe command:
ffprobe -v error -select_streams v:0 -show_entries stream=width,height,duration -of csv=nk=1:p=0 file_path
where in "csv=nk=1:p=0"
nk=1 = don't print key (like: width, height, ...)
p=0 = don't print section (like: stream)
We parse output of this command which has format like “width,height,duration”, ex: “1280,720,119.666667”. For our application, we round down the duration to nearest second.
@doc """
Returns:
On success, {:ok, [width, height, duration]}
On error, {:error, reason}
"""
def video_info(input_path) do
input_path = String.replace(input_path, " ", "_")
[cmd | args] =
String.split(
"ffprobe -v error -select_streams v:0 -show_entries stream=width,height,duration -of csv=nk=1:p=0 #{
input_path
}"
)
res = System.cmd(cmd, args)
case res do
{output, 0} ->
{:ok,
output
|> String.split(",")
|> Enum.map(fn s ->
{x, _} = Integer.parse(s)
x
end)}
error ->
{:error, error}
end
end
Extract video frame to generate thumbnail
If you see the put_video function, we are generating video file thumbnail in a two step process: we first extract a video frame (say, 10 seconds into the video) and then feed this frame to ImageStore.create_thumbnail to actually generate a thumbnail. To extract a video frame, we use ffmpeg like this:
ffmpeg -i InputFile.mp4 -vframes 1 -an -ss 30 OutputFile.jpg
-v quiet = logging level
-i = Inputfile name
-vframes 1 = Output one frame
-an = Disable audio
-ss 30 = Grab the frame from 30 seconds into the video
Here’s the code to execute this command from elixir:
defp extract_frame(input_path) do
output_path = FileUtil.tmp_filename("jpg")
seconds_into_video = 10
[cmd | args] =
String.split(
"ffmpeg -i #{input_path} -vframes 1 -an -ss #{seconds_into_video} #{output_path}"
)
res = System.cmd(cmd, args)
case res do
{_, 0} -> {:ok, output_path}
error -> {:error, error}
end
end
Security
Again, similar security warnings and possible safeguards apply as for the ImageStore module.
DocStore, VRStore, …
You get the idea :)
Summary
We started with defining our application specific requirements for ImageStore and VideoStore modules and baked them into our implementation, avoiding generalizing these modules too much. Such an approach resulted in a very straightforward code for both modules. We also avoided external dependencies where possible. We also discussed some security implications of calling out highly vulnerable applications like ImageMagick, ffmpeg together with some possible safeguards.