Proactive Compaction

for the Linux kernel

Proactive Compaction
In my previous post, I described how on-demand compaction scheme hurts hugepage allocation latencies on Linux. To improve the situation, I have been working on Proactive Compaction for the Linux kernel, which tries to reduce higher-order allocation latencies by compacting memory in the background. Design The Linux kernel uses per-NUMA-node threads called kcompactdN (where N is the NUMA node ID), which are responsible for compacting their corresponding NUMA node. We reuse these per-node threads to do proactive compaction: each of these threads now periodically calculates per-node “proactive compaction score”, and checks it against a threshold. [Read More]

Linux kernel hugepage allocation latencies

A detailed analysis

Linux kernel hugepage allocation latencies
Some drivers needs to allocate almost all memory as hugepages to reduce (on-device or CPU) TLB pressure. However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel can do on-demand compaction as we request more hugepages but this style of compaction incurs very high latency. To show the effect of on-demand compaction on hugepage allocation latency, I created a test program “frag” which allocates almost all available system memory followed by freeing $\frac{3}{4}$ of pages from each hugepage-sized aligned chunk. [Read More]

A layered object store design in Elixir (Part VI)

Summary

We built an object store from scratch in Elixir using a layered design approach. The overall theme has been to avoid generalizing the design too much which kept implementation of each layer/module simple. We were also careful when adding any third-party dependencies which has multiple advantages: deeper understanding of your codebase, easier debugging (I hate unknown code-paths in backtraces). For reference, here are links for all five parts along with their summaries: [Read More]

A layered object store design in Elixir (Part V)

The Web layer

Part I, introduces the overall design of our object store. In this post we focus on the Web layer. This is the final layer for our object store responsible for exposing it over the web. It will expose endpoints: /upload for uploading a file and /file/:file_id for getting a file by ID. A typical GraphQL application with also expose endpoint /graphql which directly plugs into your API layer, however I will not discuss this part and stay focused on the object store side of things. [Read More]

A layered object store design in Elixir (Part IV)

The API layer

Part I, introduces the overall design of our object store. In this post we focus on the API layer. All layers till now were just concerned about storing the input file together with some file-format specific transforms (like thumbnails). It is at the API layer where we will be storing per-file system and user metadata. This metadata can be used to support application specific business logic and security policies. This layer will depend on all per-file-format modules: ImageStore, VideoStore, etc. [Read More]

A layered object store design in Elixir (Part III)

ImageStore and VideoStore

Part I layer. ImageStore The ImageStore module is responsible for storing images along with their thumbnail. It will use the FileStore layer to actually store files on disk. Before we define module interfaces, lets see our application requirements: All images must be stored in the jpg format. Images cannot be larger than 1920x1080. We do not want to store user provided version at all. Thumbnails should use the same jpg format. [Read More]

A layered object store design in Elixir (Part II)

The FileStore layer

A layered object store design in Elixir (Part II)
Part I, introduces the overall design of our object store. In this post we focus on its first layer, the FileStore. The FileStore layer is responsible for actually storing the file in our object store. At this level, we are not concerned about what kind of file it is (image, video, document, or whatever else), nor do we have any notion of security. We just store whatever input path is given to us. [Read More]

A layered object store design in Elixir (Part I)

Introduction

A layered object store design in Elixir (Part I)
I recently designed an object store from scratch in Elixir. It has been serving me well as a backend for an app which needs to store all kinds of files: images, videos, documents. I wanted something simple to avoid dealing with off-the-shelf object stores which require complex configurations and to avoid cloud storage which is dead simple to use but can get very expensive, very quickly. For this project, simplicity was the key to make sure I can debug any failures quickly. [Read More]

Elixir collections

Elixir is a function programming language that I have been using a lot in recent months to build all kinds of applications. Understanding of built-in collection types is essential to use any language effectively and Elixir is no different. This posts summarizes all collection type along with pros/cons/gotchas for each one of them. Collection Example When Tuples {:ok, "All good"} Returning data from a function Lists [1, "two", :three] For a collection of items Keyword lists [one: 1, two: 2] Passing options to a function Maps %{one: 1, two: 2} Flexible key/value store Structs %User{name: "John", age: 32} Typed/fixed key/value store Tuples {:ok, foo} “tagged” tuple since begins with an atom like :ok or :error like {:error, 543, "some error"} Examples: [Read More]

Why I moved from Fedora to Ubuntu

Fedora 31 -> Ubuntu 18.04 I have been using Fedora since it was called ‘RedHat Linux’ and then ‘Fedora Core 1’ … Back then I found all other distros, including Ubuntu, Suse, less stable and usable. I mostly use of Ubuntu on work machines out of requirement. However, I recently got a new laptop and decided to try out Fedora 31. I was disappointed - the distro suffers from many of the same issues I used to deal with Fedora from nearly 10-15 years back: [Read More]