Proactive Compaction

for the Linux kernel

Proactive Compaction
In my previous post, I described how on-demand compaction scheme hurts hugepage allocation latencies on Linux. To improve the situation, I have been working on Proactive Compaction for the Linux kernel, which tries to reduce higher-order allocation latencies by compacting memory in the background. Design The Linux kernel uses per-NUMA-node threads called kcompactdN (where N is the NUMA node ID), which are responsible for compacting their corresponding NUMA node. We reuse these per-node threads to do proactive compaction: each of these threads now periodically calculates per-node “proactive compaction score”, and checks it against a threshold. [Read More]

Linux kernel hugepage allocation latencies

A detailed analysis

Linux kernel hugepage allocation latencies
Some drivers needs to allocate almost all memory as hugepages to reduce (on-device or CPU) TLB pressure. However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel can do on-demand compaction as we request more hugepages but this style of compaction incurs very high latency. To show the effect of on-demand compaction on hugepage allocation latency, I created a test program “frag” which allocates almost all available system memory followed by freeing $\frac{3}{4}$ of pages from each hugepage-sized aligned chunk. [Read More]

Why I moved from Fedora to Ubuntu

Fedora 31 -> Ubuntu 18.04 I have been using Fedora since it was called ‘RedHat Linux’ and then ‘Fedora Core 1’ … Back then I found all other distros, including Ubuntu, Suse, less stable and usable. I mostly use of Ubuntu on work machines out of requirement. However, I recently got a new laptop and decided to try out Fedora 31. I was disappointed - the distro suffers from many of the same issues I used to deal with Fedora from nearly 10-15 years back: [Read More]

Google Drive on Linux

There is no official Google drive client for Linux. I tried many different clients found all over GitHub but none of them worked reliably for me except rclone. I also tried third-party proprietary clients like Insync but allowing read-write access to all your Google drive files to a closed source blob is too much to swallow. Once caveat with rclone is that it does not natively support bi-directional sync (github issue) but someone developed a python script rclonesync-V2 which is a wrapper around rclone which does the job. [Read More]

Faster compilation with distcc

Often, you have more than one system at your disposal but no clear way of distributing your compilation workloads over to all or some of them. They might be running different OSes which makes it look even more difficult. In my case, I have one laptop (2 cores) and a desktop (4 cores) connected with a WiFi network. The laptop runs Linux (Fedora 13 64-bit) while the desktop runs Windows 7 (64-bit). [Read More]

Compressed RAM disk for Windows, The Virtual Way!

Recently, I developed Linux kernel driver which creates generic RAM based compressed block devices (called zram). Being RAM disks, they do not provide persistent storage but there are many use cases where persistence is not required: /tmp, various caches under /var, swap disks etc. These cases can benefit greatly from high speed RAM disks along with savings which compression brings! However, all this seems to be completely Linux centric. But with virtualization, zram can be used for Windows too! [Read More]

Setting up KDE API Documentation

Since last few months, I have been playing around with KDevelop which is a KDE based IDE for C/C++ any many other languages. Its a large C++ codebase and navigating through all the files, classes is quite difficult with usual VI + cscope combination. The most lacking part is a readily accessible KDE API documentation which is almost essential, no matter what component you are working on. There is a KDE API reference site but searching there for every reference is very cumbersome. [Read More]

Comprehensive graphical Git diff viewer

Comprehensive graphical Git diff viewer
Since a long time, I was looking for a graphical git diff viewer which could show original and modified file side-by-side and highlight the changes. There are few solutions but none of them is sufficient: A tool included with git called ‘git-difftool’ is partially helpful – it can show changes graphically but diff for each file is shown one-by-one. This is very irritating. In fact, unusable even with just 10-15 files. [Read More]

Linux kernel workflow with Git

You worked on some part of Linux kernel. It works great. Now, how to generate the patch series and send it out for review? For this, I always used to generate diffs, create a set of draft mails (one for each patch) in KMail or Thunderbird, and send all these mails one-by-one. This workflow quickly became a big headache. Then I learned Git (and some related tools) to do all this from command line and wow! [Read More]

Fedora 11 on ThinkPad W500

This was the release I was waiting for! The installation was a bit of a trouble but it was surely worth the effort. Its really the first Fedora release that I really liked. Its stable, has good hardware support and boots really quickly (<20 seconds on my system). In my case, Windows 7 RC was already installed on one partition, so Fedora 11 (x64) was installed in second partition. So, now I have dual boot configuration – the windows boot loader presents option to continue booting windows or launch GRUB from Linux partition. [Read More]