Proactive Compaction

for the Linux kernel

This feature has now been accepted and merged in the upstream kernel and will be part of kernel release 5.9. This post has been updated to match the upstream version of this feature. In my previous post, I described how on-demand compaction scheme hurts hugepage allocation latencies on Linux. To improve the situation, I have been working on Proactive Compaction for the Linux kernel, which tries to reduce higher-order allocation latencies by compacting memory in the background. [Read More]

Linux kernel hugepage allocation latencies

A detailed analysis

Some drivers needs to allocate almost all memory as hugepages to reduce (on-device or CPU) TLB pressure. However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel can do on-demand compaction as we request more hugepages but this style of compaction incurs very high latency. To show the effect of on-demand compaction on hugepage allocation latency, I created a test program “frag” which allocates almost all available system memory followed by freeing $\frac{3}{4}$ of pages from each hugepage-sized aligned chunk. [Read More]

There is no official Google drive client for Linux. I tried many different clients found all over GitHub but none of them worked reliably for me except rclone. I also tried third-party proprietary clients like Insync but allowing read-write access to all your Google drive files to a closed source blob is too much to swallow. Once caveat with rclone is that it does not natively support bi-directional sync (github issue) but someone developed a python script rclonesync-V2 which is a wrapper around rclone which does the job. [Read More]

Faster compilation with distcc

Often, you have more than one system at your disposal but no clear way of distributing your compilation workloads over to all or some of them. They might be running different OSes which makes it look even more difficult. In my case, I have one laptop (2 cores) and a desktop (4 cores) connected with a WiFi network. The laptop runs Linux (Fedora 13 64-bit) while the desktop runs Windows 7 (64-bit). [Read More]

Compressed RAM disk for Windows, The Virtual Way!

Recently, I developed Linux kernel driver which creates generic RAM based compressed block devices (called zram). Being RAM disks, they do not provide persistent storage but there are many use cases where persistence is not required: /tmp, various caches under /var, swap disks etc. These cases can benefit greatly from high speed RAM disks along with savings which compression brings! However, all this seems to be completely Linux centric. But with virtualization, zram can be used for Windows too! [Read More]

Setting up KDE API Documentation

Since last few months, I have been playing around with KDevelop which is a KDE based IDE for C/C++ any many other languages. Its a large C++ codebase and navigating through all the files, classes is quite difficult with usual VI + cscope combination. The most lacking part is a readily accessible KDE API documentation which is almost essential, no matter what component you are working on. There is a KDE API reference site but searching there for every reference is very cumbersome. [Read More]

Comprehensive graphical Git diff viewer

Since a long time, I was looking for a graphical git diff viewer which could show original and modified file side-by-side and highlight the changes. There are few solutions but none of them is sufficient: A tool included with git called ‘git-difftool’ is partially helpful – it can show changes graphically but diff for each file is shown one-by-one. This is very irritating. In fact, unusable even with just 10-15 files. [Read More]

Linux kernel workflow with Git

You worked on some part of Linux kernel. It works great. Now, how to generate the patch series and send it out for review? For this, I always used to generate diffs, create a set of draft mails (one for each patch) in KMail or Thunderbird, and send all these mails one-by-one. This workflow quickly became a big headache. Then I learned Git (and some related tools) to do all this from command line and wow! [Read More]