Linux

This feature has now been accepted and merged in the upstream kernel and will be part of kernel release 5.9. This post has been updated to match the upstream version of this feature.

In my previous post, I described how on-demand compaction scheme hurts hugepage allocation latencies on Linux. To improve the situation, I have been working on Proactive Compaction for the Linux kernel, which tries to reduce higher-order allocation latencies by compacting memory in the background.

Some drivers needs to allocate almost all memory as hugepages to reduce (on-device or CPU) TLB pressure. However, on a running system, higher order allocations can fail if the memory is fragmented. Linux kernel can do on-demand compaction as we request more hugepages but this style of compaction incurs very high latency.

To show the effect of on-demand compaction on hugepage allocation latency, I created a test program “frag” which allocates almost all available system memory followed by freeing $\frac{3}{4}$ of pages from each hugepage-sized aligned chunk. This allocation pattern results in ~300% fragmented address space w.r.t order 9 i.e. physical mappings of our VA space is spread over 3x the number of hugepage-aligned chunks than what is ideally required.

There is no official Google drive client for Linux. I tried many different clients found all over GitHub but none of them worked reliably for me except rclone. I also tried third-party proprietary clients like Insync but allowing read-write access to all your Google drive files to a closed source blob is too much to swallow.

Once caveat with rclone is that it does not natively support bi-directional sync (github issue) but someone developed a python script rclonesync-V2 which is a wrapper around rclone which does the job. With these two pieces of software we can get close-to-official Google drive client experience.

Often, you have more than one system at your disposal but no clear way of distributing your compilation workloads over to all or some of them. They might be running different OSes which makes it look even more difficult. In my case, I have one laptop (2 cores) and a desktop (4 cores) connected with a WiFi network. The laptop runs Linux (Fedora 13 64-bit) while the desktop runs Windows 7 (64-bit). I wanted to somehow offload Linux kernel compilation over to my powerful desktop and keep my laptop cool :)

Recently, I developed Linux kernel driver which creates generic RAM based compressed block devices (called zram). Being RAM disks, they do not provide persistent storage but there are many use cases where persistence is not required: /tmp, various caches under /var, swap disks etc. These cases can benefit greatly from high speed RAM disks along with savings which compression brings!

However, all this seems to be completely Linux centric. But with virtualization, zram can be used for Windows too! The trick is a expose zram as a ‘raw disk’ to Windows running inside a Virtual Machine (VM). I will be using VirtualBox as example but exposing raw disks should be supported by other Virtualization solutions like VMware, KVM too.

Since a long time, I was looking for a graphical git diff viewer which could show original and modified file side-by-side and highlight the changes. There are few solutions but none of them is sufficient:

• A tool included with git called ‘git-difftool’ is partially helpful – it can show changes graphically but diff for each file is shown one-by-one. This is very irritating. In fact, unusable even with just 10-15 files.
• Another alternative is the meld diff viewer which is “git aware”. The problem here is that it can show diff for uncommitted changes only which is very limiting. What if you want to see what changes between Linux kernel, say 2.6.33-rc1 and 2.6.33-rc2? or changes between last two commits? meld cannot do it, AFAIK.
• Finally, with kompare, you can do something like: ‘git diff master | kompare -o -’. This method however, does not show original and new files side-by-side. It is simply prettier diff highlighting.

None of above methods are sufficient. So, I wrote the following script which solves our problem: show complete contents of original and new files side-by-side and highlight the differences.

You worked on some part of Linux kernel. It works great. Now, how to generate the patch series and send it out for review? For this, I always used to generate diffs, create a set of draft mails (one for each patch) in KMail or Thunderbird, and send all these mails one-by-one. This workflow quickly became a big headache. Then I learned Git (and some related tools) to do all this from command line and wow! what a relief!

In case you are unfamiliar with ccache, its a “compiler cache”. Compiling is primarily CPU intensive task. So, ccache caches compiled objects - so next time we compile same code, it reuses these objects thereby significantly speeding-up compilation.

I need to recompile Linux kernel usually several times a day, with different permutations of config settings. This almost forces a ‘make clean’ or ‘make mrproper’ which deletes all compiled objects in build tree and then we have to rebuild everything all over again. This takes enormous amount of time. ccache comes to rescue! I’m surprised why I didn’t use it earlier.

Linux kernel has few SLAB allocator variants included: SLAB, SLUB and SLOB. Of these, SLOB is especially meant to be used on embedded devices – it tries to be more memory space efficient than other SLAB variants.

Yesterday, I had a detailed look at SLOB allocator for possible use in compcache poject and found it unacceptable for the purpose. I did it in response to feedback on xvmalloc allocator – as part of compcache patches posted of inclusion in mainline Linux kernel: http://lkml.org/lkml/2009/3/17/116

One of my Fedora 10 systems used to freeze very frequently. After lot of looking around I found its because of “KWin Composing” which gives OpenGL driven special effects for desktop. Unfortunately, Linux has always been bad at radeon drivers, so it better to disable these effects especially if you have radeon video cards.

in ~/.kde/share/config/kwinrc:

in [Compositing] section change Enabled=true to Enabled=false.

Reboot after this change. Now I never get any system freeze - as is expected from solid Linux system :)