Reducing Boot Times on Arch Linux

After much blood, sweat, tears and other bodily fluids, I’ve finally managed to reduce boot times on my computer. Although I’ve most definitely surpassed the time savings with time wasted getting it working, it was still a great learning experience. To avoid the rather complicated and frustrating process it was to get this working, I’ll instead explain from start to finish what needs to be done, including workarounds for some of the less functional tasks.

Quick overview

EDIT: Due to the recent re-addition of the e4rat-lite package to the AUR, some of this process has been simplified, and adjustments have been made accordingly.

I want to start off by pointing out that the method used to accomplish this is not for the faint of heart, and some of the techniques and workarounds used take more advanced Linux and programming concepts to succeed. The method being applied uses a package called e4rat-lite to optimize file placement on HDDs to improve loading times. This method will (most likely) have no impact, if not worse performance for SSDs. Note also that the explanation given is for usage on Arch Linux, and that the process may vary (or be less complicated) on other systems

The process to getting this working goes as follows:

1. Compile custom Linux kernel with auditing support
2. Build and install Audit, with custom options
3. Edit, build and install e4rat-lite
4. (optional) configure boot benchmarks
5. Run e4rat-lite tools

At the end, I’ll also display some graphs to show the improvements this had made to my system.

Building custom kernel

Update: As of kernel version 4.18, this is enabled by default, and does not require manually building the kernel. Whoho!

e4rat uses the audit package to determine what files are being used during boot. As the kernel does not have audit enabled by default, a custom kernel must be used instead. In theory a pre-built kernel can be used for this purpose, such as the linux-hardened package (installed along with linux-hardened-headers). Alternatively, one can build a custom kernel with the necessary build flags.

To build the kernel, the packages asp and ones from the base-devel are required (notably GCC). To begin the customization of the kernel, the following can be done:

$ cd ~/
$ mkdir build
$ cd build/    #Note any directory can be used for this

$ ASPROOT=. asp checkout linux

Once this is completed, move into the linux/repos/core-x86_64 directory, where changes can be made to the package. An optional, though recommended modification in the PKGBUILD file is to modify the pkgbase option. This will allow the new kernel to be installed without having to replace the current on in place. Note that, once the modified kernel is used, it will no longer be needed. I replaced the name of the package (linux) with linux-audit.

Next, a couple modifications must be done to the config (previously config.x86_64):

CONFIG_AUDIT=y    #this line is modified

CONFIG_AUDITSYSCALL=y    #this line is added

After this the checksums must be updated with this command:

$ updpkgsums

This may take some time, as this will download the full Linux package. Upon completion, the building process can begin. Note that this will take some time (30 minutes in my case), and will depend on your CPU speed as well as the number of CPU threads being utilized. See my Arch Setup page for information for configuring this.

$ makepkg -s

Note: this command may require being executed with the addition of --skippgpcheck to avoid errors due to failed checks. While the kernel is compiling, some additional flags and options may be queried to be added. Simply choose y (or m) for all of these.

Once this is completed, the new packages can be installed. It is recommended to first install the headers before the kernel package.

$ pacman -U linux-audit-headers.tar.zx

$ pacman -U linux-audit.tar.zx

Please note, the package names will vary with Linux kernel versions. Upon completion, the new kernel can be added to the boot loader (grub customizer can be used for this).

Installing Audit

While normally audit would be a simple task to install, there’s a small change that needs to be made to the package build to correct an incompatibility with e4rat. Instead of automatic installation, a manual one must be done, for example:

$ yaourt -Sb audit

Yaourt has the -b option to build the package manually. When prompted to edit the PKGBUILD, the following modification must be made:

options=(’emptydirs’ ‘staticlibs’)

With this the installation should finish without issue.

Installing e4rat-lite

Thanks to the resurgence of the lite version of the package, this step is now quite simple. Just install the e4rat-lite-git package from the AUR, once the audit package is installed. That’s it. All done. Nothing more.

… hooray!

  (Optional) Packages for performance testing

The Arch Wiki recommends using Bootchart to measure the performance of the system while booting, and to generate nice graphs to demonstrate. While I could not successfully generate charts with the original Bootchart, Bootchart2 (also listed) worked, with a minor modification. Simply installing it, and adding a kernel parameter to boot allows it to run, though with a small change to what the wiki currently illustrates:

init=/sbin/bootchartd2

After this, the pybootchartgui -i command will generate an interactive graph.

Note that, in order to use bootchartd2 with e4rat-lite-preload, the init line in the /sbin/bootchartd2 file must be modified:

init=/usr/bin/e4rat-lite-preload

Running the e4rat tools

Now that all the prep-work is done, we can actually get to the performance improvements. I’m going to start off with a screenshot of my initial benchmark.

Note that, while the total runs past 45 seconds, the analysis terminates (in my case) once mutter (gnome shell) starts up. The drop in the CPU and disk usage is when I get to login. So I can say it takes about 40 seconds to get from boot menu to login screen, in my case.

To start the analysis, the first thing to do is to add a different init script to the kernel parameters, which the Arch Wiki has a detailed guide on accomplishing. Add the following:

init=/usr/bin/e4rat-lite-collect

Note: the argument audit=1 had been previously included here, but some users have indicated issues (segfaults, etc.) when included. This will enable auditing (in case it isn’t) and collect data about the files being opened during the first 120 seconds once the boot process begins. This timer is default, and can be changed via /etc/e4rat-lite.conf. Assuming it all worked successfully, a log will be generated in /var/lib/e4rat-lite/startup.log filled with everything that was audited. Once this is done, optimizations can be applied.

The first optimization is on the hard drive. The tool included will move files around on the disk, and place them in a more efficient way towards the exterior of the disk to make better use of read patterns and speeds. Note that it is recommended (though not required) to enter in recovery mode (systemctl isolate resuce) to run this, as less files will be used and un-movable. Run the following command the terminal:

$ e4rat-lite-realloc

The process shouldn’t take very long. It is recommended to run the command a few times to assure that all files possible are moved. Note that, as far as I could find, no portion of the Linux kernel is directly included in this, meaning that performance should not change when choosing a different kernel, or upgrading one. Once this was completed, I ran another benchmark to plot the immediate difference.

In case it isn’t completely obvious, my boot time (to login) was reduced to just over 30 seconds, meaning an approximate 10 second improvement. This however is not the last component of the e4rat set of tools. The last one is intended (from what I understand) to cache files early on for later usage, to avoid delays. To accomplish this, a (permanent) init script is used:

init=/usr/bin/e4rat-lite-preload

One thing to note is that this script, unlike the previous, and contrary to some forum posts, does not require audit, so it can be used on any kernel installation without the need to compile a custom kernel for each usage. The following are my results with it:

What makes it clear that the preload is working is that, unlike the previous charts, there is no dip in this one. The preload is caching files the whole time, so it’s actually quite difficult to tell when the login screen appears. I estimate it to be around the 25 to 30 second point. I didn’t feel I noticed a significant difference leading up to the login screen. What I did notice however, was the difference made once logging in. As I have several applications on startup, they were open almost immediately after login, since the files were pre-loaded.

I also compared this tool with a different one called systemd-readahead, which accomplishes a similar task as the pre-load. Not only did it make no notable difference, in a few runs in actually increased boot times than ones without it. I’m not sure if I perhaps mis-configured it (it’s easy to setup), or I did not do enough boots for it to be optimal (I tried about 4 times). Regardless, this tool made no impact on booting.

Plymouth

One thing I had considered would make a difference in boot times is plymouth, as it just adds a splash screen while booting, which no doubt adds some overhead. I took the liberty of comparing boot times without it enabled, for comparison. First, before the disk optimization:

Note that, in this case, it did reduce my boot times by about 5 seconds.Next is the run after the optimization, again without plymouth enabled:

This can go to show the difference the disk optimization made. Here, there’s only approximately 1 second of difference, so it’s likely there was some bottleneck loading plymouth that lead to the increased delay.

Finally, the preload without plymouth:

Honestly, I see no difference here as with plymouth enabled. Needless to say, there isn’t enough of a significant difference to boot without it. That, and I like the splash screen.

Conclusion

While a 10 second boot time reduction may be more significant to some more than others, I don’t find it overly noteworthy the time it takes to get to the login prompt (GDM in my case). This is particularly emphasized by the 30 seconds it still takes to login. What I do notice however is the responsiveness once I get to the prompt, and after logging in. Normally, there is a delay between when I see the login screen, and when I can actually enter my password. Now, I find this practically non-existent. I can’t really boast about time savings, considering the effort put into getting it working definitely outweighs this, but it was an interesting learning experience. While there is definitely room for improvement for optimization of boot times (particularly off hard disks), there doesn’t seem to be much interest for developers to tackle this, what with the rising popularity of SSDs. Either way, it’s interesting to try and see what can make a difference, even if small.