Reducing boot times on Arch Linux

After much blood, sweat, tears and other bodily fluids, I’ve finally managed to reduce boot times on my computer. Although I’ve most definitely surpassed the time savings with time wasted getting it working, it was still a great learning experience. To avoid the rather complicated and frustrating process it was to get this working, I’ll instead explain from start to finish what needs to be done, including workarounds for some of the less functional tasks.

Quick overview

EDIT: Due to the recent re-addition of the e4rat-lite package to the AUR, some of this process has been simplified, and adjustments have been made accordingly.

I want to start off by pointing out that the method used to accomplish this is not for the faint of heart, and some of the techniques and workarounds used take more advanced Linux and programming concepts to succeed. The method being applied uses a package called e4rat-lite to optimize file placement on HDDs to improve loading times. This method will (most likely) have no impact, if not worse performance for SSDs. Note also that the explanation given is for usage on Arch Linux, and that the process may vary (or be less complicated) on other systems

The process to getting this working goes as follows:

  1. Compile custom Linux kernel with auditing support
  2. Build and install Audit, with custom options
  3. Edit, build and install e4rat-lite
  4. (optional) configure boot benchmarks
  5. Run e4rat-lite tools

At the end, I’ll also display some graphs to show the improvements this had made to my system.

Building custom kernel

e4rat uses the audit package to determine what files are being used during boot. As the kernel does not have audit enabled by default, a custom kernel must be used instead. In theory a pre-built kernel can be used for this purpose, but I didn’t stumble upon one, so compiling manually is the way to go.

To build the kernel, the packages asp and ones from the base-devel are required (notably GCC). To begin the customization of the kernel, the following can be done:

$ cd ~/
$ mkdir build
$ cd build/    #Note any directory can be used for this
$ ASPROOT=. asp checkout linux

Once this is completed, move into the linux/repos/core-x86_64 directory, where changes can be made to the package. An optional, though recommended modification in the PKGBUILD file is to modify the pkgbase option. This will allow the new kernel to be installed without having to replace the current on in place. Note that, once the modified kernel is used, it will no longer be needed. I replaced the name of the package (linux) with linux-audit.

Next, a couple modifications must be done to the config.x86_64:

CONFIG_AUDIT=y    #this line is modified

CONFIG_AUDITSYSCALL=y    #this line is added

After this the checksums must be updated with this command:

$ updpkgsums

This may take some time, as this will download the full Linux package. Upon completion, the building process can begin. Note that this will take some time (30 minutes in my case), and will depend on your CPU speed as well as the number of CPU threads being utilized. See my Arch Setup page for information for configuring this.

$ makepkg -s

Note: this command may require being executed with the addition of --skippgpcheck to avoid errors due to failed checks. While the kernel is compiling, some additional flags and options may be queried to be added. Simply choose y (or m) for all of these.

Once this is completed, the new packages can be installed. It is recommended to first install the headers before the kernel package.

$ pacman -U linux-audit-headers.tar.zx

$ pacman -U linux-audit.tar.zx

Please note, the package names will vary with Linux kernel versions. Upon completion, the new kernel can be added to the boot loader (grub customizer can be used for this).

Installing Audit

While normally audit would be a simple task to install, there’s a small change that needs to be made to the package build to correct an incompatibility with e4rat. Instead of automatic installation, a manual one must be done, for example:

$ yaourt -Sb audit

Yaourt has the -b option to build the package manually. When prompted to edit the PKGBUILD, the following modification must be made:

options=(’emptydirs’ ‘staticlibs’)

With this the installation should finish without issue.

Installing e4rat-lite

Thanks to the resurgence of the lite version of the package, this step is now quite simple. Just install the e4rat-lite-git package from the AUR, once the audit package is installed. That’s it. All done. Nothing more.

… hooray!

  (Optional) Packages for performance testing

The Arch Wiki recommends using Bootchart to measure the performance of the system while booting, and to generate nice graphs to demonstrate. While I could not successfully generate charts with the original Bootchart, Bootchart2 (also listed) worked, with a minor modification. Simply installing it, and adding a kernel parameter to boot allows it to run, though with a small change to what the wiki currently illustrates:

init=/sbin/bootchartd2

After this, the pybootchartgui -i command will generate an interactive graph.

Note that, in order to use bootchartd2 with e4rat-lite-preload, the init line in the /sbin/bootchartd2 file must be modified:

init=/usr/bin/e4rat-lite-preload

Running the e4rat tools

Now that all the prep-work is done, we can actually get to the performance improvements. I’m going to start off with a screenshot of my initial benchmark.

Baseline with plymouth

Note that, while the total runs past 45 seconds, the analysis terminates (in my case) once mutter (gnome shell) starts up. The drop in the CPU and disk usage is when I get to login. So I can say it takes about 40 seconds to get from boot menu to login screen, in my case.

To start the analysis, the first thing to do is to add a different init script to the kernel parameters, which the Arch Wiki has a detailed guide on accomplishing. Add the following:

audit=1 init=/usr/bin/e4rat-lite-collect

This will enable auditing (in case it isn’t) and collect data about the files being opened during the first 120 seconds once the boot process begins. This timer is default, and can be changed via /etc/e4rat-lite.conf. Assuming it all worked successfully, a log will be generated in /var/lib/e4rat-lite/startup.log filled with everything that was audited. Once this is done, optimizations can be applied.

The first optimization is on the hard drive. The tool included will move files around on the disk, and place them in a more efficient way towards the exterior of the disk to make better use of read patterns and speeds. Note that it is recommended (though not required) to enter in recovery mode (systemctl isolate resuce) to run this, as less files will be used and un-movable. Run the following command the terminal:

$ e4rat-lite-realloc

The process shouldn’t take very long. It is recommended to run the command a few times to assure that all files possible are moved. Note that, as far as I could find, no portion of the Linux kernel is directly included in this, meaning that performance should not change when choosing a different kernel, or upgrading one. Once this was completed, I ran another benchmark to plot the immediate difference.

Results without preload

In case it isn’t completely obvious, my boot time (to login) was reduced to just over 30 seconds, meaning an approximate 10 second improvement. This however is not the last component of the e4rat set of tools. The last one is intended (from what I understand) to cache files early on for later usage, to avoid delays. To accomplish this, a (permanent) init script is used:

init=/usr/bin/e4rat-lite-preload

One thing to note is that this script, unlike the previous, and contrary to some forum posts, does not require audit, so it can be used on any kernel installation without the need to compile a custom kernel for each usage. The following are my results with it:

Preload plymouth

What makes it clear that the preload is working is that, unlike the previous charts, there is no dip in this one. The preload is caching files the whole time, so it’s actually quite difficult to tell when the login screen appears. I estimate it to be around the 25 to 30 second point. I didn’t feel I noticed a significant difference leading up to the login screen. What I did notice however, was the difference made once logging in. As I have several applications on startup, they were open almost immediately after login, since the files were pre-loaded.

I also compared this tool with a different one called systemd-readahead, which accomplishes a similar task as the pre-load. Not only did it make no notable difference, in a few runs in actually increased boot times than ones without it. I’m not sure if I perhaps mis-configured it (it’s easy to setup), or I did not do enough boots for it to be optimal (I tried about 4 times). Regardless, this tool made no impact on booting.

Plymouth

One thing I had considered would make a difference in boot times is plymouth, as it just adds a splash screen while booting, which no doubt adds some overhead. I took the liberty of comparing boot times without it enabled, for comparison. First, before the disk optimization:

Baseline without plymouth

Note that, in this case, it did reduce my boot times by about 5 seconds.Next is the run after the optimization, again without plymouth enabled:

Results without plymouth

This can go to show the difference the disk optimization made. Here, there’s only approximately 1 second of difference, so it’s likely there was some bottleneck loading plymouth that lead to the increased delay.

Finally, the preload without plymouth:

Preload

Honestly, I see no difference here as with plymouth enabled. Needless to say, there isn’t enough of a significant difference to boot without it. That, and I like the splash screen.

Conclusion

While a 10 second boot time reduction may be more significant to some more than others, I don’t find it overly noteworthy the time it takes to get to the login prompt (GDM in my case). This is particularly emphasized by the 30 seconds it still takes to login. What I do notice however is the responsiveness once I get to the prompt, and after logging in. Normally, there is a delay between when I see the login screen, and when I can actually enter my password. Now, I find this practically non-existent. I can’t really boast about time savings, considering the effort put into getting it working definitely outweighs this, but it was an interesting learning experience. While there is definitely room for improvement for optimization of boot times (particularly off hard disks), there doesn’t seem to be much interest for developers to tackle this, what with the rising popularity of SSDs. Either way, it’s interesting to try and see what can make a difference, even if small.

Advertisements

14 thoughts on “Reducing boot times on Arch Linux

  1. Hi

    A very cool testing process. Many thanks for taking the time to share it with details.
    I’d love to run e4rat tools on my favorite monster (a 2010 Atom 5400 rpm hdd splendid laptop) that boots in 20 / 35 sec to cli / gui.
    Have compiled the Arch stock kernel with AUDIT=y, audit with “options (‘staticlibs’)”, e4rat and added the mandatory bootloaderentry. But e4rat collects nothing.I’d love to find out why.

    % uname -rms
    Linux 4.6.3-1-audit i686

    % zgrep -i audit /proc/config.gz

    CONFIG_LOCALVERSION=”-audit”
    CONFIG_AUDIT=y
    CONFIG_HAVE_ARCH_AUDITSYSCALL=y
    CONFIG_AUDITSYSCALL=y
    CONFIG_AUDIT_WATCH=y
    CONFIG_AUDIT_TREE=y
    # CONFIG_NETFILTER_XT_TARGET_AUDIT is not set
    CONFIG_INTEGRITY_AUDIT=y
    CONFIG_AUDIT_GENERIC=y
    # CONFIG_AUDIT_ARCH_COMPAT_GENERIC is not set

    % sudo journalctl -xb|grep e4rat

    juil. 11 23:24:49 gwenael kernel: Kernel command line: BOOT_IMAGE=../vmlinuz-linux-audit root=/dev/mapper/vg1-sys1b rw resume=UUID=b43bc921-65cb-4b55-9f7d-34b06d5dac14 audit=1 init=/sbin/e4rat-collect zswap.enabled=1 zswap.max_pool_percent=25 zswap.compressor=lz4 systemd.restore_state=0 initrd=../initramfs-linux-audit.img
    juil. 11 23:26:31 gwenael audit: EXECVE argc=9 a0=”ls” a1=”-p” a2=”–color=auto” a3=”–group-directories-first” a4=”-l” a5=”-G” a6=”-h” a7=”-p” a8=”/var/lib/e4rat/”
    juil. 11 23:26:31 gwenael audit: PATH item=0 name=”/var/lib/e4rat/” inode=428533 dev=fe:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL

    % ll /var/lib/e4rat/ (Over 5 minutes upon boot finished)
    total 0

    Do you believe the fact that I compiled a custom kernel with only the modules I’ve used over the last 6 months –or some other reason I can’t see– explains this emptyness ?

    Like

    • I’m glad to see someone else attempting what I’ve tried.

      As far as I can tell the issue is not likely the custom kernel parameters, as they seem to be configured correctly. What I think the issue is (based on the output you’ve provided) is that no files exist in /var/lib/e4rat/. With my experience, the startup.log file needed to be manually created (sudo touch /var/lib/e4rat/startup.log) in order for it to be written to. See if this helps.

      Liked by 1 person

  2. Hi! Glad you saw my comment.
    No worries I saw your note about touch e4rat log file manually and did that but it stayed empty. Thereafter I deleted it back and rebooted with the unlucky result above.

    Also, I recompiled Arch stock kernel with its modules and tried again, but no more luck –yet. I’ll get again and maybe I’ll stumb upon a little something I’m missing atm. Meanwhile if an idea flies up trough your mind please share it.

    Like

    • I’m wondering if it’s an issue with e4rat writing the output; perhaps the workaround for recompiling it doesn’t work anymore. Do you know if e4rat is attempted to be run at all? Perhaps modify the launch script for it to create a temporary file somewhere?

      Like

  3. Yes, I also put e4rat writing the output on the top possible issues. My reason is I can can perfectly see e4rat-collect starting when boot up begins. Also if I change e4rat log level to 31 it prints them for quite a while during startup. After what the output file is… empty.

    – auditd was disabled (default); I rebooted after enabling it.
    – my / partition is on LVM but not /boot. I tried after setting the output path to /boot/e4rat_startup.log
    No change –yet.

    Starting with the default Arch kernel (with AUDIT=y) journalctl gave two different outputs regarding e4rat, none that I can read.

    Unless you’re ok for me to go on polluting your article comments I’ll go share on the Arch forum.

    Like

  4. Yeah thanks, and done.
    Note: I followed up under your initial post there to help focusing the tiny bit of attention there’s for e4rat.

    Like

  5. Got e4rat-lite-{collect,realloc,preload} installed and running good, finally. “Terminator”-Dell mini now loads wi-fi, browser, word processing, loaaaads of emails, file manager and decrypt my vault in (much) less than 60 sec from cold boot. Houba hop !

    @cubethethird your paper was the initiator of this welcomed learning session. Now if you go back optimizing yours –which ain’t that low pro and boots slower :p– you may try to load your session a bit: I’m certain you’ll appreciate that ‘preload’ powaa!

    Like

  6. Thanks for the guide. but it doesn’t create startup.log. I tried editing config files, changed log directories and created empty log files but it doesn’t output log file. I am struggling for months to get this work but it doesn’t 😦

    Like

    • Are you using the e4rat-lite package, instead of the original e4rat? The original one gave me issues with the startup file and I had to originally manually edit and re-build the package. After re-uploading e4rat-lite to the AUR I removed those steps from this post since the issue was no longer present with it.

      Like

  7. Hi, i tried to do as the guide but when boot it says Segmentation fault backtrace() 9 address. I’m using e4rat-lite and did build a custom kernel with audit.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s