feed.xml

<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.3.4">Jekyll</generator><link href="https://www.baioc.dev/feed.xml" rel="self" type="application/atom+xml" /><link href="https://www.baioc.dev/" rel="alternate" type="text/html" /><updated>2024-11-17T19:25:20+00:00</updated><id>https://www.baioc.dev/feed.xml</id><title type="html">baioc</title><author><name>Gabriel B. Sant&apos;Anna</name></author><entry><title type="html">How to set up an X-based desktop from scratch</title><link href="https://www.baioc.dev/tutorial/arch-awesomewm/" rel="alternate" type="text/html" title="How to set up an X-based desktop from scratch" /><published>2024-07-27T00:00:00+00:00</published><updated>2024-07-27T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/arch-awesomewm</id><content type="html" xml:base="https://www.baioc.dev/tutorial/arch-awesomewm/"><![CDATA[<p>In the final post of this dual-boot series, I’ll show how to go from a base Arch install:</p>

<figure class=""><a href="/assets/images/dual-boot/arch-base.png" class="image-popup" title="Base Arch tty, after booting for the first time - at least all peripherals (monitor, keyboard, mouse) were working out of the box.
"><img src="/assets/images/dual-boot/arch-base.png" alt="arch-base" /></a><figcaption>
      Base Arch tty, after booting for the first time - at least all peripherals (monitor, keyboard, mouse) were working out of the box.

    </figcaption></figure>

<p>To a simple <a href="https://awesomewm.org/">awesomeWM</a> desktop environment:</p>

<figure class=""><a href="/assets/images/dual-boot/arch-rice.png" class="image-popup" title="This is a screenshot of my personal install. We’ll be going in this direction.
"><img src="/assets/images/dual-boot/arch-rice.png" alt="arch-rice" /></a><figcaption>
      This is a screenshot of my personal install. We’ll be going in this direction.

    </figcaption></figure>

<h2 id="0-manual-btrfs-snapshot">0. Manual btrfs snapshot</h2>

<p>Btrfs’s snapshot capability is one of the reasons I chose this file system, so before anything else I’ll use that to make a backup of the entire root partition.
While I’ll create the snapshot manually this time, later in this tutorial I’m going to show how to set up automatic backups.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mount <span class="nt">-o</span> <span class="nv">subvolid</span><span class="o">=</span>5 /dev/mapper/arch-root /mnt
<span class="nv">$ </span>btrfs subvolume snapshot /mnt/@ /mnt/@snapshot-baseline
<span class="nv">$ </span>umount /mnt
</code></pre></div></div>

<p>That’s all it takes, and you’ll notice it’s also instant thanks to btrfs copy-on-write.</p>

<figure class=""><a href="/assets/images/dual-boot/arch-btrfs-snapshot.png" class="image-popup" title="After creating the snapshot, we can also mount it and check its contents.
"><img src="/assets/images/dual-boot/arch-btrfs-snapshot.png" alt="arch-btrfs-snapshot" /></a><figcaption>
      After creating the snapshot, we can also mount it and check its contents.

    </figcaption></figure>

<h2 id="1-network-and-time">1. Network and time</h2>

<p>During installation, I made sure that the <a href="https://wiki.archlinux.org/title/NetworkManager"><code class="language-plaintext highlighter-rouge">NetworkManager</code></a> systemd service would auto-start.
In case you forgot to do that before, this is a good time to run:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>systemctl <span class="nb">enable </span>NetworkManager
<span class="nv">$ </span>systemctl start NetworkManager
</code></pre></div></div>

<p>I also like to always resolve <code class="language-plaintext highlighter-rouge">localhost</code> to … well … the local host.
So I make sure that my <code class="language-plaintext highlighter-rouge">/etc/hosts</code> file contains the following entries:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>127.0.0.1        localhost
::1              localhost
</code></pre></div></div>

<p>Since I also want my system time to be correct, I’ll also enable the built-in NTP service:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>timedatectl set-ntp <span class="nb">true</span>
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-network.png" class="image-popup"><img src="/assets/images/dual-boot/arch-network.png" alt="arch-network" /></a></figure>

<h2 id="2-firewall">2. Firewall</h2>

<p>The first thing to do after connecting to the internet is setting up protection - a firewall, in this case.
I like <a href="https://wiki.archlinux.org/title/Uncomplicated_Firewall">ufw</a>, which is pretty easy to install and enable by default:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pacman <span class="nt">-S</span> ufw
<span class="nv">$ </span>systemctl <span class="nb">enable </span>ufw
<span class="nv">$ </span>systemctl start ufw
<span class="nv">$ </span>ufw <span class="nb">enable</span>
<span class="nv">$ </span>ufw status verbose
Status: active
Logging: on <span class="o">(</span>low<span class="o">)</span>
Default: deny <span class="o">(</span>incoming<span class="o">)</span>, allow <span class="o">(</span>outgoing<span class="o">)</span>, deny <span class="o">(</span>routed<span class="o">)</span>
New profiles: skip
</code></pre></div></div>

<p>The default configuration is already the best option for most desktop users: deny all incoming traffic (connections initiated in the network) and allow all outgoing traffic.</p>

<p>Personally, I also like to use <a href="https://github.com/evilsocket/opensnitch">OpenSnitch</a> to block all outgoing traffic by default, then handpick exceptions based on program, protocol and target address &amp; port.
I know this is too much of a nuisance for most people, so I’ll leave this out of the tutorial.</p>

<h2 id="3-non-root-user">3. Non-root user</h2>

<p>Even if I’m the only person using this system, it’s more secure to do it from a non-root user by default.
The following commands create a user called <code class="language-plaintext highlighter-rouge">user</code> with their own home directory, sets a password, then installs <code class="language-plaintext highlighter-rouge">zsh</code> and sets that as <code class="language-plaintext highlighter-rouge">user</code>’s default shell.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>useradd <span class="nt">--create-home</span> user
<span class="nv">$ </span>passwd user
<span class="nv">$ </span>pacman <span class="nt">-S</span> zsh
<span class="nv">$ </span>chsh <span class="nt">-s</span> /usr/bin/zsh user
</code></pre></div></div>

<p>Then, in order to be able to perform administrative actions through <code class="language-plaintext highlighter-rouge">sudo</code> or Polkit, I’ll add <code class="language-plaintext highlighter-rouge">user</code> to the <code class="language-plaintext highlighter-rouge">wheel</code> group:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>usermod <span class="nt">-aG</span> wheel user
</code></pre></div></div>

<p>And also update the sudoers file with <code class="language-plaintext highlighter-rouge">EDITOR=nano visudo</code>, making sure to uncomment the line that reads:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>%wheel ALL=(ALL:ALL) ALL
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-security.png" class="image-popup" title="Using a non-privileged user by default is a security control, just like setting up a local firewall.
"><img src="/assets/images/dual-boot/arch-security.png" alt="arch-security" /></a><figcaption>
      Using a non-privileged user by default is a security control, just like setting up a local firewall.

    </figcaption></figure>

<p>You can now exit the current shell and log back in as the new user.
If you changed the default shell to <code class="language-plaintext highlighter-rouge">zsh</code>, you may also be prompted to do some initial configuration.</p>

<h2 id="4-x-initialization">4. X initialization</h2>

<p>We’re going to use X.Org as our window system, with awesome as a window manager.
Furthermore, in order to access the shell from an X(11) session, we’ll also need to install a terminal emulator.
We can get started by installing the necessary packages:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> xorg-server xorg-xinit awesome alacritty
</code></pre></div></div>

<p>I want the Xorg server to start whenever I log in as <code class="language-plaintext highlighter-rouge">user</code> from the default <code class="language-plaintext highlighter-rouge">tty1</code>.
Implementing this is as simple as configuring the <code class="language-plaintext highlighter-rouge">zsh</code> login shell (<code class="language-plaintext highlighter-rouge">~/.zprofile</code>) to run <a href="https://wiki.archlinux.org/title/Xinit"><code class="language-plaintext highlighter-rouge">startx</code></a>:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># /home/user/.zprofile</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-z</span> <span class="s2">"</span><span class="nv">$DISPLAY</span><span class="s2">"</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$XDG_VTNR</span><span class="s2">"</span> <span class="nt">-le</span> 1 <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span><span class="nb">exec </span>startx
<span class="k">fi</span>
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">startx</code> program will then run the <code class="language-plaintext highlighter-rouge">~/.xinitrc</code> script to start up client programs, including our window manager of choice.
We just need to copy the default script to the user’s home folder, remove the last few lines running programs we don’t have, and append <code class="language-plaintext highlighter-rouge">exec awesome</code>:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cp</span> /etc/X11/xinit/xinitrc ~/.xinitrc
<span class="nv">$ </span>nano ~/.xinitrc

<span class="nv">$ </span><span class="nb">cat</span> ~/.xinitrc
<span class="c">#!/bin/sh</span>

<span class="c"># merge in defaults and keymaps</span>

<span class="nv">userresources</span><span class="o">=</span><span class="nv">$HOME</span>/.Xresources
<span class="nv">usermodmap</span><span class="o">=</span><span class="nv">$HOME</span>/.Xmodmap
<span class="nv">sysresources</span><span class="o">=</span>/etc/X11/xinit/.Xresources
<span class="nv">sysmodmap</span><span class="o">=</span>/etc/X11/xinit/.Xmodmap

<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="nv">$sysresources</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span>xrdb <span class="nt">-merge</span> <span class="nv">$sysresources</span>
<span class="k">fi

if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="nv">$sysmodmap</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span>xmodmap <span class="nv">$sysmodmap</span>
<span class="k">fi

if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$userresources</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span>xrdb <span class="nt">-merge</span> <span class="s2">"</span><span class="nv">$userresources</span><span class="s2">"</span>
<span class="k">fi

if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$usermodmap</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span>xmodmap <span class="s2">"</span><span class="nv">$usermodmap</span><span class="s2">"</span>
<span class="k">fi</span>

<span class="c"># start some nice programs</span>

<span class="k">if</span> <span class="o">[</span> <span class="nt">-d</span> /etc/X11/xinit/xinitrc.d <span class="o">]</span> <span class="p">;</span> <span class="k">then
 for </span>f <span class="k">in</span> /etc/X11/xinit/xinitrc.d/?<span class="k">*</span>.sh <span class="p">;</span> <span class="k">do</span>
  <span class="o">[</span> <span class="nt">-x</span> <span class="s2">"</span><span class="nv">$f</span><span class="s2">"</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nb">.</span> <span class="s2">"</span><span class="nv">$f</span><span class="s2">"</span>
 <span class="k">done
 </span><span class="nb">unset </span>f
<span class="k">fi

</span><span class="nb">exec </span>awesome
</code></pre></div></div>

<p>Since I’m the only one who’ll be using this system, I’ll also enable <a href="https://wiki.archlinux.org/title/Getty#Automatic_login_to_virtual_console">auto-login</a> - this means I’ll only have to type in the disk decryption password before getting to my desktop.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/systemd/system/getty@tty1.service.d
<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/systemd/system/getty@tty1.service.d/autologin.conf

<span class="nv">$ </span><span class="nb">cat</span> /etc/systemd/system/getty@tty1.service.d/autologin.conf
<span class="o">[</span>Service]
<span class="nv">Type</span><span class="o">=</span>simple
<span class="nv">ExecStart</span><span class="o">=</span>
<span class="nv">ExecStart</span><span class="o">=</span>-/sbin/agetty <span class="nt">-o</span> <span class="s1">'-p -f -- \\u'</span> <span class="nt">--noclear</span> <span class="nt">--autologin</span> user %I <span class="nv">$TERM</span>
<span class="nv">Environment</span><span class="o">=</span><span class="nv">XDG_SESSION_TYPE</span><span class="o">=</span>x11
</code></pre></div></div>

<p>That’s it!
We can now reboot and drop into a barebones graphical desktop:</p>

<figure class=""><a href="/assets/images/dual-boot/arch-awesome.png" class="image-popup" title="A fresh install of awesomeWM. I have to say, getting to this point was easier than I thought it would be.
"><img src="/assets/images/dual-boot/arch-awesome.png" alt="arch-awesome" /></a><figcaption>
      A fresh install of awesomeWM. I have to say, getting to this point was easier than I thought it would be.

    </figcaption></figure>

<h2 id="5-awesomewm-configuration">5. awesomeWM configuration</h2>

<p>One of the first things you may notice if you’re following this tutorial is that the “open terminal” buttons and shortcuts are not working.
That’s because, by default, awesome is configured to use <code class="language-plaintext highlighter-rouge">xterm</code>, whereas we installed Alacritty.
You can still open Alacritty “manually”, either by typing <code class="language-plaintext highlighter-rouge">alacritty</code> in the command prompt (<code class="language-plaintext highlighter-rouge">Super + R</code>) or choosing it in the built-in application menu (<code class="language-plaintext highlighter-rouge">Super + P</code>).</p>

<p>After getting to a terminal, I suggest overriding the default configuration:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">mkdir</span> <span class="nt">-p</span> ~/.config/awesome/
<span class="nv">$ </span><span class="nb">cp</span> /etc/xdg/awesome/rc.lua ~/.config/awesome/
<span class="nv">$ </span>nano ~/.config/awesome/rc.lua
</code></pre></div></div>

<p>Make sure to change the line that reads <code class="language-plaintext highlighter-rouge">terminal = "xterm"</code> to <code class="language-plaintext highlighter-rouge">terminal = "alacritty"</code>.
In order to apply any changes, we need to reload awesome (<code class="language-plaintext highlighter-rouge">Super + Ctrl + R</code>).
We should now be able to open our terminal of choice with <code class="language-plaintext highlighter-rouge">Super + Enter</code>.</p>

<p>After that, I suggest taking a look at <a href="https://awesomewm.org/recipes/">some awesomeWM configuration recipes</a> in order to understand how to configure it, edit keybindings to your liking or add useful widgets.
Note that, at this point, you can use <code class="language-plaintext highlighter-rouge">pacman</code> to install your web browser of choice and use that to read the documentation directly from your just-configured graphical system.</p>

<p>For example, one of the first things I did was adding a blurry screen lock script:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span>

<span class="nb">set</span> <span class="nt">-e</span>

<span class="nv">screenshot</span><span class="o">=</span><span class="si">$(</span><span class="nb">mktemp</span> /tmp/lock-screen-scrot-XXXXXXXXXX<span class="si">)</span>
scrot <span class="nt">--overwrite</span> <span class="nt">--format</span> png <span class="nt">-F</span> <span class="s2">"</span><span class="nv">$screenshot</span><span class="s2">"</span>
magick <span class="s2">"</span><span class="nv">$screenshot</span><span class="s2">"</span> <span class="nt">-blur</span> 0x8 <span class="s2">"</span><span class="k">${</span><span class="nv">screenshot</span><span class="k">}</span><span class="s2">-blur.png"</span>
i3lock <span class="nt">--image</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">screenshot</span><span class="k">}</span><span class="s2">-blur.png"</span> <span class="nt">--pointer</span><span class="o">=</span>default <span class="nt">--ignore-empty-password</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</code></pre></div></div>

<p>After saving this to a file called <code class="language-plaintext highlighter-rouge">lock-screen</code>, we need to:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> i3lock scrot imagemagick               <span class="c"># install programs used in the script</span>
<span class="nv">$ </span><span class="nb">chmod</span> +x lock-screen                                  <span class="c"># make the script executable</span>
<span class="nv">$ </span><span class="nb">mkdir</span> <span class="nt">-p</span> ~/.local/bin                                 <span class="c"># create a ~/.local/bin/ folder</span>
<span class="nv">$ </span><span class="nb">echo</span> <span class="s1">'export PATH=$PATH:$HOME/.local/bin'</span> <span class="o">&gt;&gt;</span> .zshenv  <span class="c"># append it to our user's PATH</span>
<span class="nv">$ </span><span class="nb">mv </span>lock-screen ~/.local/bin/                          <span class="c"># put the lock screen script there</span>
</code></pre></div></div>

<p>Then, in order to launch it with <code class="language-plaintext highlighter-rouge">Super + L</code>, add the following snippet to the global keys section of <code class="language-plaintext highlighter-rouge">~/.config/awesome/rc.lua</code>.
After restarting awesome, the new shortcut will also show up in the hotkeys help popup (<code class="language-plaintext highlighter-rouge">Super + S</code>), and will blur &amp; lock the screen when activated.</p>

<div class="language-lua highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">awful</span><span class="p">.</span><span class="n">key</span><span class="p">(</span>
    <span class="p">{</span> <span class="n">modkey</span><span class="p">,</span> <span class="p">},</span> <span class="s2">"l"</span><span class="p">,</span>
    <span class="k">function</span><span class="p">()</span> <span class="n">awful</span><span class="p">.</span><span class="n">spawn</span><span class="p">(</span><span class="s2">"lock-screen"</span><span class="p">)</span> <span class="k">end</span><span class="p">,</span>
    <span class="p">{</span> <span class="n">description</span> <span class="o">=</span> <span class="s2">"lock the current session"</span><span class="p">,</span> <span class="n">group</span> <span class="o">=</span> <span class="s2">"launcher"</span> <span class="p">}</span>
<span class="p">),</span>
</code></pre></div></div>

<p>Another thing you might want to do is load a prettier built-in theme (before overriding it to your own tastes) - you can list the contents of <code class="language-plaintext highlighter-rouge">/usr/share/awesome/themes</code> to check what alternatives came preinstalled with awesome.
Then, you can change the <code class="language-plaintext highlighter-rouge">beautiful.init()</code> line in <code class="language-plaintext highlighter-rouge">~/.config/awesome/rc.lua</code>, like so:</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">- beautiful.init(gears.filesystem.get_themes_dir() .. "default/theme.lua")
</span><span class="gi">+ beautiful.init(gears.filesystem.get_themes_dir() .. "xresources/theme.lua")
</span></code></pre></div></div>

<h2 id="6-pipewire-audio">6. Pipewire audio</h2>

<p>A basic installation like the one we just went through mostly just works out of the box.
I didn’t have to do anything other than plug in my HDMI monitor, wireless mouse and USB keyboard.</p>

<p>Soon enough, however, I realized that audio was missing.
Desktop Linux is somewhat infamous for having audio issues, but I think <a href="https://wiki.archlinux.org/title/PipeWire">PipeWire</a> has made that a thing of the past.
All we need to do is install a few packages (and possibly reboot), then it all just works:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> pipewire <span class="se">\</span>
    pipewire-audio pipewire-alsa pipewire-pulse pipewire-jack <span class="se">\</span>
    wireplumber alsa-utils
</code></pre></div></div>

<p>You might want to set up keybindings for the following audio related commands:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>alsamixer <span class="nt">-V</span> all                               <span class="c"># control audio in terminal</span>
<span class="nv">$ </span>pactl set-sink-volume @DEFAULT_SINK@ <span class="nt">-5</span>%       <span class="c"># lower volume by 5%</span>
<span class="nv">$ </span>pactl set-sink-volume @DEFAULT_SINK@ +5%       <span class="c"># increase volume by 5%</span>
<span class="nv">$ </span>pactl set-sink-mute @DEFAULT_SINK@ toggle      <span class="c"># toggle audio mute</span>
<span class="nv">$ </span>pactl set-source-mute @DEFAULT_SOURCE@ toggle  <span class="c"># toggle microphone mute</span>
</code></pre></div></div>

<h2 id="7-pacman-and-snapshots">7. Pacman and snapshots</h2>

<p>As promised, we’ll now set up automatic system backups, making use of btrfs snapshots.
These are SYSTEM backups (of the root partition) and not DATA backups (does not include the home partition), intented to enable undoing a bad update or a breaking config change.
You should configure additional backup schemes for user data.</p>

<p>With that disclaimer out of the way, let’s install <a href="https://wiki.archlinux.org/title/Snapper">snapper</a>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> snapper
<span class="nv">$ </span><span class="nb">sudo </span>snapper <span class="nt">-c</span> root create-config /
</code></pre></div></div>

<p>Snapper saves snapshots under the <code class="language-plaintext highlighter-rouge">/.snapshots</code> mount point; that’s OK.
The issue is that, by default, it creates a subvolume under whatever is currently mounted at <code class="language-plaintext highlighter-rouge">/</code>, but I would rather keep snapshots in the top-level of the btrfs volume structure.
Therefore, I’ll delete the just-created <code class="language-plaintext highlighter-rouge">.snapshots</code> SUBVOLUME and make a new one, mounted at <code class="language-plaintext highlighter-rouge">/.snapshots</code>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>btrfs subvolume delete /.snapshots
<span class="nv">$ </span><span class="nb">sudo </span>mount <span class="nt">-o</span> <span class="nv">subvolid</span><span class="o">=</span>5 /dev/mapper/arch-root /mnt
<span class="nv">$ </span><span class="nb">sudo </span>btrfs subvolume create /mnt/@snapshots
<span class="nv">$ </span><span class="nb">sudo </span>umount /mnt
<span class="nv">$ </span><span class="nb">sudo mkdir</span> /.snapshots
<span class="nv">$ </span><span class="nb">sudo chmod </span>750 /.snapshots
<span class="nv">$ </span><span class="nb">echo</span> <span class="s1">'UUID=${same_UUID_as_other_subvolumes}	/.snapshots	btrfs	subvol=@snapshots,rw,relatime,space_cache=v2,compress=zstd:1	0	0'</span> | <span class="nb">sudo tee</span> <span class="nt">-a</span> /etc/fstab
<span class="nv">$ </span><span class="nb">sudo </span>mount /.snapshots
</code></pre></div></div>

<p>Then, I’ll disable snapper’s default time-based backups:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo sed</span> <span class="nt">-i</span> <span class="s1">'s/TIMELINE_CREATE="no"/TIMELINE_CREATE="yes"/'</span> /etc/snapper/configs/root
<span class="nv">$ </span><span class="nb">sudo </span>systemctl stop snapper-timeline.timer
<span class="nv">$ </span><span class="nb">sudo </span>systemctl disable snapper-timeline.timer
</code></pre></div></div>

<p>In order to create a snapshot manually, we can run <code class="language-plaintext highlighter-rouge">snapper create</code>.
To list existing snapshots, just use <code class="language-plaintext highlighter-rouge">snapper list</code>.
Each snapshot is a btrfs subvolume and can be mounted as such.</p>

<p>Now, in order to run backups every time we make a potentially breaking change, we’ll install <a href="https://barnettphd.com/snap-pac/configuration.html"><code class="language-plaintext highlighter-rouge">snap-pac</code></a>.
By default, it is configured to create a snapshot of the root partition before and after every time we modify the system with <code class="language-plaintext highlighter-rouge">pacman</code> (installing a package, updating the system, etc).</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> snap-pac
</code></pre></div></div>

<p>You can run <code class="language-plaintext highlighter-rouge">snapper list</code> to see the snapshot which was automatically created after installing <code class="language-plaintext highlighter-rouge">snap-pac</code> itself.
Personally, I preferred to <code class="language-plaintext highlighter-rouge">snapper delete</code> this initial one, just to make sure every change has a <code class="language-plaintext highlighter-rouge">pre</code> and <code class="language-plaintext highlighter-rouge">post</code> snapshot.</p>

<figure class=""><a href="/assets/images/dual-boot/arch-snapper.png" class="image-popup" title="Setting up snapper as an automatic system backup tool for Arch.
"><img src="/assets/images/dual-boot/arch-snapper.png" alt="arch-snapper" /></a><figcaption>
      Setting up snapper as an automatic system backup tool for Arch.

    </figcaption></figure>

<p>Although the current setup covers changes to the root partition, we might not be able to fully restore the system without kernel backups as well.
The kernel is not currently being backed up, since it lives in the boot partition.
Thankfully, the <a href="https://wiki.archlinux.org/title/System_backup#Snapshots_and_/boot_partition">Arch Wiki has this covered</a> by teaching us how to set up pacman hooks to copy the kernel from the boot partition to our snapshot-able root:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> rsync
<span class="nv">$ </span><span class="nb">sudo mkdir</span> /.bootbackup
<span class="nv">$ </span><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/pacman.d/hooks

<span class="nv">$ </span>nano /etc/pacman.d/hooks/95-bootbackup_pre.hook
<span class="nv">$ </span><span class="nb">cat</span> /etc/pacman.d/hooks/95-bootbackup_pre.hook
<span class="o">[</span>Trigger]
Operation <span class="o">=</span> Upgrade
Operation <span class="o">=</span> Install
Operation <span class="o">=</span> Remove
Type <span class="o">=</span> Path
Target <span class="o">=</span> usr/lib/modules/<span class="k">*</span>/vmlinuz
<span class="o">[</span>Action]
Depends <span class="o">=</span> rsync
Description <span class="o">=</span> Backing up pre /boot...
When <span class="o">=</span> PreTransaction
Exec <span class="o">=</span> /usr/bin/bash <span class="nt">-c</span> <span class="s1">'rsync -a --mkpath --delete /boot/ "/.bootbackup/$(date +%Y-%m-%d.%Hh%Mm%Ss).pre"/'</span>

<span class="nv">$ </span>nano /etc/pacman.d/hooks/95-bootbackup_post.hook
<span class="nv">$ </span><span class="nb">cat</span> /etc/pacman.d/hooks/95-bootbackup_post.hook
<span class="o">[</span>Trigger]
Operation <span class="o">=</span> Upgrade
Operation <span class="o">=</span> Install
Operation <span class="o">=</span> Remove
Type <span class="o">=</span> Path
Target <span class="o">=</span> usr/lib/modules/<span class="k">*</span>/vmlinuz
<span class="o">[</span>Action]
Depends <span class="o">=</span> rsync
Description <span class="o">=</span> Backing up post /boot...
When <span class="o">=</span> PostTransaction
Exec <span class="o">=</span> /usr/bin/bash <span class="nt">-c</span> <span class="s1">'rsync -a --mkpath --delete /boot/ "/.bootbackup/$(date +%Y-%m-%d.%Hh%Mm%Ss).post"/'</span>
</code></pre></div></div>

<p>We can now run a full system update, with confidence that we can revert it in case things break.
(In fact, it is a good idea to keep you Arch ISO live USB around, since <a href="https://bbs.archlinux.org/viewtopic.php?pid=1508842#p1508842">restoring your system from these snapshots</a> may require manual intervention).</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-Syu</span>
</code></pre></div></div>

<h2 id="8-other-customizations">8. Other customizations</h2>

<p>In the previous Arch tutorial, we went from a live USB to a minimal installation, with not much other than network connectivity.
In this one, we moved from a TTY login to a graphical desktop environment, with auto-login, a terminal emulator, a working audio system, automatic backups and a fully customizable window manager; with a web browser being just a <code class="language-plaintext highlighter-rouge">pacman -S</code> away.</p>

<p>The result might still look and feel too barebones for some people, but after this point each one can customize the system however they like.
That’s the beauty of (Arch) Linux: this system is now yours, do with it whatever you want.
Enjoy!</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 4: From TTY to graphical desktop]]></summary></entry><entry><title type="html">How to set up lvmcache across LUKS-encrypted partitions</title><link href="https://www.baioc.dev/tutorial/ubuntu-lvmcache/" rel="alternate" type="text/html" title="How to set up lvmcache across LUKS-encrypted partitions" /><published>2024-07-14T00:00:00+00:00</published><updated>2024-07-21T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/ubuntu-lvmcache</id><content type="html" xml:base="https://www.baioc.dev/tutorial/ubuntu-lvmcache/"><![CDATA[<p>In this tutorial, I’ll set up an <code class="language-plaintext highlighter-rouge">lvmcache</code> across LUKS-encrypted partitions in a Debian-based system.
This was the solution I chose to increase the performance of hybrid storage - in which part of the SSD acts as a cache to the HDD - in a fully-encrypted setup.</p>

<p>Starting from a fresh <a href="../ubuntu-24.04-encrypted">Ubuntu 24.04 LTS encrypted install</a>, we’ll:</p>
<ul>
  <li>Shrink the root volume in order to free disk space in the primary disk (SSD)</li>
  <li>Create an encrypted partition in the secondary disk (HDD)</li>
  <li>Configure a volume in the SSD as a cache for the HDD volume</li>
  <li>Set up initramfs hooks to mount everything on boot, with a single decryption password</li>
</ul>

<h2 id="0-prepare-operating-environment">0. Prepare operating environment</h2>

<p>While I’m sure some of the operations we’re about to perform could be applied to a live system, I prefer to do these things from a live USB.
I’ll use an Arch Linux ISO for this, since it is lightweight and includes all the tools we need.</p>

<h2 id="1-reduce-root-ssd-volume">1. Reduce root SSD volume</h2>

<p>After flashing the ISO to a USB and booting into it, we start by decrypting the Ubuntu LUKS partition (<code class="language-plaintext highlighter-rouge">/dev/sda3</code> in this case).
Then, we can resize the filesystem and reduce the logical volume in one go (in this case, freeing up 5GiB in the primary LVM physical volume):</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cryptsetup open /dev/sda3 ubuntu-root
<span class="nv">$ </span>lvreduce <span class="nt">-v</span> <span class="nt">--resizefs</span> <span class="nt">-L</span> <span class="nt">-5G</span> /dev/ubuntu-vg/ubuntu-lv
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-lvreduce.png" class="image-popup" title="Shrinking the Ubuntu root logical volume.
"><img src="/assets/images/dual-boot/lvmcache-lvreduce.png" alt="lvmcache-lvreduce" /></a><figcaption>
      Shrinking the Ubuntu root logical volume.

    </figcaption></figure>

<h2 id="2-create-an-encrypted-partition">2. Create an encrypted partition</h2>

<p>Let’s set up the HDD LUKS-encrypted partition, making sure that:</p>
<ul>
  <li>We use the same encryption password as in the root volume (this is optional, but allows us to use the <code class="language-plaintext highlighter-rouge">decrypt_keyctl</code> script to unlock both partitions with a single password prompt)</li>
  <li>The sector size used in the HDD partition matches the one in the SSD (this is mandatory as far as I’m aware, otherwise LVM won’t let us extend the volume group, issuing an error like <code class="language-plaintext highlighter-rouge">Devices have inconsistent logical block sizes</code>)</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-luksFormat.png" class="image-popup" title="Setting up a new LUKS-encrypted partition with the right sector size, as verified with lsblk -t
"><img src="/assets/images/dual-boot/lvmcache-luksFormat.png" alt="lvmcache-luksFormat" /></a><figcaption>
      Setting up a new LUKS-encrypted partition with the right sector size, as verified with lsblk -t

    </figcaption></figure>

<h2 id="3-extend-lvm-volume-group">3. Extend LVM volume group</h2>

<p>Next, we layer an LVM physical volume on top of the new LUKS partition, then add it to the existing LVM volume group.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pvcreate /dev/mapper/ubuntu-data
<span class="nv">$ </span>vgextend ubuntu-vg /dev/mapper/ubuntu-data
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-vgextend.png" class="image-popup" title="Extending an existing LVM volume group with a new physical volume.
"><img src="/assets/images/dual-boot/lvmcache-vgextend.png" alt="lvmcache-vgextend" /></a><figcaption>
      Extending an existing LVM volume group with a new physical volume.

    </figcaption></figure>

<p>NOTE: if you need to close a LUKS+LVM partition for any reason, you many need to stop the volume group with <code class="language-plaintext highlighter-rouge">vgchange -a n ubuntu-vg</code> before <code class="language-plaintext highlighter-rouge">cryptsetup close</code>ing the partition.</p>

<h2 id="4-set-up-the-lvmcache">4. Set up the lvmcache</h2>

<p>In order to create the cache, we’ll need a logical volume in each device:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>lvcreate <span class="nt">-n</span> work-hdd <span class="nt">-l</span> 100%PVS ubuntu-vg /dev/mapper/ubuntu-data
<span class="nv">$ </span>lvcreate <span class="nt">-n</span> work-ssd <span class="nt">-l</span> 100%PVS ubuntu-vg /dev/mapper/ubuntu-root
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-lvcreate.png" class="image-popup" title="Creating a logical volume in each device.
"><img src="/assets/images/dual-boot/lvmcache-lvcreate.png" alt="lvmcache-lvcreate" /></a><figcaption>
      Creating a logical volume in each device.

    </figcaption></figure>

<p>Then, we follow <a href="https://www.man7.org/linux/man-pages/man7/lvmcache.7.html">the documentation</a> in order to create the cache as either a <code class="language-plaintext highlighter-rouge">--cachepool</code> or a <code class="language-plaintext highlighter-rouge">--cachevol</code> (see <a href="https://unix.stackexchange.com/questions/647225/lvmcache-why-use-a-cachepool-instead-of-a-cachevol/767061#767061">this StackOverflow question</a> for more information on the choice):</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>lvconvert <span class="nt">--type</span> cache <span class="nt">--cachepool</span> work-ssd ubuntu-vg/work-hdd
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-lvconvert.png" class="image-popup" title="Converting volumes to an lvmcache. Notice that, like me, you may need to shrink the slower volume to accommodate LVM metadata.
"><img src="/assets/images/dual-boot/lvmcache-lvconvert.png" alt="lvmcache-lvconvert" /></a><figcaption>
      Converting volumes to an lvmcache. Notice that, like me, you may need to shrink the slower volume to accommodate LVM metadata.

    </figcaption></figure>

<p>In order to make use of the cache, just use the <code class="language-plaintext highlighter-rouge">work-hdd</code> logical volume normally.
For example, this would be a good time to add a filesystem to the cached volume:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mkfs.ext4 /dev/ubuntu-vg/work-hdd
</code></pre></div></div>

<h2 id="5-reboot-to-ubuntu">5. Reboot to Ubuntu</h2>

<p>We should now be able to reboot into Ubuntu, albeit with a small hiccup: after entering our decryption password, the boot process may drop into a BusyBox recovery shell.
There’s no need to panic, a first <code class="language-plaintext highlighter-rouge">exit</code> command will tip us that the reason for failure is that something went wrong when mounting the root volume.</p>

<p>In this case, that’s because we never told the system that it needed to decrypt the HDD partition, and LVM fails by default when a volume group is incomplete.
We can run <code class="language-plaintext highlighter-rouge">lvm lvdisplay</code> to check that, while the new logical volume is not available, the old one (containing the system root) still is.
Therefore, we just activate the volume group in partial mode with <code class="language-plaintext highlighter-rouge">lvm vgchange -a y --partial ubuntu-vg</code>, then <code class="language-plaintext highlighter-rouge">exit</code> again to finish the boot process.</p>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-busybox.png" class="image-popup" title="Continuing a failed boot process with a partial LVM volume group.
"><img src="/assets/images/dual-boot/lvmcache-busybox.png" alt="lvmcache-busybox" /></a><figcaption>
      Continuing a failed boot process with a partial LVM volume group.

    </figcaption></figure>

<h2 id="6-setting-up-decryption">6. Setting up decryption</h2>

<p>Once the boot process is done, open a terminal to decrypt the remaining partition - the LVM volumes inside should also get recognized automatically.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>cryptsetup open /dev/sdb1 ubuntu-data
</code></pre></div></div>

<p>We’ll now set up <a href="https://unix.stackexchange.com/questions/392284/using-a-single-passphrase-to-unlock-multiple-encrypted-disks-at-boot/392286#392286">a script</a> which will cache our decryption password in order to open both of our LUKS partitions during the boot process (more specifically, by the initramfs image which is loaded from the boot partition).</p>

<p>We begin by installing the <code class="language-plaintext highlighter-rouge">keyutils</code> package.
Then, edit the <code class="language-plaintext highlighter-rouge">/etc/crypttab</code> file in order to add the <code class="language-plaintext highlighter-rouge">keyscript</code> option and set up the same <code class="language-plaintext highlighter-rouge">CRYPTTAB_KEY</code> for both entries.
We also add the <code class="language-plaintext highlighter-rouge">initramfs</code> options and <a href="https://unix.stackexchange.com/questions/206754/difference-between-cryptopts-and-crypttab/447676#447676">make sure it will be handled there</a> by setting <code class="language-plaintext highlighter-rouge">CRYPTSETUP=y</code> in <code class="language-plaintext highlighter-rouge">/etc/cryptsetup-initramfs/conf-hook</code>.
Finally, re-generate the initramfs with <code class="language-plaintext highlighter-rouge">update-initramfs</code>.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>apt <span class="nb">install </span>keyutils
<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/crypttab
<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/cryptsetup-initramfs/conf-hook
<span class="nv">$ </span><span class="nb">sudo </span>update-initramfs <span class="nt">-k</span> all <span class="nt">-u</span>
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-keyscript.png" class="image-popup" title="Setting up the decrypt_keyctl script after some sanity checks.
"><img src="/assets/images/dual-boot/lvmcache-keyscript.png" alt="lvmcache-keyscript" /></a><figcaption>
      Setting up the decrypt_keyctl script after some sanity checks.

    </figcaption></figure>

<p>After those steps, we can reboot to check that we’ll no longer drop to a BusyBox recovery shell.
Furthermore, from now on, both partitions will be unlocked by a single boot password prompt.</p>

<h2 id="7-mount-the-cached-partition">7. Mount the cached partition</h2>

<p>Finally, we can set up an entry in <code class="language-plaintext highlighter-rouge">/etc/fstab</code> in order to automatically mount our cached partition.
In my case, I’ll create an <code class="language-plaintext highlighter-rouge">/hdd</code> folder that’s owned by my main user, but another possibility would be mounting to <code class="language-plaintext highlighter-rouge">/home</code> directly (make sure to have backups, if you do).</p>

<p>I should note that, in order to use UUIDs to reference an LVM logical volume in fstab, we need to combine the volume group (VG) UUID and the logical volume (LV) UUID.
We can acquire those with <code class="language-plaintext highlighter-rouge">vgdisplay</code> and <code class="language-plaintext highlighter-rouge">lvdisplay</code>, respectively, then remove the hyphens and concatenate them to get the full LVM device mapper UUID path.</p>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-fstab.png" class="image-popup" title="Setting up fstab to auto-mount the cached volume.
"><img src="/assets/images/dual-boot/lvmcache-fstab.png" alt="lvmcache-fstab" /></a><figcaption>
      Setting up fstab to auto-mount the cached volume.

    </figcaption></figure>

<p>In the end, we have most of the system stored in the faster SSD, while our user data can stay in the bigger (but slower) HDD - both devices using LUKS encryption.
The LVM subsystem will then take care of moving frequently accessed HDD data into the SSD cache - automatically improving performance for the data that’s actually being used.
Victory!</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 3: Encrypted hybrid storage]]></summary></entry><entry><title type="html">How to install Arch on Btrfs with LUKS encryption and software RAID1</title><link href="https://www.baioc.dev/tutorial/arch-btrfs-luks-raid/" rel="alternate" type="text/html" title="How to install Arch on Btrfs with LUKS encryption and software RAID1" /><published>2024-05-17T00:00:00+00:00</published><updated>2024-07-27T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/arch-btrfs-luks-raid</id><content type="html" xml:base="https://www.baioc.dev/tutorial/arch-btrfs-luks-raid/"><![CDATA[<p>This tutorial is written for my own future reference, but I’m also posting it online in case it helps someone else looking for a similar setup.
While the <a href="../dual-boot-linux-encrypted-hybrid-storage">first post on this series</a> already laid out my specs, motivations and technological choices, this one will focus on implementing the core parts of my personal subsystem: an encrypted Arch Linux installation on top of Btrfs, with RAID1.</p>

<h2 id="0-overview">0. Overview</h2>

<p>Here’s a sketch of what I’m aiming for:</p>

<p><img src="/assets/images/dual-boot/os-arch.png" alt="sketch-arch" /></p>

<p>During <a href="../ubuntu-24.04-encrypted">the previous part of this series</a>, I’ve used the Ubuntu 24.04 LTS guided installer to create an ESP partition (<code class="language-plaintext highlighter-rouge">sda1</code>, in green) and the core parts of my work subsystem (<code class="language-plaintext highlighter-rouge">sda4</code> and <code class="language-plaintext highlighter-rouge">sda5</code>, in orange).
This post will set up the remaining SSD partitions, as well as the HDD RAID1 mirror.</p>

<p>I’ll follow the general structure of <a href="https://wiki.archlinux.org/title/Installation_guide">the Arch wiki’s installation guide</a>, with a few added comments and highlighting deviations from the standard route for my customizations.</p>

<h2 id="1-pre-installation">1. Pre-installation</h2>

<h3 id="11-acquire-an-installation-image">1.1. Acquire an installation image</h3>

<p>Just download the latest ISO image from the <a href="https://archlinux.org/download/">official Arch download page</a>.</p>

<h3 id="12-verify-image-hashes">1.2. Verify image hashes</h3>

<p>Make sure the ISO’s <code class="language-plaintext highlighter-rouge">sha256sum</code> matches the expected hash (<code class="language-plaintext highlighter-rouge">1b4a04ef8a7350852a13070ee498442b087a607a18840b4dd7d99867eb5f6a4c</code> for 2024.05.01).</p>

<h3 id="13-prepare-installation-medium">1.3. Prepare installation medium</h3>

<p><a href="https://wiki.archlinux.org/title/USB_flash_installation_medium">Flash the image on a USB stick</a>.</p>

<h3 id="14-boot-the-live-environment">1.4. Boot the live environment</h3>

<p>You’ll find yourself logged as root in a Zsh shell.
<a href="https://geo.mirror.pkgbuild.com/iso/latest/arch/pkglist.x86_64.txt">These pacakges</a> come pre-installed.</p>

<figure class=""><a href="/assets/images/dual-boot/arch-tty.png" class="image-popup" title="Arch install live environment, after a few sanity checks.
"><img src="/assets/images/dual-boot/arch-tty.png" alt="arch-tty" /></a><figcaption>
      Arch install live environment, after a few sanity checks.

    </figcaption></figure>

<h3 id="15-set-the-console-keyboard">1.5. Set the console keyboard</h3>

<p>Nothing to do here, since I use a US keyboard layout.</p>

<h3 id="16-verify-the-boot-mode">1.6. Verify the boot mode</h3>

<p><code class="language-plaintext highlighter-rouge">cat /sys/firmware/efi/fw_platform_size</code> should read <code class="language-plaintext highlighter-rouge">64</code>.</p>

<h3 id="17-connect-to-the-internet">1.7. Connect to the internet</h3>

<p>Just use ethernet.
The live system connects automatically using DHCP.
<br />
Make sure your connection is actually working by <code class="language-plaintext highlighter-rouge">ping</code>ing an external server.</p>

<h3 id="18-update-the-system-clock">1.8. Update the system clock</h3>

<p>Check if the output of <code class="language-plaintext highlighter-rouge">timedatectl</code> looks sane (in UTC+0).</p>

<h3 id="19-partition-the-disks">1.9. Partition the disks</h3>

<p>This is where it gets interesting.
I already know how I want to partition my disks, but here are a few highlights copied straight from the Arch wiki:</p>

<blockquote>
  <ul>
    <li>Take time to plan a long-term partitioning scheme to avoid risky and complicated conversion or re-partitioning procedures in the future.</li>
  </ul>
</blockquote>

<p><a href="../dual-boot-linux-encrypted-hybrid-storage">Done</a>.</p>

<blockquote>
  <ul>
    <li>The following partitions are required for a chosen device: one partition for the root directory <code class="language-plaintext highlighter-rouge">/</code>; for booting in UEFI mode: an EFI system partition.</li>
    <li>If the disk from which you want to boot already has an EFI system partition, do not create another one, but use the existing partition instead.</li>
    <li>Swap space can be set on a swap file for file systems supporting it.</li>
  </ul>
</blockquote>

<p>We already have an EFI partition created by the Ubuntu installer, so we’ll keep using that.
Then, instead of a single root partition mounted at <code class="language-plaintext highlighter-rouge">/</code>, we’ll create separate root (<code class="language-plaintext highlighter-rouge">/</code>) and boot (<code class="language-plaintext highlighter-rouge">/boot/</code>) partitions, since we want the former to be encrypted (and the latter can’t be, depending on the <a href="https://en.wikipedia.org/wiki/Bootloader#Second-stage_boot_loader">second-stage boot loader</a> we end up using).
Finally, if swap is needed at all I’ll just use a swap file, since that’s supported on our filesystem of choice, Btrfs.</p>

<blockquote>
  <p>Aligning partitions correctly avoids excessive read-modify-write cycles.
A typical practice for personal computers is to have each partition’s start and size aligned to 1 MiB (1 048 576 bytes) marks.
This covers all common page and block size scenarios, as it is divisible by all commonly used sizes - 1 MiB, 512 KiB, 128 KiB, 4 KiB, and 512 B.
Warning: Misaligned partitions will prevent being able to use 4096 B sectors with LUKS.</p>
</blockquote>

<p>This important disk performance tip was hidden away in the <a href="https://wiki.archlinux.org/title/Advanced_Format#Partition_alignment">Advanced Format page</a>.
Even though most disk partitioning tools default to MiB alignment, I think this should be highlighted in the partitioning step of the installation guide.</p>

<blockquote>
  <p>If you want to create any stacked block devices for LVM, encryption or RAID, do it now.</p>
</blockquote>

<h4 id="191-securely-erasing-a-drive">1.9.1. Securely erasing a drive</h4>

<p>I’m going to err on the side of caution and <a href="https://wiki.archlinux.org/title/Dm-crypt/Drive_preparation#dm-crypt_wipe_on_an_empty_device_or_partition">securely erase my entire HDD</a>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wipefs <span class="nt">-a</span> /dev/sdb
cryptsetup open <span class="nt">--type</span> plain <span class="nt">-d</span> /dev/urandom <span class="nt">--sector-size</span> 4096 <span class="nt">--cipher</span> aes-xts-plain64:sha256 <span class="nt">--key-size</span> 256 /dev/sdb to_be_wiped
<span class="nb">dd </span><span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>/dev/mapper/to_be_wiped <span class="nv">status</span><span class="o">=</span>progress <span class="nv">bs</span><span class="o">=</span>1M
cryptsetup close to_be_wiped
</code></pre></div></div>

<p>Here I’ve manually specified the fastest (according to a <code class="language-plaintext highlighter-rouge">cryptsetup benchmark</code>) combination of cipher and key size, as well as the physical sector size of my HDD (obtained with <code class="language-plaintext highlighter-rouge">lsblk -t</code>).</p>

<h4 id="192-physical-partitioning">1.9.2. Physical partitioning</h4>

<p>This is what our disks looked like after our encrypted Ubuntu 24.04 installation:</p>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-partition-layout.png" class="image-popup" title="Disk layout after the Ubuntu install. sda is an SSD and sdb a bigger HDD.
"><img src="/assets/images/dual-boot/ubuntu-partition-layout.png" alt="ubuntu-partition-layout" /></a><figcaption>
      Disk layout after the Ubuntu install. sda is an SSD and sdb a bigger HDD.

    </figcaption></figure>

<p>Everything on the SSD is already MiB aligned.
You can check that manually by running <code class="language-plaintext highlighter-rouge">fdisk -l</code> and making sure the start sector and total sectors assigned to each partition are divisible by 8.
Then, I’ll just wipe the placeholder ext2 file systems which I used to “hide” available disk space from the Ubuntu installer:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wipefs <span class="nt">-a</span> /dev/sda2
wipefs <span class="nt">-a</span> /dev/sda3
</code></pre></div></div>

<p>As for the just-erased HDD, I’ll use <code class="language-plaintext highlighter-rouge">fdisk</code>’s interactive shell to:</p>
<ul>
  <li>Set up a fresh GPT partition table (<code class="language-plaintext highlighter-rouge">g</code>)</li>
  <li>Create a new partition the exact same size of <code class="language-plaintext highlighter-rouge">/dev/sda3</code> (<code class="language-plaintext highlighter-rouge">n</code>)</li>
  <li>Commit my changes (<code class="language-plaintext highlighter-rouge">w</code>)</li>
</ul>

<h4 id="193-raid-on-hybrid-storage">1.9.3. RAID on hybrid storage</h4>

<p>We’ll now set up software RAID1 using mdadm.
In my particular case, I’ll have a <a href="https://www.tansi.org/hybrid/">hybrid RAID1 setup</a>: one of the mirrors is in an SSD, while the other is in an HDD.
Thankfully, mdadm has a <a href="https://raid.wiki.kernel.org/index.php/Write-mostly">“write-mostly” flag</a> which can mark expected-to-be-slower devices (<code class="language-plaintext highlighter-rouge">/dev/sdb</code>) in the array in order to make most reads come from the faster device (<code class="language-plaintext highlighter-rouge">/dev/sda</code>).
Setting it up is as simple as:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mdadm <span class="nt">--verbose</span> <span class="nt">--create</span> <span class="nt">--metadata</span><span class="o">=</span>0.90 /dev/md0 <span class="nt">--level</span><span class="o">=</span>1 <span class="nt">--raid-devices</span><span class="o">=</span>2 /dev/sda3 <span class="nt">--write-mostly</span> /dev/sdb1
</code></pre></div></div>

<p>Before continuing, <code class="language-plaintext highlighter-rouge">watch /proc/mdstat</code> to make sure the array had enough time to resync.</p>

<figure class=""><a href="/assets/images/dual-boot/arch-mdadm.png" class="image-popup" title="Setting up software RAID on Linux is easier than I imagined.
"><img src="/assets/images/dual-boot/arch-mdadm.png" alt="arch-mdadm" /></a><figcaption>
      Setting up software RAID on Linux is easier than I imagined.

    </figcaption></figure>

<h4 id="194-luks-encryption">1.9.4. LUKS encryption</h4>

<p>Setting up dm-crypt / LUKS encryption on top of a RAID1 array is also quite simple:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cryptsetup <span class="nt">-v</span> luksFormat /dev/md0
cryptsetup open <span class="nt">--allow-discards</span> /dev/md0 arch-root
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-cryptsetup.png" class="image-popup" title="Save for discards, the default LUKS settings in this version of cryptsetup are exactly what I want for this particular system.
"><img src="/assets/images/dual-boot/arch-cryptsetup.png" alt="arch-cryptsetup" /></a><figcaption>
      Save for discards, the default LUKS settings in this version of cryptsetup are exactly what I want for this particular system.

    </figcaption></figure>

<h3 id="110-format-the-partitions">1.10. Format the partitions</h3>

<p>As planned, I’ll use ext4 for my boot partition and Btrfs (on LUKS, on RAID) for the rest:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkfs.ext4 /dev/sda2
mkfs.btrfs /dev/mapper/arch-root
</code></pre></div></div>

<h3 id="111-mount-the-file-systems">1.11. Mount the file systems</h3>

<p>I’ll start by temporarily mounting my Btrfs root at <code class="language-plaintext highlighter-rouge">/mnt</code>.</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount <span class="nt">-o</span> defaults,compress<span class="o">=</span>zstd:1,ssd,discard<span class="o">=</span>async,noatime /dev/mapper/arch-root /mnt
</code></pre></div></div>

<ul>
  <li>Compression is one of the reasons I chose Btrfs, and after looking at <a href="https://www.phoronix.com/review/btrfs-zstd-compress/2">a few</a> <a href="https://www.reddit.com/r/btrfs/comments/bpphbz/my_benchmarks_of_the_new_zstd_levels_in_51/">benchmarks</a>, it seems that <code class="language-plaintext highlighter-rouge">zstd:1</code> is the best compromise on compression ratio and CPU usage.</li>
  <li>Apparently, Btrfs does not add <code class="language-plaintext highlighter-rouge">ssd</code> or <code class="language-plaintext highlighter-rouge">discard</code> options by default when it is mounted on top of LUKS/RAID, so I’ll add those manually.</li>
  <li>Finally, I use <code class="language-plaintext highlighter-rouge">noatime</code> in order to avoid unnecessary writes on the hybrid RAID1 setup, since they’ll have HDD speeds and I want to preserve SSD read performance.</li>
</ul>

<p>Now, since the filesystem is empty, this is probably the most convenient moment to set up other Btrfs subvolumes.
I’ll follow the <a href="https://wiki.archlinux.org/title/snapper#Suggested_filesystem_layout">suggested filesystem layout for Snapper</a>, with separate subvolumes <code class="language-plaintext highlighter-rouge">@</code> and <code class="language-plaintext highlighter-rouge">@home</code> for <code class="language-plaintext highlighter-rouge">/</code> and <code class="language-plaintext highlighter-rouge">/home</code> respectively, then <code class="language-plaintext highlighter-rouge">@var_log</code> to exclude <code class="language-plaintext highlighter-rouge">/var/log</code> from root snapshots (note: I may have to create more subvolumes in the future in order to have separately-snapshottable containers, databases, and the like).</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>btrfs subvolume create /mnt/@
btrfs subvolume create /mnt/@home
btrfs subvolume create /mnt/@var_log
</code></pre></div></div>

<p>Now, in order to actually use the <code class="language-plaintext highlighter-rouge">@</code> subvolume as our <code class="language-plaintext highlighter-rouge">/</code> mountpoint (as opposed to mounting <code class="language-plaintext highlighter-rouge">/</code> directly on the root subvolume a.k.a. <code class="language-plaintext highlighter-rouge">subvolid=5</code>), we’ll unmount the current root:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>umount /mnt
</code></pre></div></div>

<p>And then set up mountpoints (subvolumes and normal partitions alike) in the way they’re actually going to be used in the installed system, in hierarchical order:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount <span class="nt">-o</span> <span class="nv">subvol</span><span class="o">=</span>@,defaults,compress<span class="o">=</span>zstd:1,ssd,discard<span class="o">=</span>async,noatime /dev/mapper/arch-root /mnt/

<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/home
mount <span class="nt">-o</span> <span class="nv">subvol</span><span class="o">=</span>@home,defaults,compress-force<span class="o">=</span>zstd:1,ssd,discard<span class="o">=</span>async,noatime /dev/mapper/arch-root /mnt/home

<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/var/log
mount <span class="nt">-o</span> <span class="nv">subvol</span><span class="o">=</span>@var_log,defaults,compress<span class="o">=</span>zstd:1,ssd,discard<span class="o">=</span>async /dev/mapper/arch-root /mnt/var/log

<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/boot
mount /dev/sda2 /mnt/boot

<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/boot/efi
mount /dev/sda1 /mnt/boot/efi
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-btrfs.png" class="image-popup" title="Setting up mount points for subvolumes and partitions alike.
"><img src="/assets/images/dual-boot/arch-btrfs.png" alt="arch-btrfs" /></a><figcaption>
      Setting up mount points for subvolumes and partitions alike.

    </figcaption></figure>

<h2 id="2-installation">2. Installation</h2>

<h3 id="21-select-the-mirrors">2.1. Select the mirrors</h3>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>reflector <span class="nt">-p</span> https <span class="nt">-l</span> 100 <span class="nt">--sort</span> rate <span class="nt">-n</span> 20 <span class="nt">--save</span> /etc/pacman.d/mirrorlist
</code></pre></div></div>

<h3 id="22-install-essential-packages">2.2. Install essential packages</h3>

<p>This is where we decide what packages we want in our initial install:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pacstrap <span class="nt">-K</span> /mnt <span class="se">\</span>
  base <span class="nb">sudo </span>systemd less which lsof <span class="se">\</span>
  linux-zen linux linux-firmware <span class="se">\</span>
  intel-ucode <span class="se">\</span>
  e2fsprogs dosfstools cryptsetup lvm2 mdadm btrfs-progs <span class="se">\</span>
  networkmanager <span class="nb">bind </span>whois wget curl <span class="se">\</span>
  nano vim <span class="se">\</span>
  man-db man-pages <span class="se">\</span>
  refind efibootmgr mkinitcpio <span class="se">\</span>
  base-devel cmake git python gnupg openssh diffutils <span class="se">\</span>
  bash-completion htop tree xxd dmidecode usbutils perl-image-exiftool sysstat bat inxi
</code></pre></div></div>

<h2 id="3-configure-the-system">3. Configure the system</h2>

<h3 id="30-update-raid-config-file">3.0. Update RAID config file</h3>

<p>Since we’re using RAID, we need to update the default mdadm configuration file:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mdadm <span class="nt">--detail</span> <span class="nt">--scan</span> <span class="o">&gt;&gt;</span> /mnt/etc/mdadm.conf
</code></pre></div></div>

<p>Then open it on a text editor to make sure it looks OK.</p>

<h3 id="31-fstab">3.1. Fstab</h3>

<p>Similarly, for the fstab file:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>genfstab <span class="nt">-U</span> /mnt <span class="o">&gt;&gt;</span> /mnt/etc/fstab
</code></pre></div></div>

<h3 id="32-chroot">3.2. Chroot</h3>

<p>Now, we’ll be chrooting into the new system:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>arch-chroot /mnt
</code></pre></div></div>

<h3 id="33-time">3.3. Time</h3>

<p>If you want the system’s local time to be in your own timezone instead of UTC:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ln</span> <span class="nt">-sf</span> /usr/share/zoneinfo/<span class="nv">$REGION</span>/<span class="nv">$CITY</span> /etc/localtime
hwclock <span class="nt">--systohc</span>
</code></pre></div></div>

<h3 id="34-localization">3.4. Localization</h3>

<p>Edit <code class="language-plaintext highlighter-rouge">/etc/locale.gen</code> and uncomment <code class="language-plaintext highlighter-rouge">en_US.UTF-8 UTF-8</code>, as well as any other needed UTF-8 locales.
Then, generate the locales and set the system locale with:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>locale-gen
<span class="nb">echo</span> <span class="s1">'LANG=en_US.UTF-8'</span> <span class="o">&gt;</span> /etc/locale.conf
</code></pre></div></div>

<h3 id="35-network-configuration">3.5. Network configuration</h3>

<p>Create a hostname file:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s1">'my-cool-hostname'</span> <span class="o">&gt;</span> /etc/hostname
</code></pre></div></div>

<p>And, in my case, I’ll manually enable the NetworkManager systemd unit:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ln</span> <span class="nt">-s</span> /usr/lib/systemd/system/NetworkManager.service /etc/systemd/system/multi-user.target.wants/
</code></pre></div></div>

<h3 id="36-initramfs">3.6. Initramfs</h3>

<p>In this setup, where I’m using LUKS encryption and software RAID, I needed to add <code class="language-plaintext highlighter-rouge">mdadm_udev</code> and <code class="language-plaintext highlighter-rouge">encrypt</code> to my initramfs hooks in <code class="language-plaintext highlighter-rouge">/etc/mkinitcpio.conf</code>:</p>

<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">HOOKS</span>=(<span class="n">base</span> <span class="n">udev</span> <span class="n">autodetect</span> <span class="n">microcode</span> <span class="n">modconf</span> <span class="n">kms</span> <span class="n">keyboard</span> <span class="n">keymap</span> <span class="n">consolefont</span> <span class="n">block</span> <span class="n">mdadm_udev</span> <span class="n">encrypt</span> <span class="n">filesystems</span> <span class="n">fsck</span>)
</code></pre></div></div>

<p>Then, in order to actually apply these changes and recreate the initramfs image:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkinitcpio <span class="nt">-P</span>
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-chroot.png" class="image-popup" title="Note: the mkinitcpio command may issue some warnings while generating the initramfs; I just ignored the ones I got.
"><img src="/assets/images/dual-boot/arch-chroot.png" alt="arch-chroot" /></a><figcaption>
      Note: the mkinitcpio command may issue some warnings while generating the initramfs; I just ignored the ones I got.

    </figcaption></figure>

<h3 id="37-root-password">3.7. Root password</h3>

<p>We can then set the root password with <code class="language-plaintext highlighter-rouge">passwd</code>.</p>

<h3 id="38-boot-loader">3.8. Boot loader</h3>

<p>In my case, I’ll use <a href="https://www.rodsbooks.com/refind/configfile.html">rEFInd</a> (which I already installed in the <code class="language-plaintext highlighter-rouge">pacstrap</code> command):</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>refind-install
<span class="nb">rm</span> /boot/refind_linux.conf
<span class="nb">cp</span> /boot/efi/EFI/refind/refind.conf /boot/efi/EFI/refind/refind.conf.bak
</code></pre></div></div>

<p>After creating a backup of the original config, I edited <code class="language-plaintext highlighter-rouge">/boot/efi/EFI/refind/refind.conf</code> to reduce the boot timeout, to <a href="https://wiki.archlinux.org/title/REFInd#For_kernels_automatically_detected_by_rEFInd">add support for the naming scheme of Arch Linux kernels</a> and to enable the inclusion of a separate <code class="language-plaintext highlighter-rouge">manual.conf</code> config file.</p>

<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># refind.conf
</span>...
<span class="n">timeout</span> <span class="m">5</span>
...
<span class="n">extra_kernel_version_strings</span> <span class="s2">"linux-hardened,linux-rt-lts,linux-zen,linux-lts,linux-rt,linux"</span>
...
<span class="n">include</span> <span class="n">manual</span>.<span class="n">conf</span>
</code></pre></div></div>

<p>Then, I like to set up a manual boot stanza in <code class="language-plaintext highlighter-rouge">/boot/efi/EFI/refind/manual.conf</code>, which reads:</p>

<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># manual.conf
</span><span class="n">menuentry</span> <span class="s2">"Arch Linux"</span> {
	<span class="n">icon</span>    /<span class="n">EFI</span>/<span class="n">refind</span>/<span class="n">icons</span>/<span class="n">os_arch</span>.<span class="n">png</span>
	<span class="n">volume</span>  <span class="s2">"8B2310D6-C36B-4FB9-929B-F2FF5D5B120D"</span>
	<span class="n">loader</span>  /<span class="n">vmlinuz</span>-<span class="n">linux</span>-<span class="n">zen</span>
	<span class="n">initrd</span>  /<span class="n">initramfs</span>-<span class="n">linux</span>-<span class="n">zen</span>.<span class="n">img</span>
	<span class="n">options</span> <span class="s2">"cryptdevice=UUID=6fd4672b-5745-404d-a4f2-5cf3984c0ae5:arch-root:allow-discards root=UUID=07a47b01-b163-4e7f-99ab-4c6310c4e4a1 rootflags=subvol=@ rw"</span>
	<span class="n">submenuentry</span> <span class="s2">"Boot using fallback initramfs"</span> {
		<span class="n">initrd</span> /<span class="n">initramfs</span>-<span class="n">linux</span>-<span class="n">zen</span>-<span class="n">fallback</span>.<span class="n">img</span>
	}
}
</code></pre></div></div>

<ul>
  <li>We need to specify the <code class="language-plaintext highlighter-rouge">icon</code> token before choosing a boot <code class="language-plaintext highlighter-rouge">volume</code>.</li>
  <li>The <code class="language-plaintext highlighter-rouge">volume</code> token is set to the PARTUUID of my <code class="language-plaintext highlighter-rouge">/boot</code> partition (<code class="language-plaintext highlighter-rouge">/dev/sda2</code>).</li>
  <li><code class="language-plaintext highlighter-rouge">loader</code> and <code class="language-plaintext highlighter-rouge">initrd</code> paths are set using the specified <code class="language-plaintext highlighter-rouge">volume</code> as a root, so we don’t need to prefix those with <code class="language-plaintext highlighter-rouge">/boot</code>.</li>
  <li>The <code class="language-plaintext highlighter-rouge">cryptdevice</code> kernel option must be set with the UUID of a block device of <code class="language-plaintext highlighter-rouge">TYPE="crypto_LUKS"</code>, which corresponds to my software RAID block device.</li>
  <li>The UUID in the <code class="language-plaintext highlighter-rouge">root</code> kernel option should match the <code class="language-plaintext highlighter-rouge">/</code> entry in my <code class="language-plaintext highlighter-rouge">fstab</code>.</li>
  <li>Since my root filesystem is mounted in a non-default Btrfs subvolume, I need to specify <code class="language-plaintext highlighter-rouge">rootflags=subvol=@</code> as well.</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/arch-bootloader.png" class="image-popup" title="Configuring the second-stage bootloader has always been the hardest part of installing Linux for me.
"><img src="/assets/images/dual-boot/arch-bootloader.png" alt="arch-bootloader" /></a><figcaption>
      Configuring the second-stage bootloader has always been the hardest part of installing Linux for me.

    </figcaption></figure>

<h2 id="4-reboot">4. Reboot</h2>

<p>If everything was set up correctly, we should be able to reboot into our freshly-installed system:</p>

<ul>
  <li>Leave the chroot environment by exiting the current shell</li>
  <li>(Optionally) unmount partitions with <code class="language-plaintext highlighter-rouge">umount -R /mnt</code></li>
  <li><code class="language-plaintext highlighter-rouge">reboot</code></li>
  <li>Choose the correct option in our boot loader of choice</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/refind.png" class="image-popup" title="rEFInd has already detected my existing Ubuntu install and has correctly set up a corresponding chain-load entry.
"><img src="/assets/images/dual-boot/refind.png" alt="refind" /></a><figcaption>
      rEFInd has already detected my existing Ubuntu install and has correctly set up a corresponding chain-load entry.

    </figcaption></figure>

<p>You’ll then be prompted for your disk encryption password before fully booting:</p>

<figure class=""><a href="/assets/images/dual-boot/arch-boot-encrypted.png" class="image-popup"><img src="/assets/images/dual-boot/arch-boot-encrypted.png" alt="arch-boot-encrypted" /></a></figure>

<h2 id="5-post-installation">5. Post-installation</h2>

<p>All done!
I’ll leave any further post-installation steps to <a href="../arch-awesomewm">another post</a>.</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 2: I use Arch, btw]]></summary></entry><entry><title type="html">How to install Ubuntu 24.04 LTS with Full Disk Encryption</title><link href="https://www.baioc.dev/tutorial/ubuntu-24.04-encrypted/" rel="alternate" type="text/html" title="How to install Ubuntu 24.04 LTS with Full Disk Encryption" /><published>2024-05-16T00:00:00+00:00</published><updated>2024-05-17T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/ubuntu-24.04-encrypted</id><content type="html" xml:base="https://www.baioc.dev/tutorial/ubuntu-24.04-encrypted/"><![CDATA[<p>Ubuntu 24.04 LTS, code name “Noble Numbat”, was <a href="https://wiki.ubuntu.com/Releases">released just 3 weeks ago</a>.
The new installer doesn’t make it obvious, but you can have an encrypted install without erasing / using the entire disk.
Here’s how to do it.</p>

<h2 id="1-acquire-installation-media">1. Acquire installation media</h2>

<ul>
  <li>Download the <a href="https://ubuntu.com/download/desktop">official ISO image</a>;</li>
  <li>Make sure its <code class="language-plaintext highlighter-rouge">sha256sum</code> matches <a href="https://releases.ubuntu.com/24.04/SHA256SUMS">the expected hash</a> (<code class="language-plaintext highlighter-rouge">81fae9cc21e2b1e3a9a4526c7dad3131b668e346c580702235ad4d02645d9455</code> for 24.04);</li>
  <li>Flash the image on a USB stick;</li>
</ul>

<h2 id="2-prepare-the-target-system">2. Prepare the target system</h2>

<ul>
  <li>Back up your data;</li>
  <li>Boot your computer from the live USB;</li>
  <li>Close the installer in order to see the system’s desktop;</li>
  <li>Delete or shrink existing partitions in order to make space for Ubuntu:
    <ul>
      <li>In simple setups, this can be done using GParted or <code class="language-plaintext highlighter-rouge">fdisk</code></li>
      <li>More advanced setups require extra steps (e.g; <a href="https://help.ubuntu.com/community/ResizeEncryptedPartitions">resizing encrypted partitions is tricky</a>)</li>
    </ul>
  </li>
  <li>Consider using a <a href="https://wiki.archlinux.org/title/Dm-crypt/Drive_preparation#Secure_erasure_of_the_drive">secure erasing process</a> in case your drive wasn’t encrypted before;</li>
  <li>If there’s any disk space you DON’T WANT Ubuntu to occupy, just make sure the installer doesn’t see it as free space by creating dummy partitions there;</li>
</ul>

<h3 id="partition-layout">Partition layout</h3>

<p>Based on my VM tests, here’s how the automated installer lays out partitions:</p>
<ul>
  <li>If an ESP partition already exists, it will be used by Ubuntu as well;</li>
  <li>Otherwise, a new one is created in the first sector range of the disk with enough free space for it (default size appears to be 1075 MiB);</li>
  <li>Then, it chooses another disk chunk to set up a boot partition and a root partition:
    <ul>
      <li>I’m not sure whether it uses the first free range which can fit the OS, or the biggest such range that it can find;</li>
      <li>Either way, the two partitions are laid out one after the other; where the first 1792 MiB are used by the boot partition and everything else is given to the root partition.</li>
    </ul>
  </li>
</ul>

<p>In my case, I’ve carved out the first 4 GiB and the last 10 GiB as free space.
The remaining disk space between those was reserved by two partitions which I’ll use to <a href="../arch-btrfs-luks-raid">install Arch afterwards</a>.</p>

<h2 id="3-guided-encrypted-install">3. Guided encrypted install</h2>

<ul>
  <li>Re-open the installer and follow the wizard until you get to the disk setup page:</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-install-simple.png" class="image-popup" title="The new installer doesn’t make it obvious, but you can have an encrypted install without erasing the entire disk.
"><img src="/assets/images/dual-boot/ubuntu-install-simple.png" alt="ubuntu-install-simple" /></a><figcaption>
      The new installer doesn’t make it obvious, but you can have an encrypted install without erasing the entire disk.

    </figcaption></figure>

<ul>
  <li>Toggle the “Erase disk” option, just for now;</li>
  <li>Go to the “Advanced features” menu and select “Use LVM and encryption”;</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-advanced-options.png" class="image-popup" title="This tells the installer to encrypt the root partition using LVM on top of LUKS.
"><img src="/assets/images/dual-boot/ubuntu-advanced-options.png" alt="ubuntu-advanced-options" /></a><figcaption>
      This tells the installer to encrypt the root partition using LVM on top of LUKS.

    </figcaption></figure>

<ul>
  <li>If you don’t want to erase the entire disk, make sure to toggle the “Install Ubuntu alongside other partitions” option before continuing;</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-install-encrypted.png" class="image-popup" title="Can you see the difference between this screenshot and the first one?
"><img src="/assets/images/dual-boot/ubuntu-install-encrypted.png" alt="ubuntu-install-encrypted" /></a><figcaption>
      Can you see the difference between this screenshot and the first one?

    </figcaption></figure>

<ul>
  <li>If encryption was indeed enabled, you’ll be prompted for a password in the next page:</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-passphrase.png" class="image-popup" title="Password prompt for LUKS encryption.
"><img src="/assets/images/dual-boot/ubuntu-passphrase.png" alt="ubuntu-passphrase" /></a><figcaption>
      Password prompt for LUKS encryption.

    </figcaption></figure>

<ul>
  <li>Keep following the install wizard until it asks you to review your choices;</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-install-summary.png" class="image-popup" title="Notice that LUKS disk encryption is enabled.
"><img src="/assets/images/dual-boot/ubuntu-install-summary.png" alt="ubuntu-install-summary" /></a><figcaption>
      Notice that LUKS disk encryption is enabled.

    </figcaption></figure>

<ul>
  <li>After confirming that everything is as expected, let the installer do its thing;</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-partition-layout.png" class="image-popup" title="Disk layout after the installer is done. All three ext2 partitions were created to reserve that disk space for another OS.
"><img src="/assets/images/dual-boot/ubuntu-partition-layout.png" alt="ubuntu-partition-layout" /></a><figcaption>
      Disk layout after the installer is done. All three ext2 partitions were created to reserve that disk space for another OS.

    </figcaption></figure>

<h2 id="4-reboot">4. Reboot</h2>

<ul>
  <li>When rebooting into your fresh Ubuntu installation (and on every boot from now on), you’ll be asked for the encryption password used during installation:</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-boot-encrypted.png" class="image-popup" title="Here’s how it looks like when booting in a VirtualBox VM.
"><img src="/assets/images/dual-boot/ubuntu-boot-encrypted.png" alt="ubuntu-boot-encrypted" /></a><figcaption>
      Here’s how it looks like when booting in a VirtualBox VM.

    </figcaption></figure>

<h2 id="appendix---how-to-fix-installation-failure-on-a-vm-with-low-memory">Appendix - How to fix installation failure on a VM with low memory</h2>

<p>If the installation doesn’t finish successfully (and you’re testing it in a VM):</p>
<ol>
  <li>Take a look at <a href="https://bugs.launchpad.net/ubuntu-desktop-provision/+bug/2059831">this bug report</a></li>
  <li>Increase the amount of memory assigned to the VM
    <ul>
      <li>(I got this error when using 4 GiB, but 8 GiB worked fine)</li>
    </ul>
  </li>
</ol>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 1: Ubuntu guided installation]]></summary></entry><entry><title type="html">How to set up separately-encrypted dual-booting Linux systems on hybrid storage</title><link href="https://www.baioc.dev/tutorial/dual-boot-linux-encrypted-hybrid-storage/" rel="alternate" type="text/html" title="How to set up separately-encrypted dual-booting Linux systems on hybrid storage" /><published>2024-05-05T00:00:00+00:00</published><updated>2024-06-29T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/dual-boot-linux-encrypted-hybrid-storage</id><content type="html" xml:base="https://www.baioc.dev/tutorial/dual-boot-linux-encrypted-hybrid-storage/"><![CDATA[<p>It’s time to rebuild my computer setup.
This series of blog posts will document how I did it, mainly as documentation for my own reference in case I need to re-build anything in the future, but hopefully it helps someone else out there with similar needs.</p>

<h2 id="some-context">Some context</h2>

<p>I’ve been using the same Dell laptop for the past ~7 years and have gone through:</p>
<ul>
  <li>Multiple update-revert cycles on top of the original Windows-only setup</li>
  <li>Resizing Windows partitions in order to dual-boot <a href="https://manjaro.org/">Manjaro Linux</a></li>
  <li>Upgrades in phyiscal memory, a battery swap and a new SSD</li>
  <li>Deleting Manjaro from the HDD while distro-hopping to <a href="https://pop.system76.com/">System76’s Pop!_OS Linux</a></li>
  <li>Corrupting data in the older HDD because of a power loss while moving partitions</li>
  <li>Restoring said data (<em>phew!</em>) from my <a href="https://www.borgbackup.org/">Borg backups</a></li>
  <li>Fully abandoning Windows, yet keeping it installed in the HDD “just in case”</li>
</ul>

<figure class=""><a href="https://imgs.xkcd.com/comics/old_files.png" class="image-popup" title="xkcd #1360
"><img src="https://imgs.xkcd.com/comics/old_files.png" alt="xkcd-1360" /></a><figcaption>
      xkcd #1360

    </figcaption></figure>

<p>Since my new job requires me to use a specific OS, I figured that this would be a good opportunity to start from scratch in order to make better use of the hardware I have.
I’ll keep using the same laptop, of course, since it still works well and is powerful enough for all my needs (including gaming from time to time).</p>

<p>My hardware specs. are:</p>
<ul>
  <li>A Dell motherboard/firmware with support for UEFI boot</li>
  <li>An Intel x86-64 processor with integrated graphics</li>
  <li>A discrete Nvidia GPU with 4 GiB VRAM</li>
  <li>16 GiB DDR3 phyiscal memory</li>
  <li>An old 5400 RPM HDD with ~1 TB capacity</li>
  <li>A newer M.2 2280 SSD with ~480 GB capacity</li>
</ul>

<p>What I need/want from the resulting system is:</p>
<ol>
  <li>A standard <a href="https://ubuntu.com/">Ubuntu</a> 24.04 LTS install (+encryption), for work</li>
  <li>A fully-custom <a href="https://wiki.archlinux.org/">Arch Linux</a> system, for personal use</li>
  <li>Be able to quickly (re)boot and choose an(other) OS for the occasion</li>
  <li>Security for both personal and work-related data</li>
  <li>Make good use of my hardware, especially the SSD and GPU</li>
  <li>Get the most out of my laptop’s battery life, when unplugged</li>
</ol>

<p>Additionally:</p>
<ul>
  <li>My HDD is pretty old at this point (in use for 7+ years!), so I have to assume that it may start failing at any moment.</li>
  <li>On the other hand, I’ve heard some horror stories about SSD sudden death, so I want to be prepared for failures there as well.</li>
  <li>I’m not worried about cutomizing my work OS that much, so a fresh reinstall shouldn’t be much of an issue.</li>
  <li>Furthermore, I expect work-related storage to be mostly build artifacts and caches, which can always be recovered through a clean build from the (version-controlled, cloud-backed) sources.</li>
  <li>I probably don’t need more than 400 GiB for my personal data + OS, at least based on usage in my current setup.</li>
</ul>

<h2 id="a-first-sketch">A first sketch</h2>

<p>After doing some research and a few VM tests, here’s what I want the system to look like, at a high level:
<img src="/assets/images/dual-boot/drives.png" alt="sketch-drives" /></p>

<p>A few things to note:</p>
<ol>
  <li>There’s a single ESP partition on the system.
    <ul>
      <li>According to <a href="https://askubuntu.com/questions/963282/i-have-2-efi-partitions-on-2-separate-drives-how-do-i-combine-them/1437126#1437126">this</a>, the UEFI specification does not require support for multiple ESPs.</li>
      <li>This partiton has to formatted as FAT32.</li>
    </ul>
  </li>
  <li>Each OS gets its own unencrypted <code class="language-plaintext highlighter-rouge">/boot</code> partition.
    <ul>
      <li>They are unencrypted because <a href="https://superuser.com/questions/522971/is-a-boot-partition-always-necessary/522998#522998">most bootloaders don’t support encryption</a>. GRUB is a bit of an exception, but it <a href="https://wiki.archlinux.org/title/GRUB#LUKS2">still doesn’t support default LUKS2 encryption</a>.</li>
      <li>Each OS gets to manage its own boot partition, <a href="https://superuser.com/questions/940782/do-i-need-two-boot-partitions-for-a-dual-boot-system/941881#941881">reducing the chances that they’ll mess up each other’s boot</a>.</li>
      <li>Both boot partitions will be formatted using ext4. This is how Ubuntu’s 24.04 installer does it, and how I chose to do it (as opposed to using ext2) after reading <a href="https://askubuntu.com/questions/958989/refind-very-slow-boot-when-using-efi-stub-loader/962873#962873">this</a>.</li>
    </ul>
  </li>
  <li>Personal stuff, work stuff and backups are all separately encrypted.
    <ul>
      <li>This is for added security: a compromised user/service in the work OS won’t be able to steal my personal data and vice-versa.</li>
    </ul>
  </li>
  <li>Most of the SSD is filled with personal stuff
    <ul>
      <li>This is needed in order for me to fit everything I have in the SSD. I don’t want to lose my files if/when the HDD fails.</li>
      <li>Even if the SSD fails, the <a href="https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1">RAID1</a> mirror ensures that I’ll be able to reboot and continue working on my own personal stuff on a short notice (NOTE: this does NOT substitue external backups).
        <ul>
          <li>If that happens, it should be pretty straightforward to <a href="https://nicolas.busseneau.fr/en/blog/2021/08/remove-mdadm-raid1-keep-data">disable RAID1</a>.</li>
        </ul>
      </li>
      <li>Booting from the HDD mirror in case of an SSD failure will require me to have (a) ESP and <code class="language-plaintext highlighter-rouge">/boot</code> backups in the last HDD partition and (b) a <a href="https://en.wikipedia.org/wiki/Live_CD">live ISO</a> which I can boot from and restore those partitions.</li>
    </ul>
  </li>
  <li>The work installation will be split between SSD and HDD.
    <ul>
      <li>This is because most of the SSD storage is taken by personal stuff.</li>
      <li>If the HDD fails, I’ll lose some build artifacts / caches but will still be able to keep working in the storage-constrained environment of the SSD.</li>
      <li>If the SSD fails, I can just reinstall the OS from scratch in the reserved HDD space.</li>
    </ul>
  </li>
  <li>Lack of any swap partitions.
    <ul>
      <li>On a fully-encrypted system, it seems that <a href="https://wiki.archlinux.org/title/swap#Swap_file">swap files</a> will be easier to manage than swap partitions.</li>
    </ul>
  </li>
</ol>

<h2 id="btrfs-vs-lvm-ext4">Btrfs vs LVM-ext4</h2>

<p><a href="https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup">LUKS</a> appears to be the current standard for disk encryption on Linux <a href="https://wiki.archlinux.org/title/Dm-crypt">dm-crypt</a>.
It works by creating a “virtual” block device over an existing one and transparently translating reads and writes to read+decrypt and encrypt+write operations.</p>

<p>By default, each encrypted device must be unlocked separately, which can be annoying for multi-partition systems.
Therefore, most people recommend setting up a flexible filesystem on top of LUKS in order to have a single unlock for all logical partitions needed by the system.
Ubuntu’s 24.04 installer uses ext4 partitions on top of <a href="https://wiki.archlinux.org/title/LVM">LVM</a> on top of LUKS, whereas my current Pop!_OS system was set up with <a href="https://mutschler.dev/linux/pop-os-btrfs-22-04">Btrfs on top of LVM on top of LUKS</a>.</p>

<figure class=""><a href="https://fedoramagazine.org/wp-content/uploads/2020/12/ext4-on-LVM.jpg" class="image-popup" title="LVM diagram, by Troy Curtis Jr from the Fedora Magazine.
"><img src="https://fedoramagazine.org/wp-content/uploads/2020/12/ext4-on-LVM.jpg" alt="lvm-ext4" /></a><figcaption>
      LVM diagram, by Troy Curtis Jr from the Fedora Magazine.

    </figcaption></figure>

<p>While LVM and <a href="https://wiki.archlinux.org/title/Btrfs">Btrfs</a> are not mutually exclusive, they have similar features:</p>
<ul>
  <li>Volume management: both can distribute storage across multiple devices, but in LVM you have to manually resize logical volumes.</li>
  <li>Snapshots: both can create (sub)volume snapshots for backups, but I think Btrfs’s are faster because of CoW.</li>
  <li>RAID: both provide support for RAID, but <a href="https://btrfs.readthedocs.io/en/latest/Status.html#raid1-raid10">Btrfs’ is less mature</a>.</li>
  <li>Compression: Btrfs has transparent compression and deduplication.</li>
  <li>Caching: Btrfs does not have an equivalent of LVM’s <code class="language-plaintext highlighter-rouge">lvmcache</code> for hybrid storage, unless you count hybrid RAID1 (but that requires equal-sized block devices).</li>
</ul>

<p>While I’ll keep the default LUKS -&gt; LVM -&gt; ext4 used in Ubuntu 24.04, I’ll set up Btrfs on Arch because:</p>
<ol>
  <li>I don’t want to explicitly resize subvolumes.</li>
  <li>Arch is a rolling release distro, so I’ll want fast snapshots on every update.</li>
  <li>I want a RAID1 solution that works under LUKS (such as <a href="https://en.wikipedia.org/wiki/Mdadm"><code class="language-plaintext highlighter-rouge">mdadm</code></a>), not on top of it, so I wouldn’t be using LVM or Btrfs for their RAID features anyway.</li>
  <li>Btrfs’s compression and deduplication should help reduce SSD disk usage.</li>
  <li>I don’t need a hybrid storage cache for the Arch system, since it will fit entirely in the SSD.</li>
</ol>

<p>Since I’ll have my personal stuff on top of Btrfs, I might as well use it in the HDD backup partition.
That will allow me to use <a href="https://wiki.archlinux.org/title/Btrfs#Send/receive">Btrfs’s send/receive operations</a> in order to easily move local snapshots to the HDD, and then later from the HDD to an external backup device.</p>

<p>With that in mind, here’s how I expect my personal OS to look like after booting:</p>

<p><img src="/assets/images/dual-boot/os-arch.png" alt="sketch-arch" /></p>

<h2 id="lvmcache-vs-bcache">lvmcache vs bcache</h2>

<p>I’m going to reserve 100 GiB in the SSD for my work system.
Depending on the development tools I use, I know that’s not going to be enough storage, in which case I’ll need to store data in the (<a href="#appendix-a---ssd-vs-hdd-speeds">5x slower</a>) HDD as well.</p>

<p>Setups like this are pretty common nowadays: just keep your base OS installed in the SSD, while large(r) files go in the slower HDD.
Since frequent access to large files could also benefit from SSD speed ups, some people will go the extra mile and move hot data to the SSD, then place them back on the HDD when they’re no longer being used that often.</p>

<p>The thing is: I don’t want to do that kind of optimization manually, transferring files back and forth between disks.
This is precisely why I’m interested in setting up a <a href="https://en.wikipedia.org/wiki/Hybrid_drive">hybrid storage solution</a>, in which the SSD acts as a cache to the HDD.</p>

<p>It seems that the standard options for doing this on Linux are <a href="https://www.man7.org/linux/man-pages/man7/lvmcache.7.html">lvmcache</a> and <a href="https://www.kernel.org/doc/Documentation/bcache.txt">bcache</a>.
Here’s a few points I gathered while researching those technologies:</p>
<ul>
  <li>lvmcache runs on top of LVM, which I’ll already be using in the Ubuntu install.</li>
  <li>LVM snapshots are not possible on a cached volume; that’s OK for me but could be a deal breaker for others.</li>
  <li>In an encrypted system, bcache should sit between LUKS and the file system, and I don’t see how to do that without deviating from the standard Ubuntu 24.04 encrypted install.</li>
  <li>A <a href="https://www.reddit.com/r/linux/comments/r3qe32/bcache_vs_lvmcache_in_2021_whats_the_general/">few comments on Reddit</a> and a <a href="https://omattos.com/2020/01/02/bcache.html">specific blog post</a> convinced me that bcache had many issues back in 2020-2021; I’m not sure if those were fixed since then.</li>
  <li>On the other hand, I see <a href="https://blog.patshead.com/2022/05/using-lvmcache-for-gaming-and-video-editing-what-i-have-learned-so-far.html">people happily using lvmcache</a> in 2022.</li>
</ul>

<p>In case it wasn’t obvious, I’ll be going with lvmcache for my work OS:</p>

<p><img src="/assets/images/dual-boot/os-ubuntu.png" alt="sketch-ubuntu" /></p>

<h2 id="next-steps">Next steps</h2>

<p>Of course, the next step is actually implementing these designs.</p>

<p>I’ll set up everything in a VM first and document how I did it in the next few blog posts.
Then, if everything works, I’ll apply it to my actual machine.</p>

<p>See you in the <a href="../ubuntu-24.04-encrypted">next part!</a></p>

<h2 id="appendix-a---ssd-vs-hdd-speeds">Appendix A - SSD vs HDD speeds</h2>

<p>I wanted to understand just how faster my M.2 2280 SSD was over my old 5400 RPM HDD, so I ran a few tests using <code class="language-plaintext highlighter-rouge">hdparm</code> and <code class="language-plaintext highlighter-rouge">gnome-disk-utility</code>.</p>

<p>I started by carving out 2 GiB partitions from the free space on each drive:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">/dev/sda2</code> is on the SSD</li>
  <li><code class="language-plaintext highlighter-rouge">/dev/sdb7</code> is on the HDD</li>
</ul>

<p>Running the default GNOME Disks R/W benchmark on the unformatted partitions yields the following results (read rates can be confirmed by a few runs of <code class="language-plaintext highlighter-rouge">hdparm -t</code>):</p>

<table>
  <thead>
    <tr>
      <th>Device</th>
      <th style="text-align: right">Average Read Rate</th>
      <th style="text-align: right">Average Write Rate</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>SSD</td>
      <td style="text-align: right">540.8 MB/s</td>
      <td style="text-align: right">457.8 MB/s</td>
    </tr>
    <tr>
      <td>HDD</td>
      <td style="text-align: right">112.1 MB/s</td>
      <td style="text-align: right">93.2 MB/s</td>
    </tr>
  </tbody>
</table>

<p align="center">
  <img alt="raw-ssd-benchmark" src="/assets/images/dual-boot/raw-ssd.png" width="49%" />
  <img alt="raw-hdd-benchmark" src="/assets/images/dual-boot/raw-hdd.png" width="48.5%" />
</p>

<h2 id="appendix-b---mdadm-vs-btrfs-for-hybrid-ssdhdd-raid1">Appendix B - mdadm vs Btrfs for hybrid (SSD+HDD) RAID1</h2>

<p>While mdadm is the <a href="https://raid.wiki.kernel.org/">default option for software RAID on Linux</a>, I found out that Btrfs supports RAID1 natively.
Therefore, I could choose between the following stacks for my personal OS:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  mdadm RAID1 -&gt; LUKS -&gt; Btrfs  |   LUKS -&gt; Btrfs with RAID1
                                |
SSD: [ mdadm | LUKS | Btrfs ]   |  SSD: [  LUKS  |  Btrfs  ]
         |                      |                     |
         | RAID1 mirroring      |                     | RAID1 mirroring
         v                      |                     v
HHD: [ mdadm | LUKS | Btrfs ]   |  HDD: [  LUKS  |  Btrfs  ]
</code></pre></div></div>

<p>If I wasn’t using disk encryption, I might have gone with the simpler Btrfs-only stack.
However, since Btrfs sits on top of LUKS, RAID1 would only kick in after both partitions were decrypted.</p>

<p>Another issue is performance on a hybrid storage setup: according to <a href="https://superuser.com/questions/874673/btrfs-raid1-ssd-non-ssd/875655#875655">this answer</a>, “the btrfs raid 1 disk access algorithms work by reading from one disk for even-numbered PIDs, and the other for odd-numbered PIDs”.
In other words: disk access speed will suck for half of your processes.
Online discussions pointed me to <a href="https://github.com/torvalds/linux/blob/master/fs/btrfs/volumes.c#L6045">the source code</a>, which does appear to be distributing reads based on PIDs as per a <code class="language-plaintext highlighter-rouge">BTRFS_READ_POLICY_PID</code>.</p>

<p>In conclusion: I’ll forget about Btrfs RAID1 for now.
Maybe in the future, when it provides a better RAID1 access distribution policy (and maybe native encryption, I’ve heard that’s on the roadmap), I’ll consider it again for a setup like mine.</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 0: Motivation and design]]></summary></entry><entry><title type="html">Time is a Lie</title><link href="https://www.baioc.dev/rant/time-is-a-lie/" rel="alternate" type="text/html" title="Time is a Lie" /><published>2023-07-17T00:00:00+00:00</published><updated>2023-07-17T00:00:00+00:00</updated><id>https://www.baioc.dev/rant/time-is-a-lie</id><content type="html" xml:base="https://www.baioc.dev/rant/time-is-a-lie/"><![CDATA[<p>In order to measure time, we need some periodic event to use as a time unit.
Then, if we can order two events one after the other, we can count elapsed time units between them.
That’s enough to measure relative time, but if two or more systems want to share timing information about events which were not observed by all participants, they also need a fixed frame of reference, an epoch.
For example, take two arbitrary events A and B, then count N units between them; what time was it when A happened?
With the information at hand, the best answer you can give is that A happened N units before B.
Then, what time was it when B happened? N units after A, of course.
If I didn’t observe neither A or B, that answer might be completely useless to me.</p>

<p>If you think the previous example is unrealistic, you probably haven’t tried to read Linux <code class="language-plaintext highlighter-rouge">dmesg</code> logs.
By default, <code class="language-plaintext highlighter-rouge">dmesg</code> adds a timestamp to each kernel message, and this timestamp is measured in terms of uptime, i.e., seconds since kernel startup.
If a user doesn’t know when the system was last turned on, it is impossible to answer things like “was this kernel warning issued in the last week?” unless you have also registered the start/end of the week using these timestamps, or you know for sure there was no reboot in the last week (to make things worse, uptime does not tick while the system is sleeping or hibernating, so you would also have to take standby time into account).</p>

<p>We could even use these requirements to synthesize a formal definition of a calendar.
I won’t bother, however, because as you’ll see, dealing with time and human calendars is incredibly complex.
If you’ve ever seen (or maybe written) code like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">days</span> <span class="o">=</span> <span class="n">wholeMonths</span><span class="o">*</span><span class="mi">30</span> <span class="o">+</span> <span class="n">wholeYears</span><span class="o">*</span><span class="mi">365</span>
<span class="n">timeInSeconds</span> <span class="o">=</span> <span class="n">days</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span>
<span class="nf">createAlert</span><span class="p">(</span><span class="nf">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">timeInSeconds</span><span class="p">)</span>
</code></pre></div></div>
<p>Then you probably knew, deep down, that it was imprecise.
But how bad can it get, really?
That calculation assumes:</p>
<ul>
  <li>Every month has 30 days.</li>
  <li>Every year has 365 days.</li>
  <li>Every day has \(24 \times 60 \times 60\) seconds.</li>
</ul>

<p>The truth is that every single one of these assumptions is wrong.</p>

<h2 id="ancient-lunisolar-calendars">Ancient Lunisolar Calendars</h2>

<p>One of the first periodic events we could observe was the cycle of day and night.
So the primitive man decided to use days as a base time unit, and to this days we’re paying for that decision while trying to maintain backwards compatibility to some extent.</p>

<p>Of course, the primitive man wasn’t that good with big numbers, and eventually chose a more coarsely-granular time scale: eras following lunar and/or solar cycles.
Meanwhile, their time anchors were based on political events, such as the crowning of a new ruler.
Using these systems, we can imagine someone referring to a specific day as “the 3rd day of the 200th crescent moon of the 11th Dynasty”.</p>

<p>Unfortunately, these astronomical events which guide our calendars don’t repeat on an integer number of days.
Still, we kept the day as a base time unit and decided that months and years must have a whole number of days.
As a result, we have to make our calendars slightly irregular in order to compensate the frequency difference between astronomical and civil calendars.
These drift-compensation mechanisms are called <a href="https://en.wikipedia.org/wiki/Intercalation_(timekeeping)">intercalary seconds/days/weeks/months</a>.</p>

<p>For example: if the lunar month had a period of exactly 29.5 days, and we alternated between 29 and 30 -day calendar months, these would be in sync with a maximum error of \(\pm 0.5\) days.
Then, in order to compute how long to wait until “this day next month”, we would need to take that into account:</p>
<div class="language-d highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">days</span> <span class="p">=</span> <span class="n">currentMonth</span><span class="p">.</span><span class="n">isEven</span> <span class="p">?</span> <span class="mi">30</span> <span class="p">:</span> <span class="mi">29</span><span class="p">;</span>
<span class="n">timeInSeconds</span> <span class="p">=</span> <span class="n">days</span> <span class="p">*</span> <span class="mi">24</span> <span class="p">*</span> <span class="mi">60</span> <span class="p">*</span> <span class="mi">60</span><span class="p">;</span>
</code></pre></div></div>
<p>But is the moon really that important, to the point where we need to change our calendar in order to keep them in sync?
Probably not, but the sun apparently is.</p>

<h2 id="leap-days-and-the-gregorian-reform">Leap Days and the Gregorian Reform</h2>

<p>We’re all familiar with <a href="Leap_year">leap (or bissextile) years</a>: every four years or so, the year has 366 days instead of the usual 365, and February ends on the 29th instead of the 28th.
This intercalary event happens because we want our civil calendar to have a whole number of days each year, while keeping it in sync with the solar calendar.
Unfortunately, the cosmos doesn’t care about integers nearly as much as we do, and the period of Earth’s revolution around the sun is approximately 365.2422 days (365 full days plus ~5.8128 hours).
This difference would cause the calendar year to drift over time with respect to certain astronomical events, notably seasons.</p>

<p>If we round the period difference between solar and calendar years from ~5.8128 to 6 hours, then adding an extra day to our calendars every four years would be enough to fix this drift precisely, and seasons would be off by at most 18 hours.
Since the period offset is actually smaller than 6 hours, doing this every four years would lead to an overcompensation.
This is precisely how the Julian calendar worked (or, how it didn’t work).</p>

<p>Currently, most of the world uses the Gregorian calendar, which was introduced in 1582 A.D. as a substitute to the Julian Calendar.
Before the Gregorian calendar was established, the Julian calendar had been running since 45 B.C., using excessive leap years.
This led to an offset (w.r.t. the solar year) of approximately 10 days, all of which were adjusted at once: in 1582, the next day after October <strong>4</strong> (a Thursday) was October <strong>15</strong> (a Friday).
After that point, we’ve been skipping leap years (we don’t add the leap day) on years which are multiples of 100 and not multiples of 400.
In the end, our maximum error with respect to seasonal events is approximately 2 days, and an era in the Gregorian calendar (a full period, taking leap days into account) is 400 years long.</p>

<p>The Julian-Gregorian transition doesn’t make time calculations any more complicated if you pretend the Gregorian calendar had always been in effect; in which case you would be using the <em>proleptic</em> Gregorian calendar.
Unfortunately, you might be surprised by date and time libraries trying to be smart about it.
<a href="https://stackoverflow.com/questions/23457470/calendar-to-date-conversion-for-dates-before-15-oct-1582-gregorian-to-julian-ca">Java is a notable example</a>, as converting a <code class="language-plaintext highlighter-rouge">java.util.Date</code> to a string will use a default-parameter <a href="https://docs.oracle.com/javase/8/docs/api/java/util/GregorianCalendar.html">GregorianCalendar</a>.
Despite the name, that implementation actually switches between Gregorian and Julian calendars when going across the transition, so these strings might lead to time calculations which are off by 10 days.</p>

<h2 id="iso-8601-and-leap-seconds">ISO 8601 and Leap Seconds</h2>

<p>Nowadays, we have standard representations / interchange formats for time and dates.
<a href="https://en.wikipedia.org/wiki/ISO_8601">ISO 8601</a> was first introduced in 1988, and it has some nice features:</p>

<ul>
  <li>Fixes the proleptic Gregorian calendar as an international standard for civil dates.</li>
  <li>Lexicographical order of the representation corresponds to chronological order.</li>
  <li>Robust applications can easily parse dates with higher-than-needed precision.</li>
</ul>

<p>Unfortunately, we’re still left with a couple of problems, mostly artifacts from the past.</p>

<p>When the <em>day</em> was used as a base time unit, we needed to compensate calendars based on units bigger than that, and anything smaller was simply defined as a fraction of a day.
So, in the past, the second was defined to be \(\frac{1}{24 \times 60 \times 60} = \frac{1}{86400}\) of a day.
Currently, in the <a href="https://en.wikipedia.org/wiki/International_System_of_Units">S.I.</a>, seconds are the base time unit, where 1 second is defined as “the duration of 9,192,631,770 periods of the radiation corresponding to the transition between the two hyperfine levels of the ground state of the caesium-133 atom”.
If we ignore relativistic frequency shifts, that’s pretty much a constant, and <a href="https://en.wikipedia.org/wiki/International_Atomic_Time">TAI</a> is a time standard which tracks time based on a monotonic clock, ticking precisely every second and centered around the Unix epoch <code class="language-plaintext highlighter-rouge">1970-01-01T00:00:00Z</code>.
Unfortunately, now that our base unit became smaller, solar time and civil time start differing, and calendar days need drift compensation as well.</p>

<p><a href="https://en.wikipedia.org/wiki/Universal_Time">UT1</a> is a time standard which tracks the Earth’s rotation angle, also known as solar time.
However, the Earth spins irregularly, and the length of a day has been slowly increasing for the past centuries (this is apparently caused in part by the moon).
This means that UT1 (solar time) and TAI (atomic time) are drifting apart.
Now, what we actually use to coordinate time worldwide is the UTC standard, which we decided would have seconds elapsing at the same rate as TAI, all while keeping it’s offset to UT1 to less than a second.
In this case, drift compensation is done by adding a <a href="https://en.wikipedia.org/wiki/Leap_second">leap second</a> every now and then.</p>

<p>Unlike leap days, however, the addition of leap seconds to UTC does not follow a regular schedule.
When it does happen, the last minute of the day goes from 00 to 60 (and this is compatible with ISO 8601), for a total of 61 seconds in that minute.
As of 2023, UTC is 37 seconds behind TAI, and the next scheduled leap second will be advertised in <a href="https://hpiers.obspm.fr/iers/bul/bulc/bulletinc.dat">“Bulletin C”</a>.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Remember our initial assumptions?</p>

<ul>
  <li>Every month has 30 days.</li>
  <li>Every year has 365 days.</li>
  <li>Every day has \(24 \times 60 \times 60\) seconds.</li>
</ul>

<p>I’m sure you knew the first one was wrong already, since you’re used to the Gregorian calendar.</p>

<p>The second is also common knowledge, but we often forget about leap days.</p>

<p>The third is false in both UT1 and UTC, since days are getting longer, and every now and then we have a leap second to account for.</p>

<p>Furthermore, we can’t even refer to specific times in the past or future:</p>
<ul>
  <li>In ISO 8601, dates before 1582-10-15 can only be transmitted after mutual agreement of the parts exchanging information; this is done in order to avoid ambiguities and confusion related to the Julian-Gregorian transition.</li>
  <li>The fact that we can’t predict the addition of leap seconds to UTC makes it impossible to reference precise date-times in the future.</li>
</ul>

<p>In reality, the best we could do would probably be to forget about days, months and years altogether; use the Unix epoch as a reference, and atomic clocks to make sure frequency doesn’t drift away from the standard second.
Although this is feasible when we’re talking about computer systems, I’m not sure it will ever catch on.</p>

<p>In the end, UTC is the best we currently have for civil time, even if it is a lie.</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="rant" /><category term="timing" /><summary type="html"><![CDATA[Why your time calculation is wrong, and what do the sun and the moon have to do with that]]></summary></entry><entry><title type="html">On the performance of D sets</title><link href="https://www.baioc.dev/experiment/d-sets-btree/" rel="alternate" type="text/html" title="On the performance of D sets" /><published>2023-07-04T00:00:00+00:00</published><updated>2023-07-11T00:00:00+00:00</updated><id>https://www.baioc.dev/experiment/d-sets-btree</id><content type="html" xml:base="https://www.baioc.dev/experiment/d-sets-btree/"><![CDATA[<p>If I google “dlang sets”, the first result is <a href="https://forum.dlang.org/post/pzrahdwnnphcmdaevaln@forum.dlang.org">this forum post, “How to use sets in D?”</a>.
The main alternatives, brought up in the replies, consist of using <a href="https://dlang.org/phobos/std_container_rbtree.html">std.container.rbtree</a> or making a custom wrapper around the built-in associative arrays (AAs).
I’ll explore and compare the performance of those options, then try to come up with something better and, more importantly, something that works in <a href="https://dlang.org/spec/betterc.html">DasBetterC</a> mode.</p>

<h2 id="experimental-setup">Experimental Setup</h2>

<p>I’ve written four small parameterized benchmarks, one for each relevant operation on a mutable set data type:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">upsert</code>: creates an empty set container and upserts \(n\) random elements (of a given element type), measuring elapsed CPU time of all \(n\) insertions.</li>
  <li><code class="language-plaintext highlighter-rouge">lookupFind</code>: inserts \(n\) random elements, then looks up these same elements in the set (checking that they were found), while measuring only the lookups.</li>
  <li><code class="language-plaintext highlighter-rouge">lookupFail</code>: inserts \(2n\) random elements, removes the first \(n\), then looks them up (checking that they were NOT found).</li>
  <li><code class="language-plaintext highlighter-rouge">remove</code>: inserts \(n\) random elements, then removes them one by one, measuring only removals.</li>
</ul>

<p>In each of these, we measure elapsed CPU time (using <code class="language-plaintext highlighter-rouge">clock</code>) of performing all \(n\) operations.
Then, we repeat that 30 times in order to eliminate some measurement noise.
All elements are randomized once (using <code class="language-plaintext highlighter-rouge">rand</code>), in the very beginning of the test, with a fixed random seed, and the D garbage collector (GC) is executed before each of those repetitions.
Finally, we vary \(n\) across powers of two, \(2^{4} \le n \le 2^{20}\).</p>

<p>Since the payload does not change while repeating each experiment, we estimate performance using <a href="https://arxiv.org/abs/1608.04295">Chen and Revel’s</a> model: if \(measuredTime = realTime + randomDelay\), and \(randomDelay\) is always non-negative (the system can become slower at times due to other processes running, but not randomly faster), then the minimum value we can obtain of \(measureTime\) should be the best estimate of \(realTime\).
Once we have the estimate of how long it takes to perform an operation \(n\) times, we’ll report that time normalized by the number of repetitions, that is, \(estimateTime(n) \div n\).</p>

<p>AAs and <code class="language-plaintext highlighter-rouge">std.container.rbtree</code> both lack support for custom allocators, which makes it really hard to precisely measure their memory usage.
I’m aware of <a href="https://dlang.org/phobos/core_memory.html#.GC">core.memory.GC</a>, but it didn’t work as expected when I tried it and it could never be as precise as a custom allocator anyhow.
Since I’m lazy, I chose to use GNU <code class="language-plaintext highlighter-rouge">time</code>’s report of resident set size (RSS) as a single way to measure memory usage across D, C and C++ programs.</p>

<p>GNU time can be used to report the maximum RSS of a program, and in each experiment that should correspond to the point in <code class="language-plaintext highlighter-rouge">lookupFail</code> after inserting \(2n\) elements.
Then, we could get a (very rough) estimate of memory usage per element by dividing that value by the number of elements in each set.
It should be noted, however, that these \(2n\) elements were generated at random and are being deduplicated by the set, so the <a href="https://en.wikipedia.org/wiki/Birthday_problem">birthday problem</a> applies more and more as \(n\) grows.
Therefore, we also need to record the maximum number of elements stored in the set during a program run.
We also note that, for small \(n\), we expect max RSS to be incredibly imprecise, since most of it is being used by the language runtime.</p>

<p>For the sake of completeness, here’s more information about the system these micro-benchmarks ran on:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CPU: Intel i7-6700HQ (8) @ 3.500GHz
Caches (L1d, L1i, L2, L3): 128 KiB, 128 KiB, 1 MiB, 6 MiB
Memory: 16 GiB
OS: Pop!_OS 22.04 LTS x86_64
Kernel: Linux 6.2.6-76060206-generic
D compiler: ldc2 1.32.2
D flags: -O -release -inline -boundscheck=off
C++ compiler: g++ (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
C++ flags: -std=c++17 -O3
time: time (GNU Time) UNKNOWN
Shell: bash 5.1.16
</code></pre></div></div>

<p>And <a href="/assets/docs/sets-raw.csv">here’s</a> a table with raw experimental results.</p>

<h2 id="aa-as-set-vs-stdcontainerrbtree">AA-as-set vs std.container.rbtree</h2>

<p>If we limit ourselves to the D runtime and standard library, we’ll be comparing hash-tables (AAs) and red-black trees (<code class="language-plaintext highlighter-rouge">std.container.rbtree</code>).
Since our inputs follow a uniform distribution, their hashes ought to be uniformly distributed as well, which is pretty much the ideal situation for hash tables.
Therefore, I expect the AA-based set to have much faster (average \(\mathcal{O}(1)\)) lookups than the tree-based set (\(\Theta(log_2 n)\)), and for lookup time to dominate other operations as well.</p>

<figure class=""><a href="/assets/images/eris/D-lookupFind-int.png" class="image-popup"><img src="/assets/images/eris/D-lookupFind-int.png" alt="D lookupFind - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/D-lookupFail-int.png" class="image-popup"><img src="/assets/images/eris/D-lookupFail-int.png" alt="D lookupFail - int" /></a></figure>

<p>The two graphs above show results for <code class="language-plaintext highlighter-rouge">lookupFind</code> and <code class="language-plaintext highlighter-rouge">lookupFail</code>, respectively, using integer elements.
As predicted, AAs exhibit near-constant time complexity.
Meanwhile, lookups on the tree-based set become logarithmically slower as \(n\) increases – it only looks linear in our graphs because of the logarithmic scale on the horizontal axis.</p>

<p>The graph of our tree-based set looks like two connected lines, with the second having a larger slope.
I’m guessing this happens because, after a certain point, we’re hitting the next level of the memory hierarchy much more frequently.
Why do I think this?
Well, my CPU’s L1d, L2 and L3 sizes are 128 KiB, 1 MiB and 6 MiB, respectively.
Therefore, if we were to completely fill my L2 cache with 4-byte integers, it would fit at most \(262,144\) elements.
Since our data structures are never as memory-efficient as a giant array with contiguous elements, we start hitting L3 a little earlier than \(n = 262,144\), and we see that that’s approximately where the slope changes.</p>

<p>I won’t bother showing graphs for the other operations, since their shapes are pretty much the same.
In fact, I’m only showing both <code class="language-plaintext highlighter-rouge">lookupFind</code> and <code class="language-plaintext highlighter-rouge">lookupFail</code> (and in the same vertical scale) to note that the difference between a successful and a non-successful lookup is much more significant in a tree-based set (find is 1.56x faster than fail at \(n = 2^{20}\)) than in a hash-based set (resp. 1.18x).
Here’s a table summarizing the relative perfomance of these sets at a few input sizes:</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">Operation</th>
      <th style="text-align: right">AA speedup at \(n = 2^{7}\)</th>
      <th style="text-align: right">\(n = 2^{10}\)</th>
      <th style="text-align: right">\(n = 2^{20}\)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">upsert</code></td>
      <td style="text-align: right"><span style="color:orange">0.92x</span></td>
      <td style="text-align: right"><span style="color:green">1.74x</span></td>
      <td style="text-align: right"><span style="color:green"> 3.53x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">lookupFind</code></td>
      <td style="text-align: right"><span style="color:green">1.50x</span></td>
      <td style="text-align: right"><span style="color:green">5.00x</span></td>
      <td style="text-align: right"><span style="color:green">10.17x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">lookupFail</code></td>
      <td style="text-align: right"><span style="color:green">2.50x</span></td>
      <td style="text-align: right"><span style="color:green">3.94x</span></td>
      <td style="text-align: right"><span style="color:green">13.43x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">remove</code></td>
      <td style="text-align: right"><span style="color:green">1.67x</span></td>
      <td style="text-align: right"><span style="color:green">4.60x</span></td>
      <td style="text-align: right"><span style="color:green">11.60x</span></td>
    </tr>
  </tbody>
</table>

<p>It’s pretty clear that using AAs instead of <code class="language-plaintext highlighter-rouge">std.container.rbtree</code> should be preferred whenever you don’t need in-order iteration over set elements, as the performance gains range from pretty small to an order of magnitude (on most operations).
That is, of course, if you can get a uniform distribution of hashes.</p>

<p>For memory results, see the next section.</p>

<h2 id="d-vs-c">D vs C++</h2>

<p>Some people consider D to be a successor to C++, so I’m also interested in comparing the set data structures in each language’s standard libraries.
So, after some messing around with C++’s terrible templates and macros, I’ve ported our micro-benchmark program in order to add <code class="language-plaintext highlighter-rouge">std::set</code> and <code class="language-plaintext highlighter-rouge">std::unordered_set</code> to our comparison.</p>

<p>Before running anything, my expectations were that performance differences would come mainly from the different allocation strategies: garbage collection in D versus manual (de)allocations in C++.
In particular, I expected D sets to have slightly faster removals due to the GC-deferred deallocations.
I would normally expect allocations to be faster in C++ on average, but because the D GC is being executed before each round of operations, it shouldn’t affect upserts that much.
I also expected memory usage to be higher in D, mainly due to its heavier runtime.</p>

<p>Since tree-based and hash-based sets have quite a big performance gap in these benchmarks, we’ll separate these two groups.
With that said, let’s start with the tree-based containers, <code class="language-plaintext highlighter-rouge">std.container.rbtree</code> and <code class="language-plaintext highlighter-rouge">std::set</code> (charts below).
Both have very similar performance (which is to be expected, since they’re both red-black trees), but the D version is usually a bit faster at larger \(n\).
I was a bit surprised by the performance of upserts, which are faster in the C++ version for sets with less than a hundred thousand elements.
This might be related to the performance of memory allocation with the D GC, as opposed to C++’s <code class="language-plaintext highlighter-rouge">new</code>.</p>

<figure class=""><a href="/assets/images/eris/C++-tree-upsert-int.png" class="image-popup"><img src="/assets/images/eris/C++-tree-upsert-int.png" alt="D vs C++ - tree-based set - upsert - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/C++-tree-lookupFind-int.png" class="image-popup"><img src="/assets/images/eris/C++-tree-lookupFind-int.png" alt="D vs C++ - tree-based set - lookupFind - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/C++-tree-remove-int.png" class="image-popup"><img src="/assets/images/eris/C++-tree-remove-int.png" alt="D vs C++ - tree-based set - remove - int" /></a></figure>

<p>Our hash-based set comparison went similarly, with the D version being slightly faster on lookups, but slightly slower on insertions.
There was a bigger difference in removal performance in this group, with D AAs being consistently faster than <code class="language-plaintext highlighter-rouge">std::unordered_set</code>.
Removal speedups range from ~1.5x, up to 4.25x for bigger sets.
Once again, this might be explained by the different deallocation strategies; after all, D memory is only actually being deallocated during GC cycles, which either happen on insertions, or before the next experiment repetition (a manually-triggered full GC cycle which is not measured).</p>

<figure class=""><a href="/assets/images/eris/C++-hash-upsert-int.png" class="image-popup"><img src="/assets/images/eris/C++-hash-upsert-int.png" alt="D vs C++ - hash-based set - upsert - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/C++-hash-lookupFind-int.png" class="image-popup"><img src="/assets/images/eris/C++-hash-lookupFind-int.png" alt="D vs C++ - hash-based set - lookupFind - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/C++-hash-remove-int.png" class="image-popup"><img src="/assets/images/eris/C++-hash-remove-int.png" alt="D vs C++ - hash-based set - remove - int" /></a></figure>

<p>As for memory usage, here’s how our sets compare (graph below, but note the logarithmic scale on the vertical axis this time).
Once again, we’re normalizing our estimate of memory usage by the number of elements in each set.
As expected of our chosen memory metric (maximum process RSS), this leads to a huge overstatement of how much each element actually takes, at least for small \(n\).
Therefore, this measurement is better interpreted as “what is the maximum amount of bytes each element could be taking, assuming nothing else in the entire program contributed to RSS”.
For larger \(n\), however, I expect this estimate to tend towards the real per-element memory usage.</p>

<figure class=""><a href="/assets/images/eris/sets-rss-int.png" class="image-popup"><img src="/assets/images/eris/sets-rss-int.png" alt="set containers - memory usage - max rss per element - int" /></a></figure>

<p>Assuming the measure for our largest \(n\) is the better estimate, we end up with the following:</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">container</th>
      <th style="text-align: right">max RSS per element (bytes)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left">AA-as-set</td>
      <td style="text-align: right">2759 B</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std.container.rbtree</code></td>
      <td style="text-align: right">2371 B</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::set</code></td>
      <td style="text-align: right">51 B</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::unordered_set</code></td>
      <td style="text-align: right">46 B</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">eris.btree</code></td>
      <td style="text-align: right">12 B</td>
    </tr>
  </tbody>
</table>

<p>While final values for the C++ containers seem reasonable, that’s far from true for the D sets.
I expected D’s heavier runtime to inflate memory usage measurements, and we can see that in the left part of the graph as a roughly constant offset between the logarithmic curves of D and C++ containers.
As we increased \(n\), the C++ programs started to converge towards something close to the true memory usage per element, but the D ones kept to a maximum RSS two orders of magnitude higher.
Remember that max RSS stands for the greatest amount of physical memory the process occupied during its entire execution.
Since both AAs and <code class="language-plaintext highlighter-rouge">std.container.rbtree</code> make heavy use of the GC, I assume that is what’s keeping the RSS high; perhaps it keeps freed memory around in order to reuse it later, instead of giving it back to the operating system.</p>

<p>Something you might notice in the last chart is that it includes memory usage for another set container, <code class="language-plaintext highlighter-rouge">eris.btree</code>.
That’s my custom B-tree, which is implemented in D, but uses a custom allocator (based on <code class="language-plaintext highlighter-rouge">malloc</code> and <code class="language-plaintext highlighter-rouge">free</code>) instead of the D GC.
We can see that, despite being implemented in D (and being run with the same heavier runtime), this data structure has similar behaviour to the C++ ones with respect to our measure of memory usage, beating them by approximately 4x at \(n = 2^{20}\).
This also strenghtens the argument that the huge max RSS per element, obtained when benchmarking the D containers, is due to their usage of the GC and probably not even close to true memory usage (if only they supported custom allocators …).</p>

<p>The next section compares my custom <code class="language-plaintext highlighter-rouge">eris.btree</code> container with the other sets we’ve seen so far.</p>

<h2 id="b-tree-container">B-Tree Container</h2>

<p>While D’s standard set containers have roughly equivalent (often slightly better) performance compared to their C++ counterparts, it turns out that C++ standard set containers are not actually that good to begin with.
From what I could gather, these standard containers can’t be the fastest possible data types due to requirements of reference and iterator stability across mutating operations on the containers, as well as a few requirements related to const-correctness.
This means that <code class="language-plaintext highlighter-rouge">std::set</code> (like <code class="language-plaintext highlighter-rouge">std::map</code>) pretty much has to be implemented with a binary tree (a red-black tree, to be more precise), and that <code class="language-plaintext highlighter-rouge">std::unordered_set</code> (like <code class="language-plaintext highlighter-rouge">std::unordered_map</code>) can’t use closed hashing (i.e. their hash table implementation has to use separate chaining).
All of these restrictions lead to worse performance, mainly through less-than-optimal cache usage.</p>

<p>As far as I know, relational databases rely on B+ and B-trees because these data structures provide increased data locality on each node and, due to a fan-out that’s much higher than a binary tree, less pointer chasing when going down the tree.
While these factors are especially relevant for data structures using hard disks as their underlying storage, <a href="https://panthema.net/2007/stx-btree/speedtest/">they also apply to in-memory data strucutres</a>.
Therefore, I figured that a simple B-tree-based set would provide a good contestant against D and C++ standard containers.
Let’s check the results:</p>

<figure class=""><a href="/assets/images/eris/sets-upsert-int.png" class="image-popup"><img src="/assets/images/eris/sets-upsert-int.png" alt="set containers - upsert - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-lookupFind-int.png" class="image-popup"><img src="/assets/images/eris/sets-lookupFind-int.png" alt="set containers - lookupFind - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-lookupFail-int.png" class="image-popup"><img src="/assets/images/eris/sets-lookupFail-int.png" alt="set containers - lookupFail - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-remove-int.png" class="image-popup"><img src="/assets/images/eris/sets-remove-int.png" alt="set containers - remove - int" /></a></figure>

<p>As we can see, for \(2^{14} \le n \le 2^{20}\), the performance of our B-tree is closer to hash-based sets than tree-based ones.
B-tree operations have a theoretical time complexity of \(\Theta(log_m n)\), for some constant \(m \gt 2\); as \(m\) (the order of the B-tree) increases, this becomes “effectively \(\mathcal{O}(1)\)” for any real-world \(n\), which is precisely what we’re observing in these results.
If you prefer relative speedups, here’s how our B-tree compares to other sets at \(n = 2^{20}\):</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">container / operation</th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">upsert</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">lookupFind</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">lookupFail</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">remove</code></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left">AA-as-set</td>
      <td style="text-align: right"><span style="color:orange">1.09x</span></td>
      <td style="text-align: right"><span style="color:red">0.40x</span></td>
      <td style="text-align: right"><span style="color:red">0.32x</span></td>
      <td style="text-align: right"><span style="color:red">0.35x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::set</code></td>
      <td style="text-align: right"><span style="color:green">4.31x</span></td>
      <td style="text-align: right"><span style="color:green">7.27x</span></td>
      <td style="text-align: right"><span style="color:green">4.70x</span></td>
      <td style="text-align: right"><span style="color:green">5.93x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::unordered_set</code></td>
      <td style="text-align: right"><span style="color:green">2.02x</span></td>
      <td style="text-align: right"><span style="color:red">0.58x</span></td>
      <td style="text-align: right"><span style="color:red">0.42x</span></td>
      <td style="text-align: right"><span style="color:green">1.50x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std.container.rbtree</code></td>
      <td style="text-align: right"><span style="color:green">3.86x</span></td>
      <td style="text-align: right"><span style="color:green">4.02x</span></td>
      <td style="text-align: right"><span style="color:green">4.29x</span></td>
      <td style="text-align: right"><span style="color:green">4.10x</span></td>
    </tr>
  </tbody>
</table>

<p>Looking at just the tree-based containers, we can see speedups of approximately 4x (and up to ~7x) across the board by using our custom B-tree-based set.
In most cases, we still fall behind the hash-based containers, albeit by less than the order-of-magnitude margin they had over the red-black trees.
The exception here is <code class="language-plaintext highlighter-rouge">upsert</code>, where the B-tree gets a little faster than the hash tables.
As for memory, we covered that in the previous section already.</p>

<p>A B-tree’s main disadvantage is the fact that mutating operations may need to move multiple elements around in memory.
This invalidates iterators and references to internally-held elements, but may also decrease performance for bigger elements.</p>

<h2 id="bigger-elements">Bigger Elements</h2>

<p>A big part of “modern C++” is copy elision and move semantics.
These features are supposed to interact nicely with RAII, and improve performance by avoiding unnecessary copies.
While D also has some features to avoid copies, there’s definitely much less of a focus in that.
Therefore, it should be interesting to compare our sets with bigger element types.
In our case, we’ll use fixed-size arrays, each with exactly 32 bytes; they’re ordered by <code class="language-plaintext highlighter-rouge">strncmp</code> and hashed by a 32-bit <a href="http://www.isthe.com/chongo/tech/comp/fnv/#FNV-1a">FNV-1a function</a>
(in the previous experiments, we were relying on each language’s built-in ordering and hashing functions for integers).</p>

<p>In these benchmarks, I expected cache locality to play a smaller role, since fewer elements can fit in the cache now – our B-tree probably won’t get the same speedups it did before.
To my surprise, the performance of C++’s <code class="language-plaintext highlighter-rouge">std::unordered_set</code> completely broke down with this bigger element type, to the point where I gave up on measuring its performance for any \(n &gt; 10,000\).
I’m not sure what’s happening, but <code class="language-plaintext highlighter-rouge">std::unordered_set</code> behaves linearly when holding elements with 24+ bytes, and <a href="https://quick-bench.com/q/Akkh2rveMUZ5DSCXuDy-5_HRcik">this is reproducible</a> with clang as well.</p>

<figure class=""><a href="/assets/images/eris/unordered_set-String32.png" class="image-popup"><img src="/assets/images/eris/unordered_set-String32.png" alt="std::unordered_set String32" /></a></figure>

<p>After removing the problematic <code class="language-plaintext highlighter-rouge">std::unordered_set&lt;String32&gt;</code>, our results (disappointingly) match my expectations, with the performance of <code class="language-plaintext highlighter-rouge">eris.btree</code> falling behind the AA implementation, and getting closer to (albeit still faster than) the other tree-based sets.</p>

<figure class=""><a href="/assets/images/eris/sets-upsert-String32.png" class="image-popup"><img src="/assets/images/eris/sets-upsert-String32.png" alt="set containers - upsert - String32" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-lookupFind-String32.png" class="image-popup"><img src="/assets/images/eris/sets-lookupFind-String32.png" alt="set containers - lookupFind - String32" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-lookupFail-String32.png" class="image-popup"><img src="/assets/images/eris/sets-lookupFail-String32.png" alt="set containers - lookupFail - String32" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-remove-String32.png" class="image-popup"><img src="/assets/images/eris/sets-remove-String32.png" alt="set containers - remove - String32" /></a></figure>

<p>Here’s another speedup table for our B-tree at \(n = 2^{20}\):</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">container / operation</th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">upsert</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">lookupFind</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">lookupFail</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">remove</code></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left">AA-as-set</td>
      <td style="text-align: right"><span style="color:red">0.35x</span></td>
      <td style="text-align: right"><span style="color:red">0.20x</span></td>
      <td style="text-align: right"><span style="color:red">0.16x</span></td>
      <td style="text-align: right"><span style="color:red">0.17x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::set</code></td>
      <td style="text-align: right"><span style="color:green">1.73x</span></td>
      <td style="text-align: right"><span style="color:green">1.77x</span></td>
      <td style="text-align: right"><span style="color:green">1.51x</span></td>
      <td style="text-align: right"><span style="color:green">1.59x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std.container.rbtree</code></td>
      <td style="text-align: right"><span style="color:green">1.37x</span></td>
      <td style="text-align: right"><span style="color:orange">0.98x</span></td>
      <td style="text-align: right"><span style="color:green">1.57x</span></td>
      <td style="text-align: right"><span style="color:green">1.22x</span></td>
    </tr>
  </tbody>
</table>

<h2 id="checking-assumptions">Checking Assumptions</h2>

<p>As I explained previously, I’m estimating time by picking the minimum measurement across 30 repetitions of each experiment.
This might be surprising for some of you, who would have expected a report of average timing instead.
I did some research on benchmarking techniques, and professional teams seem to prefer percentiles when measuring application performance.
This is because most benchmark results don’t actually follow normal distributions, where mean and standard deviation would do the job.</p>

<p>Since we already have the data for that, we can perform a simple normality test to check whether we should have used the mean instead.
For each of our parameterized experiments, we collect minimum, maximum, mean and the standard deviation of our 30 measurements.
With these, we can compute the z-score of our extrema, and compare that to the 68-95-99.7 (a.k.a the \(1\sigma-2\sigma-3\sigma\)) rule.</p>

<figure class=""><a href="/assets/images/eris/sets-normality-int.png" class="image-popup"><img src="/assets/images/eris/sets-normality-int.png" alt="set containers - normality test - int" /></a></figure>

<p>As we can see, there are many \(3\sigma\) (and even \(4\sigma\)) events in our maximum measurements.
These imply that modeling our performance with a normal distribution would lead us to underestimate how much deviation there really is in our measurements.
At that point, average elapsed time is not as meaningful, and, since I’m lazy, I didn’t want to implement a more advanced statistics collector.</p>

<h2 id="whats-next">What’s Next</h2>

<p>For now, I’m satisfied with the performance my custom B-tree set was able to achieve.
My next step is probably going to be improving its API before making it open-source.</p>

<p>As for further benchmarks, we could try exploring other payloads.
Due to the uniform distribution of our inputs, hash sets have the upper hand.
It would be interesting to test how well that performance stands up to less-than-ideal hash distributions.</p>

<p>At some point, I’ll also try to make a faster hash table in D (and make it <code class="language-plaintext highlighter-rouge">-betterC</code> compatible, of course).
When I do, I’ll add it to these benchmarks as well, and maybe the next time I’ll be using the P50 percentile (the median), since now I have an efficient ordered set data structure which I can use to compute percentiles: <code class="language-plaintext highlighter-rouge">eris.btree</code>.</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="experiment" /><category term="algorithms" /><summary type="html"><![CDATA[Better than C++, but we can still do betterC]]></summary></entry><entry><title type="html">Dlang Woes</title><link href="https://www.baioc.dev/rant/d-woes/" rel="alternate" type="text/html" title="Dlang Woes" /><published>2023-04-11T00:00:00+00:00</published><updated>2023-04-11T00:00:00+00:00</updated><id>https://www.baioc.dev/rant/d-woes</id><content type="html" xml:base="https://www.baioc.dev/rant/d-woes/"><![CDATA[<blockquote>
  <p>woe (noun)</p>
  <ul>
    <li>great sorrow or distress (often used hyperbolically).</li>
  </ul>
</blockquote>

<p>There’s a quote by Bjarne Stroustrup saying that “<em>there are only two kinds of languages: the ones people complain about and the ones nobody uses</em>”.
Complaining is therefore a natural result of using a language, and I’ve been using <a href="https://dlang.org/">D</a> a bit more in my latest side projects.</p>

<p>While there’s a lot of great things to be said about this <a href="https://dlang.org/comparison.html">language</a> and its <a href="https://github.com/Pure-D/code-d">tooling</a> (especially when compared to C, C++ and other contenders), there are also some warts here and there.
So, here’s some complaining, either pointing out bugs or simply violations of the principle of least astonishment (POLA) in the language (in my opinion, that is).</p>

<h3 id="struct-this-vs-class-this">struct this vs class this</h3>

<p>Q: Can you pass <code class="language-plaintext highlighter-rouge">this</code> by <code class="language-plaintext highlighter-rouge">ref</code>?
<br />
A1: Only inside structs, not classes.
<br />
A2: Actually, you can assign <code class="language-plaintext highlighter-rouge">this</code> to a variable, then pass that variable by ref.
<br />
A3: Consider passing by <a href="https://dlang.org/spec/template.html#auto-ref-parameters">auto ref</a> instead, or maybe <a href="https://dlang.org/spec/function.html#in-params">in</a> if the argument is also <code class="language-plaintext highlighter-rouge">const</code>.</p>

<div class="language-d highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// struct.this vs class.this</span>
<span class="c1">// or, pass this by ref</span>

<span class="kt">int</span> <span class="n">bar</span><span class="p">(</span><span class="n">T</span><span class="p">)(</span><span class="k">ref</span> <span class="n">T</span> <span class="n">x</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="mi">1</span><span class="p">;</span> <span class="p">}</span>

<span class="k">class</span> <span class="n">C</span> <span class="p">{</span>
    <span class="kt">int</span> <span class="n">foo</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="n">bar</span><span class="p">;</span> <span class="p">}</span> <span class="c1">// &lt;--- error</span>
<span class="p">}</span>

<span class="k">struct</span> <span class="n">S</span> <span class="p">{</span>
    <span class="kt">int</span> <span class="n">foo</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="n">bar</span><span class="p">;</span> <span class="p">}</span> <span class="c1">// &lt;--- but this works?</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
    <span class="k">auto</span> <span class="n">c</span> <span class="p">=</span> <span class="k">new</span> <span class="n">C</span><span class="p">();</span>
    <span class="n">c</span><span class="p">.</span><span class="n">bar</span><span class="p">;</span> <span class="c1">// &lt;--- also works</span>
<span class="p">}</span>

<span class="c1">// Error: none of the overloads of template `onlineapp.bar` are callable using argument types `!()(C)`</span>
<span class="c1">// Candidate is: `bar(T)(ref T x)`</span>
</code></pre></div></div>

<h3 id="ufcs-vs-method-overloading">UFCS vs Method Overloading</h3>

<p>Q: Does <strong>Universal</strong> Function Call Syntax (UFCS) turn <code class="language-plaintext highlighter-rouge">variable.foo(arguments)</code> into <code class="language-plaintext highlighter-rouge">foo(variable, arguments)</code> ?
<br />
A1: <a href="https://ddili.org/ders/d.en/ufcs.html">Yeah! Cool, huh?</a>
<br />
A2: Actually, UFCS only applies to functions in module scope (lambdas or local ones don’t count).
<br />
A3: They also don’t work in the presence of methods called <code class="language-plaintext highlighter-rouge">foo</code> (never mind the full typed signature).</p>

<div class="language-d highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// UFCS vs Method Overloading</span>

<span class="kt">int</span> <span class="n">foo</span><span class="p">(</span><span class="n">T</span><span class="p">)(</span><span class="k">ref</span> <span class="n">T</span> <span class="n">x</span><span class="p">,</span> <span class="kt">float</span> <span class="n">f</span><span class="p">){</span> <span class="k">return</span> <span class="mi">1</span><span class="p">;</span> <span class="p">}</span>

<span class="k">struct</span> <span class="n">A</span> <span class="p">{}</span>

<span class="k">struct</span> <span class="n">B</span> <span class="p">{</span>
    <span class="kt">int</span> <span class="n">foo</span><span class="p">(</span><span class="kt">bool</span> <span class="n">b</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="mi">2</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
    <span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">stdio</span> <span class="p">:</span> <span class="n">writeln</span><span class="p">;</span>

    <span class="n">A</span> <span class="n">a</span><span class="p">;</span>
    <span class="n">writeln</span><span class="p">(</span><span class="s">"A * bool  -&gt; "</span><span class="p">,</span> <span class="n">a</span><span class="p">.</span><span class="n">foo</span><span class="p">(</span><span class="kc">true</span><span class="p">));</span>
    <span class="n">writeln</span><span class="p">(</span><span class="s">"A * float -&gt; "</span><span class="p">,</span> <span class="n">a</span><span class="p">.</span><span class="n">foo</span><span class="p">(</span><span class="mf">1.23</span><span class="p">));</span> <span class="c1">// &lt;--- ok, uses UFCS</span>

    <span class="n">B</span> <span class="n">b</span><span class="p">;</span>
    <span class="n">writeln</span><span class="p">(</span><span class="s">"B * bool -&gt; "</span><span class="p">,</span> <span class="n">b</span><span class="p">.</span><span class="n">foo</span><span class="p">(</span><span class="kc">true</span><span class="p">));</span>
    <span class="n">writeln</span><span class="p">(</span><span class="s">"B * float -&gt; "</span><span class="p">,</span> <span class="n">b</span><span class="p">.</span><span class="n">foo</span><span class="p">(</span><span class="mf">1.23</span><span class="p">));</span> <span class="c1">// &lt;--- error (ignores UFCS?)</span>
    <span class="n">writeln</span><span class="p">(</span><span class="s">"B * float (without UFCS) -&gt; "</span><span class="p">,</span> <span class="n">foo</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="mf">1.23</span><span class="p">));</span> <span class="c1">// &lt;--- ok</span>
<span class="p">}</span>

<span class="c1">// Error: function `B.foo(bool b)` is not callable using argument types `(double)`</span>
<span class="c1">// cannot pass argument `1.23` of type `double` to parameter `bool b`</span>
</code></pre></div></div>

<h3 id="taggedpointer-vs-safe">taggedPointer vs @safe</h3>

<p>Q: Are <a href="https://dlang.org/library/std/bitmanip/tagged_pointer.html">std.bitmanip.taggedPointer</a>s safe?
<br />
A1: Yes they’re safe: they only use pointer bits which are known (at compile time!) to be zero due to alignment constraints.
<br />
A2: But they’re not <code class="language-plaintext highlighter-rouge">scope @safe</code> though. <a href="https://issues.dlang.org/show_bug.cgi?id=23095">Here’s the bug report</a>.
<br />
A3: Also, they don’t work in <a href="https://tour.dlang.org/tour/en/gems/compile-time-function-evaluation-ctfe">CTFE</a> because pointer casts are not allowed there.</p>

<div class="language-d highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// taggedPointer accessors are not scope</span>
<span class="c1">// Compile with: -dip1000 -main</span>
<span class="c1">// Bugzilla #3095, reported in 2022-05-05</span>

<span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">bitmanip</span> <span class="p">:</span> <span class="n">taggedPointer</span><span class="p">;</span>

<span class="k">struct</span> <span class="n">S</span> <span class="p">{</span>
    <span class="k">mixin</span><span class="p">(</span><span class="n">taggedPointer</span><span class="p">!(</span>
        <span class="kt">int</span><span class="p">*,</span> <span class="s">"ptr"</span><span class="p">,</span>
        <span class="kt">bool</span><span class="p">,</span> <span class="s">"flag"</span><span class="p">,</span> <span class="mi">1</span>
    <span class="p">));</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="n">foo</span><span class="p">(</span><span class="k">scope</span> <span class="k">ref</span> <span class="n">S</span> <span class="n">s</span><span class="p">)</span> <span class="nd">@safe</span> <span class="p">{</span>
    <span class="n">s</span><span class="p">.</span><span class="n">flag</span><span class="p">;</span> <span class="c1">// &lt;--- error</span>
<span class="p">}</span>

<span class="c1">// Error: scope variable `s` assigned to non-scope parameter `this` calling `flag`</span>
</code></pre></div></div>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="rant" /><category term="programming-languages" /><summary type="html"><![CDATA[There are programming languages that people complain about and languages that nobody uses; and I'm starting to use D]]></summary></entry><entry><title type="html">Fibonacci numbers: from zero to hero</title><link href="https://www.baioc.dev/tutorial/fibonacci/" rel="alternate" type="text/html" title="Fibonacci numbers: from zero to hero" /><published>2020-01-15T00:00:00+00:00</published><updated>2020-01-15T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/fibonacci</id><content type="html" xml:base="https://www.baioc.dev/tutorial/fibonacci/"><![CDATA[<p>Fibonacci numbers appear unexpectedly often in Mathematics and in Computer Science.
They are related to the <a href="https://en.wikipedia.org/wiki/Golden_ratio">golden ratio</a> and just like the magical number, have a very extensive <a href="https://en.wikipedia.org/wiki/Fibonacci_number">Wikipedia entry</a>.
The Fibonacci sequence is defined by a <a href="https://en.wikipedia.org/wiki/Recurrence_relation">recurrence relation</a> and is used as a canonical example of recursion in introductory programming and/or discrete math courses for computer science.
This means that pretty much every programmer has implemented an algorithm to find Fibonacci numbers.
Here we explore some possible approaches.</p>

<p class="notice--info">For a TLDR, jump to the bottom of the page.</p>

<h2 id="a-golden-sequence">A golden sequence</h2>

<p>The Fibonacci numbers can be defined as</p>

<p>\(F_0 = 0, F_1 = 1\) and</p>

<p>\(F_n = F_{n-1} + F_{n-2}\) for \(n &gt; 1\)</p>

<p>Most programmers think of a function when reading the previous definition, where the subscript number is its single argument.
Others see it as some data structure being indexed; this idea will be useful later.
The naive approach then translates math directly into some programming language, for instance C.</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span> <span class="nf">fib</span><span class="p">(</span><span class="kt">int</span> <span class="n">n</span><span class="p">)</span>
<span class="p">{</span>
	<span class="k">if</span> <span class="p">(</span><span class="n">n</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
	<span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">n</span> <span class="o">==</span> <span class="mi">1</span><span class="p">)</span> <span class="k">return</span> <span class="mi">1</span><span class="p">;</span>
	<span class="k">else</span> <span class="k">return</span> <span class="n">fib</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="n">fib</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">2</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>That definition has some obvious practical issues.
For any negative input, the procedure launches into an infinite loop and eventually crashes the program.
Secondly, it is extremely inefficient: on my machine it takes around 15 seconds to compute and print <code class="language-plaintext highlighter-rouge">fib(47)</code> (gcc 9.2.0 with <code class="language-plaintext highlighter-rouge">-O3</code>).
At last, we will quickly run into <a href="https://wiki.sei.cmu.edu/confluence/display/c/INT32-C.+Ensure+that+operations+on+signed+integers+do+not+result+in+overflow">undefined behaviour as we overflow</a> through the limits of our signed integer representation.
In fact, with 32-bit signed ints we can only correctly calculate the sequence for values of <code class="language-plaintext highlighter-rouge">n</code> from zero to 46.
That limit goes up to 93 when using longer 64-bit numbers and making them <code class="language-plaintext highlighter-rouge">unsigned</code> allows us to find a single extra number of the sequence.</p>

<p>There are two easy ways to remove the danger of an infinite loop: we can assert against negative values and crash the program (or throw an exception) intentionally or we can change the conditional to avoid the loop and return a technically incorrect result.
Another option is to extend the sequence for negative integers, this shall be done latter on.
We also solve the numeric representation issue by using a programming language which supports arbitrary-precision arithmetic (aka <a href="https://en.wikipedia.org/wiki/Arbitrary-precision_arithmetic">bignums</a>) by default, so let’s <code class="language-plaintext highlighter-rouge">goto</code> Python.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">fib</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
    <span class="k">if</span> <span class="n">n</span> <span class="o">&lt;=</span> <span class="mi">1</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">n</span><span class="p">;</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="k">return</span> <span class="nf">fib</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="nf">fib</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">2</span><span class="p">)</span>
</code></pre></div></div>

<p>While we could theoretically compute bigger Fibonacci numbers now, it would take ages to do so using this implementation.
While Python made the performance worse by a factor of ~50x in relation to C, the problem is the underlying algorithm being used.
By substituting each call to <code class="language-plaintext highlighter-rouge">fib</code> by the subsequent multiple-recursive calls it generates, we find out that there is lots of redundant computation being done.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>                         <span class="nf">fib</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
             <span class="nf">fib</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>        <span class="o">+</span>        <span class="nf">fib</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
     <span class="nf">fib</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>     <span class="o">+</span>  <span class="nf">fib</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>       <span class="nf">fib</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="nf">fib</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="nf">fib</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="nf">fib</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</code></pre></div></div>

<p>Notice that an invocation of <code class="language-plaintext highlighter-rouge">fib(n)</code> reaches the base cases (<code class="language-plaintext highlighter-rouge">fib(0)</code> and <code class="language-plaintext highlighter-rouge">fib(1)</code>) \(F_{n+1}\) times.
In fact, let \(C_n\) be the number of calls executed when computing <code class="language-plaintext highlighter-rouge">fib(n)</code>, then for all \(n &gt; 1\)</p>

\[C_n = 1 + C_{n-1} + C_{n-2}\]

<p>Meanwhile, for the base cases we have</p>

\[C_0 + 1 = 2 = 2 F_1\\
C_1 + 1 = 2 = 2 F_2\]

<p>So we can try to conjecture that</p>

\[C_n + 1 = 2 F_{n+1}\]

<p>Supposing that holds for \(C_{n-1}\) and \(C_{n-2}\) the proof goes as follows:</p>

\[C_n = 1 + C_{n-1} + C_{n-2}\\
C_n + 1 = (C_{n-1} + 1) + (C_{n-2} + 1)\\
C_n + 1 = 2 F_n + 2 F_{n-1}\\
C_n + 1 = 2 (F_n + F_{n-1})\\
C_n + 1 = 2 F_{n+1}\\\]

<p>Thus we have proved that for all \(n \ge 0\)</p>

\[C_n = 2 F_{n+1} - 1\]

<p>With this we have proved that the naive implementation presented earlier has exponential time complexity (the Fibonacci sequence grows exponentially), more specifically \(\Theta(\phi^{n})\), where \(\phi\) is the golden ratio.
The previous theorem is also the sole reason this binary-recursive procedure is useful in <a href="https://github.com/drujensen/fib">benchmarks</a>, as we may easily calculate the exact number of function calls performed and find the average invocation time.</p>

<h2 id="going-down-the-rabbit-hole">Going down the rabbit hole</h2>

<p>Looking back upon the recurrence relation \(F_k = F_{k-1} + F_{k-2}\) one may intuitively deduce that it should be possible to compute \(F_{k+n}\) in linear time if values of \(F_{k+n-1}\) and \(F_{k+n-2}\) are known.
For instance when \(n=0\) only a single addition is needed; for \(n=1\) we need two: the first to compute \(F_{k}\) and another to find \(F_{k+1}\); and so son.
The most common “efficient” procedure to compute Fibonacci numbers leverages this notion in order to reach \(\Theta(n)\) time complexity and looks something like this:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">fib</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
    <span class="n">previous</span> <span class="o">=</span> <span class="mi">1</span>
    <span class="n">current</span> <span class="o">=</span> <span class="mi">0</span>
    <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
        <span class="n">previous</span><span class="p">,</span> <span class="n">current</span> <span class="o">=</span> <span class="n">current</span><span class="p">,</span> <span class="n">previous</span> <span class="o">+</span> <span class="n">current</span>
    <span class="k">return</span> <span class="n">current</span>
</code></pre></div></div>

<p>A Python program that simply prints the result of <code class="language-plaintext highlighter-rouge">fib(47)</code> now takes about 0.04 seconds to complete on my machine (CPython v3.8.1).
In fact, that time is mostly the Python environment starting up and it is approximately the same when computing the 10,000th Fibonacci number.
We can even turn this function into a <a href="https://wiki.python.org/moin/Generators">generator</a> so as to compute Fibonacci numbers on demand as we iterate through them.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">fibs</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
    <span class="n">previous</span> <span class="o">=</span> <span class="mi">1</span>
    <span class="n">current</span> <span class="o">=</span> <span class="mi">0</span>
    <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
        <span class="k">yield</span> <span class="n">current</span>
        <span class="n">previous</span><span class="p">,</span> <span class="n">current</span> <span class="o">=</span> <span class="n">current</span><span class="p">,</span> <span class="n">previous</span> <span class="o">+</span> <span class="n">current</span>

<span class="c1"># printing the first 100 numbers of the sequence
</span><span class="k">for</span> <span class="n">n</span><span class="p">,</span> <span class="n">fn</span> <span class="ow">in</span> <span class="nf">enumerate</span><span class="p">(</span><span class="nf">fibs</span><span class="p">(</span><span class="mi">100</span><span class="p">)):</span>
    <span class="nf">print</span><span class="p">(</span><span class="sh">"</span><span class="s">fib(%d) = %d</span><span class="sh">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">fn</span><span class="p">))</span>
</code></pre></div></div>

<p>Although the loop-based version is definitely more efficient than the previous one, its implementation is not as obvious and does not resemble the sequence’s mathematical definition.
Perhaps we can improve this by sending <code class="language-plaintext highlighter-rouge">previous</code> and <code class="language-plaintext highlighter-rouge">current</code> as additional arguments to a recursive procedure:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">fib</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">previous</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span> <span class="n">current</span> <span class="o">=</span> <span class="mi">0</span><span class="p">):</span>
    <span class="k">if</span> <span class="n">n</span> <span class="o">&lt;=</span> <span class="mi">0</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">current</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="k">return</span> <span class="nf">fib</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="n">current</span><span class="p">,</span> <span class="n">previous</span> <span class="o">+</span> <span class="n">current</span><span class="p">)</span>
</code></pre></div></div>

<p>While performance remains mostly the same and the procedure is a tad more versatile (by setting <code class="language-plaintext highlighter-rouge">previous</code> and <code class="language-plaintext highlighter-rouge">current</code> to another pair of consecutive Fibonacci numbers we can now compute the sequence from any starting point, as if with a different “seed”), calling this new routine with somewhat bigger values of <code class="language-plaintext highlighter-rouge">n</code> yields a crash with the message <code class="language-plaintext highlighter-rouge">RecursionError: maximum recursion depth exceeded ...</code>.</p>

<p>This happens because Python does not optimize <a href="https://en.wikipedia.org/wiki/Tail_call">tail calls</a>, so each iteration takes some space in the call stack, eventually reaching its limit.
In fact, our procedure now has a space complexity of \(\Theta(n)\) while the loop version was just \(\Theta(1)\).
Most functional languages support proper tail recursion, but before we go there let’s see how C++ fares in this case (since C doesn’t have default argument values without <a href="https://modelingwithdata.org/arch/00000022.htm">macros</a>).</p>

<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">constexpr</span> <span class="kt">int</span> <span class="nf">fib</span><span class="p">(</span><span class="kt">int</span> <span class="n">n</span><span class="p">,</span> <span class="kt">int</span> <span class="n">prev</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span> <span class="kt">int</span> <span class="n">curr</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
	<span class="k">if</span> <span class="p">(</span><span class="n">n</span> <span class="o">&lt;=</span> <span class="mi">0</span><span class="p">)</span>
		<span class="k">return</span> <span class="n">curr</span><span class="p">;</span>
	<span class="k">else</span> <span class="p">[[</span><span class="n">likely</span><span class="p">]]</span>
		<span class="k">return</span> <span class="n">fib</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="n">curr</span><span class="p">,</span> <span class="n">prev</span> <span class="o">+</span> <span class="n">curr</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Modern compilers are able to recognize this pattern and perform Tail Call Optimization (TCO), so the procedure above is made equivalent to the constant-space <code class="language-plaintext highlighter-rouge">for</code> loop implementation.
Making the function <code class="language-plaintext highlighter-rouge">constexpr</code> also signals to the compiler that it should try to optimize calls for values known at compile time.
For instance, a <code class="language-plaintext highlighter-rouge">main</code> function that simply returns <code class="language-plaintext highlighter-rouge">fib(40)</code> gets compiled into</p>

<pre><code class="language-asm">main:
        mov     eax, 102334155
        ret
</code></pre>

<p>Since we wish for proper tail recursion as well as infinite-precision arithmetic, I choose Scheme as our next programming language.
In the example below, <code class="language-plaintext highlighter-rouge">iterate</code> is a tail-recursive function defined inside the body of <code class="language-plaintext highlighter-rouge">fib</code> that performs the computation in constant space and linear time.</p>

<div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">fib</span> <span class="nv">n</span><span class="p">)</span>
  <span class="p">(</span><span class="k">let</span> <span class="nv">iterate</span> <span class="p">((</span><span class="nf">n</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nf">previous</span> <span class="mi">1</span><span class="p">)</span> <span class="p">(</span><span class="nf">current</span> <span class="mi">0</span><span class="p">))</span>
    <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">&lt;=</span> <span class="nv">n</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">current</span>
        <span class="p">(</span><span class="nf">iterate</span> <span class="p">(</span><span class="nb">-</span> <span class="nv">n</span> <span class="mi">1</span><span class="p">)</span> <span class="nv">current</span> <span class="p">(</span><span class="nb">+</span> <span class="nv">previous</span> <span class="nv">current</span><span class="p">)))))</span>
</code></pre></div></div>

<h2 id="fft-the-fast-fibonacci-transform">FFT: the Fast Fibonacci Transform</h2>

<p>One may realize that each call to <code class="language-plaintext highlighter-rouge">iterate</code> is basically a transformation applied to the last two parameters while <code class="language-plaintext highlighter-rouge">n</code> is simply a countdown controling how many times this is repeated.
This can be represented as follows:</p>

\[T(c, p) = (c + p, c)\]

<p>If \(p\) and \(c\) are consecutive Fibonacci numbers, we can say that</p>

\[T(F_k, F_{k-1}) = (F_k + F_{k-1}, F_k) = (F_{k+1}, F_k)\]

<p>and thus</p>

\[T^n(F_k, F_{k-1}) = (F_{k+n}, F_{k+n-1})\]

<p>\(T\) happens to be a <a href="https://en.wikipedia.org/wiki/Linear_map">linear transformation</a> (proving that property is left as an exercise to the reader).
Its transformation matrix, sometimes called the Fibonacci matrix, is easily found:</p>

\[A \begin{bmatrix}c \\ p\end{bmatrix} = \begin{bmatrix}c + p \\ c\end{bmatrix}\\
A = \begin{bmatrix}A \begin{bmatrix}1\\0\end{bmatrix} &amp; A\begin{bmatrix}0\\1\end{bmatrix}\end{bmatrix}\\
A = \begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}\]

<p>After that, we can accomplish transformation composition through matrix multiplication, in other words:</p>

\[A^n \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}F_n \\ F_{n-1}\end{bmatrix}\]

<p>At this point we have found a method to compute Fibonacci numbers through matrix exponentiation, and since the latter can be achieved – through a <a href="https://en.wikipedia.org/wiki/Exponentiation_by_squaring">Successive Squaring</a> algorithm – in \(\Theta(\log_2 n)\) time, so can the former.
The Octave code below does exactly that.</p>

<pre><code class="language-octave">function F_n = fib(n)

	A = [1, 1;
	     1, 0];

	x = [0;
	     1];

	y = A^n * x;

	F_n = y(1);

endfunction
</code></pre>

<p>If you are into linear algebra, you may remember that there is a special basis in which a linear transformation can be represented as a diagonal matrix.
This makes it a little bit easier to perform matrix exponentiation: simply raise each diagonal element to the desired power and then go back to the canonical basis to find the end result.
Using all this <a href="https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors">eigen-stuff</a> leads us to Binet’s formula, the closed-form solution for Fibonacci numbers:</p>

\[F_n = {(\phi^n - \psi^n) \over (\phi - \psi)} = {(\phi^n - \psi^n) \over \sqrt{5}}\]

<p>where \(\phi\) is the golden ratio and \(\psi\) its complement (these are the eigenvalues for the Fibonacci transform).</p>

<p>So theoretically we could use that in our implementation:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="n">math</span> <span class="kn">import</span> <span class="n">sqrt</span>

<span class="n">PHI</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span> <span class="o">+</span> <span class="nf">sqrt</span><span class="p">(</span><span class="mi">5</span><span class="p">))</span> <span class="o">/</span> <span class="mi">2</span>  <span class="c1"># golden ratio number
</span><span class="n">PSI</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">PHI</span>            <span class="c1"># and its complement
</span>
<span class="k">def</span> <span class="nf">fib</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
    <span class="k">return</span> <span class="nf">round</span><span class="p">((</span><span class="n">PHI</span><span class="o">**</span><span class="n">n</span> <span class="o">-</span> <span class="n">PSI</span><span class="o">**</span><span class="n">n</span><span class="p">)</span> <span class="o">/</span> <span class="nf">sqrt</span><span class="p">(</span><span class="mi">5</span><span class="p">))</span>
</code></pre></div></div>

<p>However, for big values of <code class="language-plaintext highlighter-rouge">n</code> we still get an <code class="language-plaintext highlighter-rouge">OverflowError: (..., 'Numerical result out of range')</code>; in some languages instead of throwing an exception the program would return its representation of infinity.
This happens because now we’re dealing with floating-point numbers, which are not big enough for our desired range of values.</p>

<p>Going back to the Fibonacci matrix, we can assume that for some \(k\) (notice that it is true when \(k=1\))</p>

\[\begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}^k = \begin{bmatrix}F_{k+1} &amp; F_k\\ F_k &amp; F_{k-1}\end{bmatrix}\]

<p>then</p>

\[\begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}^k \begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix} = \begin{bmatrix}F_{k+1} &amp; F_k\\ F_k &amp; F_{k-1}\end{bmatrix} \begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}\\
\begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}^{k+1} = \begin{bmatrix}F_{k+1} + F_k &amp; F_k + F_{k-1}\\ F_{k+1} &amp; F_k\end{bmatrix}\\
\begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}^{k+1} = \begin{bmatrix}F_{k+2} &amp; F_{k+1}\\ F_{k+1} &amp; F_k\end{bmatrix}\]

<p>this means for all \(n \ge 1\)</p>

\[\begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}^n = \begin{bmatrix}F_{n+1} &amp; F_n\\ F_n &amp; F_{n-1}\end{bmatrix}\]

<p>At this point, we can square both sides:</p>

\[\begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}^n \begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}^n = \begin{bmatrix}F_{n+1} &amp; F_n\\ F_n &amp; F_{n-1}\end{bmatrix} \begin{bmatrix}F_{n+1} &amp; F_n\\ F_n &amp; F_{n-1}\end{bmatrix}\\
\begin{bmatrix}1 &amp; 1\\ 1 &amp; 0\end{bmatrix}^{2n} = \begin{bmatrix}F_{n+1}^{2} + F_n^2 &amp; F_n (F_{n+1} + F_{n-1})\\ F_n (F_{n+1} + F_{n-1}) &amp; F_n^2 + F_{n-1}^{2}\end{bmatrix}\]

<p>And since the general rule still applies, we have</p>

\[F_{2n} = F_n (F_{n+1} + F_{n-1}) = F_n (F_{n+1} + F_{n+1} - F_n) = F_n (2F_{n+1} - F_n)\\
F_{2n+1} = F_{n+1}^{2} + F_n^2\]

<p>This means we can compute Fibonacci numbers through a successive squaring method, thus achieving \(\Theta(\log_2 n)\) bigint arithmetic operations.
Evidently, this method has the same asymptotic time complexity as exponentating the Fibonacci matrix, but it may save a few operations.
The fast recursive procedure is given below in Haskell.
In case you’ve noticed, space complexity – the call stack – has risen from constant to \(\Theta(\log_2 n)\), but since logarithmic growth is so slow, most of the time there is no need to worry about stack overflows.</p>

<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fib_</span> <span class="o">::</span> <span class="kt">Int</span> <span class="o">-&gt;</span> <span class="p">(</span><span class="kt">Integer</span><span class="p">,</span> <span class="kt">Integer</span><span class="p">)</span>
<span class="n">fib_</span> <span class="mi">0</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">fib_</span> <span class="n">n</span> <span class="o">=</span> <span class="kr">let</span> <span class="p">(</span><span class="n">fk</span><span class="p">,</span> <span class="n">fk1</span><span class="p">)</span> <span class="o">=</span> <span class="n">fib_</span> <span class="p">(</span><span class="n">div</span> <span class="n">n</span> <span class="mi">2</span><span class="p">)</span>
             <span class="n">fn</span>        <span class="o">=</span> <span class="n">fk</span> <span class="o">*</span> <span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="n">fk1</span> <span class="o">-</span> <span class="n">fk</span><span class="p">)</span>
             <span class="n">fn1</span>       <span class="o">=</span> <span class="n">fk1</span><span class="o">^</span><span class="mi">2</span> <span class="o">+</span> <span class="n">fk</span><span class="o">^</span><span class="mi">2</span>
         <span class="kr">in</span> <span class="kr">if</span> <span class="n">mod</span> <span class="n">n</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span>
            <span class="kr">then</span> <span class="p">(</span><span class="n">fn</span><span class="p">,</span> <span class="n">fn1</span><span class="p">)</span>
            <span class="kr">else</span> <span class="p">(</span><span class="n">fn1</span><span class="p">,</span> <span class="n">fn</span> <span class="o">+</span> <span class="n">fn1</span><span class="p">)</span>

<span class="n">fib</span> <span class="o">::</span> <span class="kt">Int</span> <span class="o">-&gt;</span> <span class="kt">Integer</span>
<span class="n">fib</span> <span class="n">n</span> <span class="o">=</span> <span class="n">fn</span> <span class="kr">where</span> <span class="p">(</span><span class="n">fn</span><span class="p">,</span> <span class="kr">_</span><span class="p">)</span> <span class="o">=</span> <span class="n">fib_</span> <span class="n">n</span>
</code></pre></div></div>

<p>Another way to see this is to think of the previously mentioned transformation \(T\) as a special case of \(T_{pq}(a, b) = (a (q + p) + bq, aq + bp)\) when \(p = 0, q = 1\).
It is then always possible to find \(p', q'\) such that \(T_{p'q'} = T_{pq}^2\): just let \(p' = q^2 + p^2\) and \(q' = q^2 + 2qp\).
Let <code class="language-plaintext highlighter-rouge">n</code> be the number of transformations that need to be applied.
Then for each step, if <code class="language-plaintext highlighter-rouge">n</code> is odd we simply apply the transformation and go to the next iteration; otherwise <code class="language-plaintext highlighter-rouge">n</code> is even and we change <code class="language-plaintext highlighter-rouge">p</code> and <code class="language-plaintext highlighter-rouge">q</code> such that only half the number of transformations is now needed.
This is an exercise in Abelson &amp; Sussman’s classic, <a href="https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book-Z-H-11.html#footnote_Temp_70">Structure and Interpretation
of Computer Programs</a> (a.k.a. “The Wizard Book”).</p>

<div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">fib</span> <span class="nv">n</span><span class="p">)</span>
  <span class="p">(</span><span class="k">let</span> <span class="nv">iter</span> <span class="p">([</span><span class="nf">p</span> <span class="mi">0</span><span class="p">]</span> <span class="p">[</span><span class="nf">q</span> <span class="mi">1</span><span class="p">]</span> <span class="p">[</span><span class="nf">n</span> <span class="nv">n</span><span class="p">]</span> <span class="p">[</span><span class="nf">a</span> <span class="mi">1</span><span class="p">]</span> <span class="p">[</span><span class="nf">b</span> <span class="mi">0</span><span class="p">])</span>
    <span class="p">(</span><span class="k">cond</span> <span class="p">[(</span><span class="nb">&lt;=</span> <span class="nv">n</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">b</span><span class="p">]</span>
          <span class="p">[(</span><span class="nb">even?</span> <span class="nv">n</span><span class="p">)</span>
           <span class="p">(</span><span class="nf">iter</span> <span class="p">(</span><span class="nb">+</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">q</span> <span class="nv">q</span><span class="p">)</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">p</span> <span class="nv">p</span><span class="p">))</span>
                 <span class="p">(</span><span class="nb">+</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">q</span> <span class="nv">q</span><span class="p">)</span> <span class="p">(</span><span class="nb">*</span> <span class="mi">2</span> <span class="nv">p</span> <span class="nv">q</span><span class="p">))</span>
                 <span class="p">(</span><span class="nb">/</span> <span class="nv">n</span> <span class="mi">2</span><span class="p">)</span> <span class="nv">a</span> <span class="nv">b</span><span class="p">)]</span>
          <span class="p">[</span><span class="k">else</span>
           <span class="p">(</span><span class="nf">iter</span> <span class="nv">p</span> <span class="nv">q</span> <span class="p">(</span><span class="nb">-</span> <span class="nv">n</span> <span class="mi">1</span><span class="p">)</span>
                 <span class="p">(</span><span class="nb">+</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">a</span> <span class="p">(</span><span class="nb">+</span> <span class="nv">q</span> <span class="nv">p</span><span class="p">))</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">b</span> <span class="nv">q</span><span class="p">))</span>
                 <span class="p">(</span><span class="nb">+</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">a</span> <span class="nv">q</span><span class="p">)</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">b</span> <span class="nv">p</span><span class="p">)))])))</span>
</code></pre></div></div>

<h2 id="memoization-and-streams">Memoization and Streams</h2>

<p>Another common way to obtain Fibonacci numbers is to pre-compute them such that each input <code class="language-plaintext highlighter-rouge">n</code> maps to a memory location where \(F_n\) is stored.
\(F_n\) can thus be thought of as a vector being indexed at its nth position.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fibs_</span> <span class="o">=</span> <span class="p">[</span><span class="n">fn</span> <span class="k">for</span> <span class="n">fn</span> <span class="ow">in</span> <span class="nf">fibs</span><span class="p">(</span><span class="mi">1000</span><span class="p">)]</span> <span class="c1"># fibs is the generator
</span>
<span class="k">def</span> <span class="nf">memo_fib</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
    <span class="k">if</span> <span class="n">n</span> <span class="o">&lt;</span> <span class="nf">len</span><span class="p">(</span><span class="n">fibs_</span><span class="p">):</span>
        <span class="k">return</span> <span class="n">fibs_</span><span class="p">[</span><span class="n">n</span><span class="p">]</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="n">curr</span><span class="p">,</span> <span class="n">prev</span> <span class="o">=</span> <span class="n">fibs_</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">fibs_</span><span class="p">[</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span>
        <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="nf">len</span><span class="p">(</span><span class="n">fibs_</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span>
            <span class="n">prev</span><span class="p">,</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">curr</span><span class="p">,</span> <span class="n">curr</span> <span class="o">+</span> <span class="n">prev</span>
            <span class="n">fibs_</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">curr</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">curr</span>
</code></pre></div></div>

<p>The Python code above allocates a chunk of memory and fills it sequentially with the Fibonacci sequence such that later accesses need no computation and could, theoretically, be made in \(\Theta(1)\) time.
When the number is too far down the sequence and hasn’t been calculated yet, we start with the closest pair of Fibonacci numbers and continue to fill the vector until the desired number is reached.</p>

<p>This approach may end up taking too much space, so there’s still some optimization to be done.
Another issue is that memory is updated dynamically, requiring eventual reallocations and other such expensive operations.</p>

<p>Using a language with better support for lambdas and first-class functions, we could turn this memoization / tabling technique into something a little more generic.
In this case we use Guile’s hash-tables to map a procedure’s argument list to its cached results.</p>

<div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">;; make a function that caches its past results</span>
<span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">memoize</span> <span class="nv">proc</span><span class="p">)</span>
  <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">cache</span> <span class="p">(</span><span class="nf">make-table</span><span class="p">)))</span>
    <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">delegate</span> <span class="o">.</span> <span class="nv">args</span><span class="p">)</span>
      <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">hit</span> <span class="p">(</span><span class="nf">lookup</span> <span class="nv">cache</span> <span class="nv">args</span><span class="p">)))</span>
        <span class="p">(</span><span class="k">or</span> <span class="nv">hit</span>
            <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">result</span> <span class="p">(</span><span class="nb">apply</span> <span class="nv">proc</span> <span class="nv">args</span><span class="p">)))</span>
              <span class="p">(</span><span class="nf">insert!</span> <span class="nv">cache</span> <span class="nv">args</span> <span class="nv">result</span><span class="p">)</span>
              <span class="nv">result</span><span class="p">))))</span>
    <span class="nv">delegate</span><span class="p">))</span>

<span class="c1">;; GUILE specific</span>
<span class="p">(</span><span class="k">define</span> <span class="nv">make-table</span> <span class="nv">make-hash-table</span><span class="p">)</span>
<span class="p">(</span><span class="k">define</span> <span class="nv">lookup</span> <span class="nv">hash-ref</span><span class="p">)</span>
<span class="p">(</span><span class="k">define</span> <span class="nv">insert!</span> <span class="nv">hash-set!</span><span class="p">)</span>

<span class="c1">;; example</span>
<span class="p">(</span><span class="k">define</span> <span class="nv">memo-fib</span>
  <span class="p">(</span><span class="nf">memoize</span>
    <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">n</span><span class="p">)</span>
      <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">&lt;=</span> <span class="nv">n</span> <span class="mi">1</span><span class="p">)</span> <span class="nv">n</span>
          <span class="p">(</span><span class="nb">+</span> <span class="p">(</span><span class="nf">memo-fib</span> <span class="p">(</span><span class="nb">-</span> <span class="nv">n</span> <span class="mi">1</span><span class="p">))</span> <span class="p">(</span><span class="nf">memo-fib</span> <span class="p">(</span><span class="nb">-</span> <span class="nv">n</span> <span class="mi">2</span><span class="p">)))))))</span>
</code></pre></div></div>

<p>This memoization process can be thought of as a data structure which contains past results and generates new ones on demand.
The same thing happens with <a href="https://en.wikipedia.org/wiki/Lazy_evaluation">lazy</a> lists, which are potentially infinite data structures also known as streams.
Haskell, being a lazy programming language, has this behaviour by default:</p>

<div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fibs</span> <span class="o">::</span> <span class="kt">Integer</span> <span class="o">-&gt;</span> <span class="kt">Integer</span> <span class="o">-&gt;</span> <span class="p">[</span><span class="kt">Integer</span><span class="p">]</span>
<span class="n">fibs</span> <span class="n">prev</span> <span class="n">curr</span> <span class="o">=</span> <span class="n">prev</span> <span class="o">:</span> <span class="n">fibs</span> <span class="n">curr</span> <span class="p">(</span><span class="n">prev</span> <span class="o">+</span> <span class="n">curr</span><span class="p">)</span>

<span class="n">main</span> <span class="o">=</span> <span class="kr">do</span>
    <span class="kr">let</span> <span class="n">fib</span> <span class="o">=</span> <span class="n">fibs</span> <span class="mi">0</span> <span class="mi">1</span> <span class="c1">-- fib contains the entire sequence</span>
    <span class="n">print</span> <span class="p">(</span><span class="n">fib</span> <span class="o">!!</span> <span class="mi">47</span><span class="p">)</span>  <span class="c1">-- Fn = fib !! n</span>
</code></pre></div></div>

<p>The snippet above shows a possibly infinite list named <code class="language-plaintext highlighter-rouge">fib</code> which contains the whole Fibonacci sequence.
It is returned by the function <code class="language-plaintext highlighter-rouge">fibs</code> in constant time and each indexing operation on it sequentially computes Fibonacci numbers while storing past results in the beggining of the list.</p>

<h2 id="negafibonacci">Negafibonacci</h2>

<p>All of the techniques and algorithms shown so far consider the Fibonacci sequence to be indexed by natural numbers.
Most implementations seen in the wild also make this assumption, sometimes throwing an error when negative integers are used, others just blatantly ignoring these inputs.</p>

<p>The sequence can be easily extended with the “Negafibonacci” numbers by following</p>

\[F_n = F_{n-1} + F_{n-2}\\
F_{n-2} = F_n - F_{n-1}\]

<p>which leads us to</p>

\[F_{-1} = F_{1} - F_{0} = 1 - 0 = 1\\
F_{-2} = F_{0} - F_{-1} = 0 - 1 = -1\\
F_{-3} = ... = 2\\
F_{-4} = -3\\
F_{-5} = 5\\
...\]

<p>It is then easy to notice that whenever \(n &lt; 0\)</p>

<p>\(F(n) = F(-n)\) if \(n\) is odd, and</p>

<p>\(F(n) = -F(-n)\) if \(n\) is instead even.</p>

<p>Some may be tempted to write this in the mathematically equivalent form</p>

\[F(n &lt; 0) = (-1)^{n-1} F(-n)\]

<p>but I personally discourage doing this in actual code as it makes it much less evident what the property is and may lead to an expensive exponentiation process just to get the right sign when compared to a simple conditional:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fn</span> <span class="o">=</span> <span class="nf">fib</span><span class="p">(</span><span class="nf">abs</span><span class="p">(</span><span class="n">n</span><span class="p">))</span>
<span class="k">if</span> <span class="n">n</span> <span class="o">&lt;</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">n</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
    <span class="k">return</span> <span class="o">-</span><span class="n">fn</span>
<span class="k">else</span><span class="p">:</span>
    <span class="k">return</span> <span class="n">fn</span>
</code></pre></div></div>

<h2 id="tldr">TL;DR</h2>

<p>If you came here to see a code snippet with a final, optimal algorithm for Fibonacci numbers, then I’m sorry to disappoint you.
Instead, the conclusion of this overly long post is that there will always be many ways to compute something, each with its own tradeoffs.</p>

<p>In the end, though, I can say that:</p>

<ul>
  <li>Unless you’re benchmarking something, forget about the double-recursive procedure.</li>
  <li>The sequence grows exponentially, so bignums are a must.</li>
  <li>When Fibonacci numbers are only required sparsely, use some form of the fast transform to guarantee \(\Theta(\log_2 n)\) time and \(\Theta(1)\) space complexity.</li>
  <li>A generator is a very clean way to iterate through Fibonacci numbers sequentially.</li>
  <li>If the whole sequence – or perhaps some slice of it – is needed, streams are the way to go.</li>
  <li>Dynamic memoization and/or static tabling is the approach people usually suggest to optimize Fibonacci numbers, but the memory overhead is hardly ever worth it: consider using the fast transform instead.
Of course, if a value can be statically optimized (during compilation, for instance, with C++’s <code class="language-plaintext highlighter-rouge">constexpr</code>), then this is usually preferred.</li>
  <li>Translating math directly into code usually turns out terribly inefficient (see the double-recursive procedure), but using it smartly gives you very useful properties (see the FFT).</li>
  <li>Knowing different programming languages gives insight into useful techniques you wouldn’t normally consider.</li>
</ul>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="algorithms" /><summary type="html"><![CDATA[One problem, multiple algorithms]]></summary></entry><entry><title type="html">On the computation of π</title><link href="https://www.baioc.dev/tutorial/pi-methods/" rel="alternate" type="text/html" title="On the computation of π" /><published>2019-07-07T00:00:00+00:00</published><updated>2019-07-07T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/pi-methods</id><content type="html" xml:base="https://www.baioc.dev/tutorial/pi-methods/"><![CDATA[<head>
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>On the computation of &pi;</title>
  <meta name="generator" content="Org mode" />
  <meta name="author" content="Gabriel B. Sant'Anna" />
  <script type="text/x-mathjax-config">
    MathJax.Hub.Config({
        displayAlign: "center",
        displayIndent: "0em",

        "HTML-CSS": { scale: 100,
                      linebreaks: { automatic: "false" },
                      webFont: "TeX"},
        SVG: {scale: 100,
              linebreaks: { automatic: "false" },
              font: "TeX"},
        NativeMML: {scale: 100},
        TeX: { equationNumbers: {autoNumber: "AMS"},
               MultLineWidth: "85%",
               TagSide: "right",
               TagIndent: ".8em"}
    });
  </script>
  <script type="text/javascript"
    src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS_HTML">
  </script>
</head>

<body>
  <div id="content">

    <div id="outline-container-orga7b4068" class="outline-2">
      <h2 id="orga7b4068"><span class="section-number-2">1</span> Asking the math library</h2>
      <div class="outline-text-2" id="text-1">
        <p>My computer tells me that &pi; is <i>approximatively</i></p>

        <div class="org-src-container">
          <pre class="src src-python"><span style="color: #F92672;">from</span> math <span style="color: #F92672;">import</span> *pi</pre>
        </div>

        <pre class="example">3.141592653589793</pre>
      </div>
    </div>


    <div id="outline-container-orgeb0560f" class="outline-2">
      <h2 id="orgeb0560f"><span class="section-number-2">2</span> Buffon's needle</h2>
      <div class="outline-text-2" id="text-2">
        <p>
          By the method of <a href="http://www.cut-the-knot.org/fta/Buffon/buffon9.shtml">Buffon's needle</a> we
          get the <b>approximation</b>
        </p>

        <div class="org-src-container">
          <pre class="src src-python"><span style="color: #F92672;">import</span> numpy <span style="color: #F92672;">as</span> np

<span style="color: #FD971F;">N</span> = <span style="color: #AE81FF;">10000</span>
np.random.seed<span style="color: #AE81FF;">(</span>seed=<span style="color: #AE81FF;">42</span><span style="color: #AE81FF;">)</span>

<span style="color: #FD971F;">x</span> = np.random.uniform<span style="color: #AE81FF;">(</span>size=N, low=<span style="color: #AE81FF;">0</span>, high=<span style="color: #AE81FF;">1</span><span style="color: #AE81FF;">)</span>
<span style="color: #FD971F;">theta</span> = np.random.uniform<span style="color: #AE81FF;">(</span>size=N, low=<span style="color: #AE81FF;">0</span>, high=pi/<span style="color: #AE81FF;">2</span><span style="color: #AE81FF;">)</span>

<span style="color: #FD971F;">P</span> = <span style="color: #F92672;">sum</span><span style="color: #AE81FF;">(</span>x + np.sin<span style="color: #66D9EF;">(</span>theta<span style="color: #66D9EF;">)</span> &gt; <span style="color: #AE81FF;">1</span><span style="color: #AE81FF;">)</span> / N
<span style="color: #AE81FF;">2</span>/P
</pre>
        </div>

        <pre class="example">3.128911138923655</pre>
      </div>
    </div>


    <div id="outline-container-orgb6fc6bd" class="outline-2">
      <h2 id="orgb6fc6bd"><span class="section-number-2">3</span> Using Monte Carlo experiments</h2>
      <div class="outline-text-2" id="text-3">
        <p>
          A method that is easier to understand and does not make use of the \(\sin\) function is based on the fact that
          if \(X \sim U(0,1)\) and \(Y \sim U(0,1)\), then \(P[X^2+Y^2 \leq 1] = \pi/4\) (see <a
            href="https://en.wikipedia.org/wiki/Monte_Carlo_method">"Monte Carlo method" on Wikipedia</a>).
          The following code uses this approach:
        </p>

        <div class="org-src-container">
          <pre class="src src-python"><span style="color: #F92672;">import</span> matplotlib.pyplot <span style="color: #F92672;">as</span> plt

np.random.seed<span style="color: #AE81FF;">(</span>seed=<span style="color: #AE81FF;">42</span><span style="color: #AE81FF;">)</span>
<span style="color: #FD971F;">x</span> = np.random.uniform<span style="color: #AE81FF;">(</span>size=N, low=<span style="color: #AE81FF;">0</span>, high=<span style="color: #AE81FF;">1</span><span style="color: #AE81FF;">)</span>
<span style="color: #FD971F;">y</span> = np.random.uniform<span style="color: #AE81FF;">(</span>size=N, low=<span style="color: #AE81FF;">0</span>, high=<span style="color: #AE81FF;">1</span><span style="color: #AE81FF;">)</span>

<span style="color: #FD971F;">inside</span> = <span style="color: #AE81FF;">(</span>x**<span style="color: #AE81FF;">2</span> + y**<span style="color: #AE81FF;">2</span><span style="color: #AE81FF;">)</span> &lt;= <span style="color: #AE81FF;">1</span>
<span style="color: #FD971F;">outside</span> = np.logical_not<span style="color: #AE81FF;">(</span>inside<span style="color: #AE81FF;">)</span>

<span style="color: #FD971F;">fig</span>, <span style="color: #FD971F;">ax</span> = plt.subplots<span style="color: #AE81FF;">(</span><span style="color: #AE81FF;">1</span><span style="color: #AE81FF;">)</span>
ax.scatter<span style="color: #AE81FF;">(</span>x<span style="color: #66D9EF;">[</span>inside<span style="color: #66D9EF;">]</span>, y<span style="color: #66D9EF;">[</span>inside<span style="color: #66D9EF;">]</span>, c=<span style="color: #E6DB74;">'b'</span>, alpha=<span style="color: #AE81FF;">0</span>.<span style="color: #AE81FF;">2</span>, edgecolor=<span style="color: #AE81FF;">None</span><span style="color: #AE81FF;">)</span>
ax.scatter<span style="color: #AE81FF;">(</span>x<span style="color: #66D9EF;">[</span>outside<span style="color: #66D9EF;">]</span>, y<span style="color: #66D9EF;">[</span>outside<span style="color: #66D9EF;">]</span>, c=<span style="color: #E6DB74;">'r'</span>, alpha=<span style="color: #AE81FF;">0</span>.<span style="color: #AE81FF;">2</span>, edgecolor=<span style="color: #AE81FF;">None</span><span style="color: #AE81FF;">)</span>
ax.set_aspect<span style="color: #AE81FF;">(</span><span style="color: #E6DB74;">'equal'</span><span style="color: #AE81FF;">)</span>

plt.savefig<span style="color: #AE81FF;">(</span>matplot_lib_filename<span style="color: #AE81FF;">)</span>
<span style="color: #F92672;">print</span><span style="color: #AE81FF;">(</span>matplot_lib_filename<span style="color: #AE81FF;">)</span>
</pre>
        </div>

        <div class="figure">
          <p><img src="/assets/images/pi-square.png" alt="/assets/images/pi-square.png"/></p>
        </div>

        <p>
          It is then straightforward to obtain an approximation to &pi; by counting how many times, on
          average, \(X^2 + Y^2 \le 1\):
        </p>

        <div class="org-src-container">
          <pre class="src src-python"><span style="color: #AE81FF;">4</span> * np.mean<span style="color: #AE81FF;">(</span>inside<span style="color: #AE81FF;">)</span></pre>
        </div>

        <pre class="example">3.1556</pre>
      </div>
    </div>


    <div id="outline-container-orga44e510" class="outline-2">
      <h2 id="orga44e510"><span class="section-number-2">4</span> Through Wallis' product</h2>
      <div class="outline-text-2" id="text-4">
        <p>
          <a href="https://en.wikipedia.org/wiki/Wallis_product">John Wallis' product</a> for &pi; states that the
          infinite product below converges to \(\pi / 2\).
          Check out the <a href="https://www.youtube.com/watch?v=8GPy_UMV-08">geometrical proof</a>.
        </p>

        <div class="org-src-container">
          <pre class="src src-python"><span style="color: #FD971F;">prod</span> = <span style="color: #AE81FF;">1</span>
<span style="color: #F92672;">for</span> i <span style="color: #F92672;">in</span> <span style="color: #F92672;">range</span><span style="color: #AE81FF;">(</span><span style="color: #AE81FF;">2</span>, N<span style="color: #AE81FF;">)</span>:
    <span style="color: #FD971F;">prod</span> *= i / <span style="color: #AE81FF;">(</span>i-<span style="color: #AE81FF;">1</span><span style="color: #AE81FF;">)</span> <span style="color: #F92672;">if</span> i % <span style="color: #AE81FF;">2</span> == <span style="color: #AE81FF;">0</span> <span style="color: #F92672;">else</span> <span style="color: #AE81FF;">(</span>i-<span style="color: #AE81FF;">1</span><span style="color: #AE81FF;">)</span> / i

prod * <span style="color: #AE81FF;">2</span>
</pre>
        </div>

        <pre class="example">3.141435562175509</pre>
      </div>
    </div>


    <div id="outline-container-org3475054" class="outline-2">
      <h2 id="org3475054"><span class="section-number-2">5</span> As the root of a function</h2>
      <div class="outline-text-2" id="text-5">
        <p>
          As we know, \(sin(\pi) = 0\), thus value for &pi; can be found by approximating the root (one of them, that is) of
          \(sin(x)\).
          This can be done using <a href="https://en.wikipedia.org/wiki/Newton's_method">Newton's method</a>, for
          instace.
        </p>

        <div class="org-src-container">
          <pre class="src src-python"><span style="color: #F92672;">from</span> scipy.optimize <span style="color: #F92672;">import</span> fsolve
<span style="color: #FD971F;">results</span> = fsolve<span style="color: #AE81FF;">(</span>sin, x0=<span style="color: #AE81FF;">3</span><span style="color: #AE81FF;">)</span>
results<span style="color: #AE81FF;">[</span><span style="color: #AE81FF;">0</span><span style="color: #AE81FF;">]</span>
</pre>
        </div>

        <pre class="example">3.141592653589793</pre>
      </div>
    </div>

  </div>
</body>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="numerical" /><category term="algorithms" /><summary type="html"><![CDATA[How to approximate the constant through 5 different methods]]></summary></entry></feed>