my love hate relationship with open source

Ladybird is a browser with its own engine, built from scratch, which feels small but let me tell you how amazing it is. The engine is the part of a browser that takes the HTML and CSS and JavaScript a website sends you and turns it into the page we look at. Ladybird has its own engine, LibWeb, and its own JavaScript engine, LibJS. That is pretty rare. Chrome is Google’s build of a project called Chromium, and Chromium’s engine is called Blink. Edge uses Blink. So do Brave, Opera and Vivaldi. Safari uses a different engine called WebKit, and Blink was split off from WebKit back in 2013, so those two started from the same code. Firefox has its own engine called Gecko and a few percent of the market. So most of the web is basically being rendered by three engines, and one company, which is Google, controls the one almost everybody uses.

Ladybird is not built on any of them. So obviously I wanted to see and try what it was like. Being able to open something up and look at how it works is the reason I like open source. It is also the reason it eats whole days of mine.

The first problem was installing it. On Arch I install things with a command called yay, so I ran yay -S ladybird and waited. Chrome takes like five seconds to download and install because Google already compiled it for us, prepackaged and ready. The Arch User Repository does not often hand out finished programs. You download the source code and compile it yourself. Compiling means taking the source code from the programmer and translating it into the instructions a processor actually runs. So my laptop had to compile all of Ladybird, and then compile everything Ladybird depends on. The project lists about forty libraries directly and each of those needs libraries of its own, so the build came to seventy seven packages, because a browser has to open every image format, play video, and draw text in every writing system in the world. The whole thing took two hours. I cancelled it by accident halfway through, which did not help.

While it builds, the names of those libraries print out in the terminal, and a lot of them I noticed turn out to be the same ones Chrome uses. Skia, which is the library that draws shapes and text onto the screen, is Google’s. ANGLE, which translates graphics instructions so they work on different hardware, is also Google’s. They use third party libraries for things like images and graphics and encryption, and harfbuzz for text shaping, and their FAQ says as much. So when they say independent it means the part that reads a page and decides what it should look like. This is not a contradiction to Ladybird being independent. It is just that they are not trying to rewrite every piece of software that exists for this. The important part is that they are not using somebody else’s browser engine, and I think that is a pretty reasonable definition of independent.

When it finally finished, it crashed immediately. The error was a segfault, which is what happens when a program touches a piece of memory it was not supposed to touch and Linux kills it. Linux keeps a record of these, and inside that record is a stack trace, which is a list of every function that was in the middle of running at the moment things went wrong. Reading down the list, the crash was inside a library called harfbuzz. Its job is to work out the exact position of every single letter before any text appears on our screen. That may not mean much for English, where letters are just set up in a row. In Devanagari the letters change shape and connect to each other depending on what comes before and after them, and something has to figure all of that out. Suddenly putting letters on a screen becomes a pretty complicated problem.

Anyway, the reason it broke is that there were two copies of harfbuzz in the same program. Ladybird ships with its own copy, built along with everything else, and my system already had one because a bunch of other programs use it too as a shared library to display text, so some other program on my laptop pulled it in long ago. The desktop theme plugin was loading the system version while Ladybird was using its own version, and two versions of the same library inside one process did not survive with each other. So I disabled the theme plugin and the browser finally opened. Progress.

Then nothing loaded over HTTPS. Plain HTTP pages turned out fine, and the log said curl error 77, problem with the SSL CA cert.

Basically, when you connect to a website over HTTPS your computer needs a list of certificate authorities it trusts, and that list is a file on disk called the CA bundle. Error 77 means curl could not read that file. Which was weird, because the file existed and was readable by every other program on my computer.

The difference is that Ladybird runs its networking in a separate process and locks that process down using a Linux feature called Landlock. Landlock lets a program voluntarily give up access to almost the entire filesystem and keep only a short list of directories it promises it will need. That way, if somebody breaks in through a malicious website, the part they broke into cannot read the rest of your files. Ladybird’s list is six entries long and one of them is /etc/ssl, which is the normal home of the CA bundle.

So the thing breaking my browser is a safety feature. Somebody sat down and wrote that sandbox to make the browser harder to attack, and on my machine it is the exact reason nothing would load. It is not a bug somebody left behind by being careless. It came from someone being careful.

On Arch, though, /etc/ssl/certs/ca-certificates.crt is not actually the real file. It is a symlink, a pointer to the real certificate bundle, which lives at /etc/ca-certificates/extracted/. So Ladybird was allowed to access /etc/ssl, but the thing /etc/ssl was pointing at lived outside the sandbox. Landlock checks the actual location being accessed and not just the symlink, so curl was blocked from opening a file I had checked and could read without any problem.

I went and read the code afterwards because I wanted to understand why the fix works. It turns out that when you give Ladybird a certificate path, it takes the directory that certificate is in and adds it to the sandbox permissions before locking everything down. So passing --certificate /etc/ca-certificates/extracted/tls-ca-bundle.pem works and the sandbox stays on.

There is a catch with that fix. The desktop entry that gets installed alongside the browser registers Ladybird as a handler for https links, and the command in it does not include the certificate flag. So if you set it as your default browser, every link you click from another program still fails.

Working all of that out took me most of the day.

Then I went to report it and found out someone had already filed it back in June. A few other people on Arch had confirmed it since. There is already a fix sitting in an open pull request, but it is stuck because the reviewer pointed out a problem with it. That fix reads the certificate file into memory first, before the sandbox turns on, so the sandbox never blocks it. But if you leave the browser open for days and your system updates its certificates in the meantime, the browser keeps using the old copy it read at startup. Two other people said it would be better to just follow the symlink, find where the file really is, and add that place to the allowed list. Then it can keep reading from disk like normal. Nobody has written that one yet.

That tracker has 499 open issues.

I keep thinking about that number. I spent a full day on one of them and added nothing that was not already there. Somebody has to read all of them. Most are duplicates, and someone still has to reply politely to every one. I have repos with a few hundred stars that I stopped maintaining because people kept opening issues and I got tired of it.

And that is the part of open source I have a weird love hate relationship with. I love that I can look inside something this complicated. I hate how much time it takes. I wanted to contribute, but since I was not sure the maintainers would prefer my approach when others had already given suggestions, I felt it would be a waste of time and effort. Plus I would have had to wait another two hours to compile and run it.

There is so much time already inside all of this. The interstate highway system in the United States was the largest public works project the country has ever done. Around forty eight thousand miles of road, hundreds of billions of dollars in today’s money, and roughly thirty five years of work to finish it. Nobody has to be convinced that was hard. We point at a bridge and cannot help thinking, holy shit, someone built that.

Software does not feel that way at all. It weighs almost nothing, and since it rarely has a physical body it is easy to forget that people made it. A few megabytes sitting on your SSD can represent years of someone’s whole life.

Later that evening I bought some chicken tenders on my way back to the dorm, and I looked at the barcode on the receipt and started calculating. There are thousands of these store chains everywhere, and every second somebody is scanning a barcode, and that scan has to end up in the right place in a system that keeps everything together. Somebody had to design the barcode format, write the software, write the database, write the language all of this was built in, write the compiler for that language, and before any of that, sit down with everyone else in history working on it and decide how it should work at all, lol. Which is something to feel holy shit about. And most of us never think about any of it.