Everything you wanted to know about Raspberry Pi (RPI) images – Part 2

In Part 1 of this blog post I described the Raspberry Pi boot image structure (a very simple version) and provided a very brief description of how the boot process works.

This part talks about how you can build your own images for a Raspberry Pi and setups a framework for people to collaborate and share their image components with each other.

In subsequent blogs I’ll build on the concepts here and describe other more sophisticated configurations and progressively share more of the elements I have built for myself, cleaning them up, before I make them public.

This blog post is structured into three parts:

  • Building your own images
  • Writing your own elements
  • Collaborating with elements

I build my own Raspberry Pi boot images from scratch. I do this so I can build images in a repeatable way, configuring the software I want in just the way I want it.

It is much more effective to do this, than to take an image from some other source and then remove things which I don’t want and work around the quirks of the person who build that image to begin with.

In doing this, I can control exactly what goes on the image and make sure that it comes out the exact same way each time.

My tool of choice (as I use this for work related things as well) is diskimage-builder. Like other image building tools, this one is quirky, but I’ve got to understand it and I prefer it to the pi-gen tool that is used to build the official Raspbian images.

Building your own images

This will work for you if you have any old Linux machine (I use an Ubuntu 16.04LTS x86_64 machine). A simple to follow Quick Start Guide is available here.

To a reasonably Linux savvy person, building your first image should take no more than 20 minutes and you can have your Raspberry Pi up and running on your own custom image in about 30 minutes.

The steps are simple:

  • Clone the repository of tools and elements
  • Install prerequisite software
  • Build your image
  • Install your image
  • Have fun!

Writing your own elements

diskimage-builder is a very flexible tool and it is very easy to write elements once you understand the general flow of things. Like all image building tools, it uses two environments, a host environment and an image environment. Some tools make the image environment a chroot (like diskimage-builder) others use a VM or a docker container.

diskimage-builder works by allowing users to select elements, define dependencies between elements, and execute scripts provided in those elements in a deterministic order. Some of the scripts are executed in the host environment, others are executed in the image environment. Finally, it takes the image environment that was created and packages it up into the format of your choice.

I use diskimage-builder to make images for a cloud (and I use qcow2 for that). For Raspberry Pi, I use the ‘raw’ format which is nothing more than a disk image complete with MBR (boot record), partition table, and volumes.

To write your own element, you can follow along with one of the many elements provided and insert your own commands. It is a very short, and not very steep learning curve.

Collaborating with elements

I’ve shared (and I will continue to share) elements that build a basic Raspbian based image, a basic Debian based image, and elements that will install specific packages, configure WiFi, configure a secure user, and so on.

For example, consider the element rpi-generic which produces a Raspberry Pi image that I use as a basic building block for many of my applications. It does not, by itself, provide an operating system. So, I could choose either rpi-debian-core or rpi-raspbian-core. With just those two elements, I could execute the command

disk-image-create rpi-debian-core rpi-generic \
-a armhf -o debian -t raw -n

and this would produce a file (debian.raw) which was a properly bootable Raspberry Pi image.

But, since rpi-generic depends on other elements the diskimage-builder tool looks for those and uses them. One of the elements it depends on is rpi-resize-root. Let’s look at that for a second.

The MicroSD card I use is often 8 or 16GB, the image (for speed) is just 2GB. So, when you install the image onto the MicroSD card, and you boot the machine for the first time, the software installed and configured by the rpi-resize-root element expands the root file system to take the full available space.

Similarly, rpi-generic depends on rpi-wifi. That element does the few things to configure my Raspberry Pi to automatically connect to the WiFi network.

If someone else wrote an element that I wanted to use, all I would have to do is get that element (clone the repository that they shared) and tell diskimage-builder where to find the element!

For example, the sample build.bash script contains this line, which can easily be extended like a standard Linux PATH variable!

As a practical matter, I have an element for my WiFi access point that I use at home (which I will share as soon as I can clean it up). That element is on my machine at /rpi-image-builder-private/elements/rpi-wifi-access-point.

The elements which I’ve shared on the rpi-image-builder repository are at /rpi-image-builder/elements/.

To build my WiFi access point, I specify:

export ELEMENTS_PATH=/rpi-image-builder-private/elements:/rpi-image-builder/elements

I encourage you to write and share elements, and I believe that this will be a great way for all Raspberry Pi users to collaborate on their cool projects.

Everything you wanted to know about Raspberry Pi (RPI) images

This is a two part blog post that covers a lot of topics about a rather boring aspect of the RPI. I suspect that the vast majority of people using a RPI wouldn’t be interested in most of this stuff. But, there are a small number of people out there (like myself) who will no doubt find this stuff very useful.

This (the first of two) posts covers the booting process of an RPI. You need to have some idea of how this works to make your own images.

I have a RPI 3, some of this stuff may not be accurate for other models.

RPI Booting demystified

I have not been able to find any documentation on the bootloader that runs in the firmware but by trial and error, here is what I’ve found.

The firmware accesses the partition table on the MicroSD card which it assumes is an MS-DOS style partition table. The partition table is at the very beginning of the card and is part of the Master Boot Record (MBR). A good write up about this layout is found here [1].

The bootloader looks (it appears) for an MS-DOS LBA addressable partition. I have found that partition types 0x0C and 0x0E work. In order to boot reliably, I have also found that this must be the first partition on the disk.

RPI’s don’t use bootstrap code on the MicroSD card.

This is very important to if you wish to make your own images; it makes that process a whole lot simpler. The bootloader in firmware looks instead at this small MS-DOS partition to find a kernel, all the drivers it needs, and a pointer to the root file system.

The bootloader parses the cmdline.txt file to find the root partition. Here is a small portion of the cmdline.txt on one of my RPIs.

root=/dev/mmcblk0p2 rootfstype=ext4

The RPI addresses the MicroSD as /dev/mmcblk0 and the partitions on it are (in order) addressed as /dev/mmcblk0p1, /dev/mmcblk0p2, and so on.

The cmdline.txt file identifies the root partition either by providing this name, or by specifying a partition UUID, or a label. In the example above, cmdline.txt points to the second partition on the MicroSD card.

I found exactly one place that appears to document this in some detail, that is here[2].

Troubleshooting the boot process

When you power on the RPI, the red light indicates power. The green light (ACT/D5) indicates SD card access. If the red light comes on but the green one doesn’t it means that the firmware did not find a card it liked. This has been the most common problem described by people online.

If you have the tools to debug it, here are some things that I have found to be helpful.

If you can put this MicroSD card into a machine where you can mount it (and hopefully that machine is Linux based; if it is Windows based, my apologies, I can’t help you).

lsblk will help you identify the device. For example on my machine it is /dev/sdc. Here is the output of lsblk on my machine. The MicroSD card is shown in bold letters.

$ lsblk
sda 8:0 0 150G 0 disk
├─sda1 8:1 0 134G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 16G 0 part [SWAP]
sdc 8:32 1 29.7G 0 disk
├─sdc1 8:33 1 100M 0 part
└─sdc2 8:34 1 29.6G 0 part
sr0 11:0 1 1024M 0 rom

And (as root) runnint ‘parted’ on that device gives me

(parted) unit b
(parted) print
Model: SD SL16G (sd/mmc)
Disk /dev/mmcblk0: 15931539456B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
 1 1048576B 1529000447B 1527951872B primary fat32 lba

As highlighted above, the partition table is ‘msdos’ and the partition is shown as ‘fat32 lba’. Both of these are essential.

Here are some examples of a closer look at the MicroSD card using hexdump.

# sudo dd if=/dev/sdc of=/tmp/head bs=4K count=1
# hexdump -C /tmp/head
00000000 fa b8 00 10 8e d0 bc 00 b0 b8 00 00 8e d8 8e c0 |................|
00000010 fb be 00 7c bf 00 06 b9 00 02 f3 a4 ea 21 06 00 |...|.........!..|
00000020 00 be be 07 38 04 75 0b 83 c6 10 81 fe fe 07 75 |....8.u........u|
00000030 f3 eb 16 b4 02 b0 01 bb 00 7c b2 80 8a 74 01 8b |.........|...t..|
00000040 4c 02 cd 13 ea 00 7c 00 00 eb fe 00 00 00 00 00 |L.....|.........|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001b0 00 00 00 00 00 00 00 00 d4 5c 0b 00 00 00 00 00 |.........\......|
000001c0 01 20 0e 03 d0 ff 00 08 00 00 59 89 2d 00 00 03 |. ........Y.-...|
000001d0 d0 ff 05 03 d0 ff 59 91 2d 00 a7 3a ad 01 00 00 |......Y.-..:....|
000001e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

The thing I’ve highlighted is the byte at offset 0x1c2 into the card, here it says 0x0E. That is the code for a Fat16 LBA mapped partition table.

On another RPI, I have the following.

$ sudo dd if=/dev/sdc of=/tmp/header bs=4K count=1
$ hexdump -C /tmp/header
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001b0 00 00 00 00 00 00 00 00 99 56 4c 6b 00 00 80 20 |.........VLk... |
000001c0 21 00 0c eb 17 0c 00 08 00 00 00 20 03 00 00 eb |!.......... ....|
000001d0 17 0c 83 f9 71 05 00 28 03 00 00 d0 3c 00 00 00 |....q..(....<...|
000001e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

Both of these work well for me. If you don’t see one of these, look up the number you do see here [3]. If it is not a MS-DOS LBA-mapped partition type, you have a problem.

If you made your own SD card and you have the image file handy, you can inspect it to see what it has as follows.

Here is an image I built for myself.

$ ls -l raspbian.raw
-rw-rw-r-- 1 amrith amrith 2147483648 Jul 16 22:26 raspbian.raw

It is a 2GB Raspbian clone that I use for some of my machines. It contains two partitions, the output below shows offsets and sizes in Bytes because of the ‘unit b’ command.

$ sudo parted ./raspbian.raw unit b print
Model: (file)
Disk /home/amrith/images/raspbian.raw: 2147483648B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
 1 1048576B 105906175B 104857600B primary fat32 boot, lba
 2 105906176B 2146435071B 2040528896B primary ext4

Armed with that information, you can now do this.

$ sudo losetup --show -o 1048576 -f ./raspbian.raw
$ sudo losetup --show -o 105906176 -f ./raspbian.raw
$ mkdir /tmp/boot
$ mkdir /tmp/root
$ sudo mount /dev/loop0 /tmp/boot
$ sudo mount /dev/loop1 /tmp/root

/tmp/boot and /tmp/root now mount the MS-DOS and ext4 partitions in the image using loopback file systems. This is very useful in debugging. You can, of course, do the exact same thing on a MicroSD card and make corrections to the card if you so desire.

Making your own images

The next installment of this blog post will describe how you can build your own images.


[1] https://en.wikipedia.org/wiki/Master_boot_record
[2] https://github.com/raspberrypi/noobs/wiki/Standalone-partitioning-explained
[3] http://www.win.tue.nl/~aeb/partitions/partition_types-1.html

Blockchain is an over hyped technology solution looking for a problem

The article makes a very simple argument for something that I have felt for a while, block chain is a cool technology but the majority of the use cases people talk about are just bull shit.


Troubleshooting OpenStack gate issues with logstash

From time to time, I have to figure out why the Trove CI failed some job. By “from to time”, I really mean “constantly”, “all the time”, and “everyday”.

Very often the issue is some broken change that someone pushed up, easy ones are pep8 or pylint failures, slightly harder are the py27/py35 failures. The really hard ones are failures in the Trove scenario tests.

Very often the failures are transient and a recheck fixes them (which is annoying in itself) but sometimes the failure is repeatable.

In the past week, I’ve had to deal with one such issue; I first realized that it was a repeated failure after about a dozen rechecks of various changes.

I realized that the change had a telltale signature that looked like this:

Test Replication Instance Multi-Promote functionality.
Test promoting a replica to replica source (master).        SKIP: Failure in <function wait_for_delete_non_affinity_master at 0x7f1aafd53320>
Verify data is still on new master.                         SKIP: Failure in <function wait_for_delete_non_affinity_master at 0x7f1aafd53320>
Add data to new master to verify replication.               SKIP: Failure in <function wait_for_delete_non_affinity_master at 0x7f1aafd53320>
Verify data exists on new master.                           SKIP: Failure in <function wait_for_delete_non_affinity_master at 0x7f1aafd53320>

The important part of the message (I realized later) was the part that read:

Failure in <function wait_for_delete_non_affinity_master ...

Looking a bit above this, I found the test that had in fact failed

Wait for the non-affinity master to delete.                 FAIL

One thing important in this kind of debugging is to try and figure out when this failure really started to happen, and that’s one of the places where logstash comes in really handy.

For every single CI job run by the OpenStack integrated gate, the result artifacts are parsed and some of them are indexed in an elasticsearch database.

It is trivial now to pick up the string that I felt was the ‘signature’ and search for it in logstash. Within seconds I can tell that this error began to occur on 4/11.

This, by itself was not sufficient to figure out what the problem was, but once Matt Riedemann identified a probable cause, I was able to confirm that the problem started occurring shortly after that change merged.

Logstash is a really really powerful tool, give it a shot, you’ll find it very useful.