Running your IRC bouncer on a Raspberry Pi

This blog post is dedicated to Sean McGinnis, the PTL of the OpenStack Cinder project. If you are an OpenStack type, you should really follow him on Twitter and read his blog which he promises to spruce up.


IRC is the lifeblood of communication and collaboration in many open source ecosystems, like OpenStack.

One of the biggest issues in working in this environment is that IRC is inherently an asynchronous communications mechanism and if you aren’t on IRC, you aren’t plugged in.

A bouncer is a simple app that will keep a connection open to the IRC network and allow your device (phone, laptop, tablet) connect and disconnect at will, and easily catch up on what you missed.

There are many other ways to accomplish this, some leave an emacs session running on a machine and connect to it remotely. Whatever floats your boat.

But, you need a place to run this solution, I run mine on a Raspberry Pi.

To install and configure ZNC (my bouncer of choice) on my Raspberry Pi, I use this diskimage-builder element. (https://github.com/amrith/rpi-image-builder/tree/master/elements/rpi-znc)


Using this element is very straightforward, it is a dependency from rpi-generic and you can follow the simple image building tutorial here. Since I enable the log module on ZNC, I run my Raspberry Pi with a read-only MicroSD card and a writable root file system on a USB Flash Drive. Instructions for that setup are found here.


Once you boot the system with this image, you first configure znc.

Simple instructions on how to do this are in the element listed above (in the README file). Those steps are

sudo -u znc /usr/bin/znc –datadir=/var/lib/znc –makeconf

echo “PidFile = /var/run/znc/znc.pid” | sudo su \
– znc -s /bin/bash \
-c “tee -a /var/lib/znc/configs/znc.conf”

That’s it! You can now enable and launch the znc service. In my case, I configured ZNC to run with SSL on port 6697.

root@pi:/var/lib/znc# netstat -nl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp6 0 0 :::6697 :::* LISTEN

[…]

This means that I can get to the webadmin console simply by pointing my browser to

https://<IP ADDRESS>:6697

The only other thing that I do is to enable the log module which will persist IRC logs on the Raspberry Pi. I can search them later, I also setup logrotate to purge them periodically.

Running your Raspberry Pi on a conventional (or SSD) hard disk or a USB thumb drive

A Raspberry Pi (mine is an RPI 3) requires a MicroSD card to boot, but once booted it is not required any longer. This is great because while very convenient, a MicroSD card is not as robust and hardy as a regular SSD card or even a regular USB flash device.

One of the things that I therefore do is to run my RPI’s on either SSD or a USB thumb drive. I’ve also run mine with a 1TB external spinning rust disk with external power. The technique illustrated here works on all of these.

My earlier post described the RPI boot process. The picture here shows a simple MicroSD card image for an RPI. The disk is partitioned into two parts, the first partition is a small FAT32 LBA addressable partition and the second is a larger ext4 partition. The FAT32 partition contains the bootloader and the ext4 partition contains the root filesystem.

The thing that these two together is cmdline.txt which defines the root device with a declaration like:

root=/dev/mmcblk0p2 rootfstype=ext4

Since the RPI always mounts the MicroSD card as /dev/mmcblk0 and the partitions are numbered p1, p2, and so on, this indicates that the root partition is the ext4 partition as shown above.

To move this to a different location is a mere matter of adjusting cmdline.txt (and updating /etc/fstab) as shown below.

Here is my RPI with a USB thumb drive running the root (/) filesystem.

As you can see, I have a USB drive which shows up as /dev/sda and the MicroSD card.

amrith@rpi:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 115.7G 0 disk
└─sda1 8:1 1 115.7G 0 part /
mmcblk0 179:0 0 29G 0 disk
└─mmcblk0p1 179:1 0 100M 0 part /boot

amrith@rpi:~$ blkid
/dev/mmcblk0p1: LABEL="BOOT" UUID="E4F6-9E9D" TYPE="vfat" PARTUUID="01deb70e-01"
/dev/sda1: LABEL="root" UUID="7f4e0807-d745-4d6e-af6f-799d23a6450e" TYPE="ext4" PARTUUID="88578723-01"

I have changed cmdline.txt as shown below.

amrith@rpi:~$ more /boot/cmdline.txt
[...] root=/dev/sda1 rootfstype=ext4 [...]

and updated /etc/fstab (on the USB drive) as shown below.

amrith@rpi:~$ more /etc/fstab
# fstab generated by rpi-base element
proc /proc proc defaults 0 0
LABEL=BOOT /boot vfat defaults,ro 0 2
PARTUUID=88578723-01 / ext4 defaults,noatime 0 1

As you can see, I’ve also marked the MicroSD card (which provides /boot) to be readonly; in the unlikely event that I have to modify it, I can remount it without the ‘ro’ option and make any changes.

This is illustrated below

On the left hand side is the MicroSD card. Note that on the MicroSD card, in the FAT32 partition, cmdline.txt is in ‘/’ (there’s no /boot on the MicroSD card). the cmdline.txt points the root partition to /dev/sda1 which is the USB flash drive.

On the right hand side is the USB flash drive, it has an ext4 partition with an /etc/fstab entry which mounts the MicroSD card’s fat32 partition on /boot and mounts itself on /.

This works just as well with any external disk; just make sure that you have adequate power!

Everything you wanted to know about Raspberry Pi (RPI) images – Part 2

In Part 1 of this blog post I described the Raspberry Pi boot image structure (a very simple version) and provided a very brief description of how the boot process works.

This part talks about how you can build your own images for a Raspberry Pi and setups a framework for people to collaborate and share their image components with each other.

In subsequent blogs I’ll build on the concepts here and describe other more sophisticated configurations and progressively share more of the elements I have built for myself, cleaning them up, before I make them public.

This blog post is structured into three parts:

  • Building your own images
  • Writing your own elements
  • Collaborating with elements

I build my own Raspberry Pi boot images from scratch. I do this so I can build images in a repeatable way, configuring the software I want in just the way I want it.

It is much more effective to do this, than to take an image from some other source and then remove things which I don’t want and work around the quirks of the person who build that image to begin with.

In doing this, I can control exactly what goes on the image and make sure that it comes out the exact same way each time.

My tool of choice (as I use this for work related things as well) is diskimage-builder. Like other image building tools, this one is quirky, but I’ve got to understand it and I prefer it to the pi-gen tool that is used to build the official Raspbian images.

Building your own images

This will work for you if you have any old Linux machine (I use an Ubuntu 16.04LTS x86_64 machine). A simple to follow Quick Start Guide is available here.

To a reasonably Linux savvy person, building your first image should take no more than 20 minutes and you can have your Raspberry Pi up and running on your own custom image in about 30 minutes.

The steps are simple:

  • Clone the repository of tools and elements
  • Install prerequisite software
  • Build your image
  • Install your image
  • Have fun!

Writing your own elements

diskimage-builder is a very flexible tool and it is very easy to write elements once you understand the general flow of things. Like all image building tools, it uses two environments, a host environment and an image environment. Some tools make the image environment a chroot (like diskimage-builder) others use a VM or a docker container.

diskimage-builder works by allowing users to select elements, define dependencies between elements, and execute scripts provided in those elements in a deterministic order. Some of the scripts are executed in the host environment, others are executed in the image environment. Finally, it takes the image environment that was created and packages it up into the format of your choice.

I use diskimage-builder to make images for a cloud (and I use qcow2 for that). For Raspberry Pi, I use the ‘raw’ format which is nothing more than a disk image complete with MBR (boot record), partition table, and volumes.

To write your own element, you can follow along with one of the many elements provided and insert your own commands. It is a very short, and not very steep learning curve.

Collaborating with elements

I’ve shared (and I will continue to share) elements that build a basic Raspbian based image, a basic Debian based image, and elements that will install specific packages, configure WiFi, configure a secure user, and so on.

For example, consider the element rpi-generic which produces a Raspberry Pi image that I use as a basic building block for many of my applications. It does not, by itself, provide an operating system. So, I could choose either rpi-debian-core or rpi-raspbian-core. With just those two elements, I could execute the command

disk-image-create rpi-debian-core rpi-generic \
-a armhf -o debian -t raw -n

and this would produce a file (debian.raw) which was a properly bootable Raspberry Pi image.

But, since rpi-generic depends on other elements the diskimage-builder tool looks for those and uses them. One of the elements it depends on is rpi-resize-root. Let’s look at that for a second.

The MicroSD card I use is often 8 or 16GB, the image (for speed) is just 2GB. So, when you install the image onto the MicroSD card, and you boot the machine for the first time, the software installed and configured by the rpi-resize-root element expands the root file system to take the full available space.

Similarly, rpi-generic depends on rpi-wifi. That element does the few things to configure my Raspberry Pi to automatically connect to the WiFi network.

If someone else wrote an element that I wanted to use, all I would have to do is get that element (clone the repository that they shared) and tell diskimage-builder where to find the element!

For example, the sample build.bash script contains this line, which can easily be extended like a standard Linux PATH variable!

As a practical matter, I have an element for my WiFi access point that I use at home (which I will share as soon as I can clean it up). That element is on my machine at /rpi-image-builder-private/elements/rpi-wifi-access-point.

The elements which I’ve shared on the rpi-image-builder repository are at /rpi-image-builder/elements/.

To build my WiFi access point, I specify:

export ELEMENTS_PATH=/rpi-image-builder-private/elements:/rpi-image-builder/elements

I encourage you to write and share elements, and I believe that this will be a great way for all Raspberry Pi users to collaborate on their cool projects.

Everything you wanted to know about Raspberry Pi (RPI) images

This is a two part blog post that covers a lot of topics about a rather boring aspect of the RPI. I suspect that the vast majority of people using a RPI wouldn’t be interested in most of this stuff. But, there are a small number of people out there (like myself) who will no doubt find this stuff very useful.

This (the first of two) posts covers the booting process of an RPI. You need to have some idea of how this works to make your own images.

I have a RPI 3, some of this stuff may not be accurate for other models.

RPI Booting demystified

I have not been able to find any documentation on the bootloader that runs in the firmware but by trial and error, here is what I’ve found.

The firmware accesses the partition table on the MicroSD card which it assumes is an MS-DOS style partition table. The partition table is at the very beginning of the card and is part of the Master Boot Record (MBR). A good write up about this layout is found here [1].

The bootloader looks (it appears) for an MS-DOS LBA addressable partition. I have found that partition types 0x0C and 0x0E work. In order to boot reliably, I have also found that this must be the first partition on the disk.

RPI’s don’t use bootstrap code on the MicroSD card.

This is very important to if you wish to make your own images; it makes that process a whole lot simpler. The bootloader in firmware looks instead at this small MS-DOS partition to find a kernel, all the drivers it needs, and a pointer to the root file system.

The bootloader parses the cmdline.txt file to find the root partition. Here is a small portion of the cmdline.txt on one of my RPIs.

root=/dev/mmcblk0p2 rootfstype=ext4

The RPI addresses the MicroSD as /dev/mmcblk0 and the partitions on it are (in order) addressed as /dev/mmcblk0p1, /dev/mmcblk0p2, and so on.

The cmdline.txt file identifies the root partition either by providing this name, or by specifying a partition UUID, or a label. In the example above, cmdline.txt points to the second partition on the MicroSD card.

I found exactly one place that appears to document this in some detail, that is here[2].

Troubleshooting the boot process

When you power on the RPI, the red light indicates power. The green light (ACT/D5) indicates SD card access. If the red light comes on but the green one doesn’t it means that the firmware did not find a card it liked. This has been the most common problem described by people online.

If you have the tools to debug it, here are some things that I have found to be helpful.

If you can put this MicroSD card into a machine where you can mount it (and hopefully that machine is Linux based; if it is Windows based, my apologies, I can’t help you).

lsblk will help you identify the device. For example on my machine it is /dev/sdc. Here is the output of lsblk on my machine. The MicroSD card is shown in bold letters.

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 150G 0 disk
├─sda1 8:1 0 134G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 16G 0 part [SWAP]
sdc 8:32 1 29.7G 0 disk
├─sdc1 8:33 1 100M 0 part
└─sdc2 8:34 1 29.6G 0 part
sr0 11:0 1 1024M 0 rom

And (as root) runnint ‘parted’ on that device gives me

(parted) unit b
(parted) print
Model: SD SL16G (sd/mmc)
Disk /dev/mmcblk0: 15931539456B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
 1 1048576B 1529000447B 1527951872B primary fat32 lba
[...]

As highlighted above, the partition table is ‘msdos’ and the partition is shown as ‘fat32 lba’. Both of these are essential.

Here are some examples of a closer look at the MicroSD card using hexdump.

# sudo dd if=/dev/sdc of=/tmp/head bs=4K count=1
# hexdump -C /tmp/head
00000000 fa b8 00 10 8e d0 bc 00 b0 b8 00 00 8e d8 8e c0 |................|
00000010 fb be 00 7c bf 00 06 b9 00 02 f3 a4 ea 21 06 00 |...|.........!..|
00000020 00 be be 07 38 04 75 0b 83 c6 10 81 fe fe 07 75 |....8.u........u|
00000030 f3 eb 16 b4 02 b0 01 bb 00 7c b2 80 8a 74 01 8b |.........|...t..|
00000040 4c 02 cd 13 ea 00 7c 00 00 eb fe 00 00 00 00 00 |L.....|.........|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001b0 00 00 00 00 00 00 00 00 d4 5c 0b 00 00 00 00 00 |.........\......|
000001c0 01 20 0e 03 d0 ff 00 08 00 00 59 89 2d 00 00 03 |. ........Y.-...|
000001d0 d0 ff 05 03 d0 ff 59 91 2d 00 a7 3a ad 01 00 00 |......Y.-..:....|
000001e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
[...]

The thing I’ve highlighted is the byte at offset 0x1c2 into the card, here it says 0x0E. That is the code for a Fat16 LBA mapped partition table.

On another RPI, I have the following.

$ sudo dd if=/dev/sdc of=/tmp/header bs=4K count=1
$ hexdump -C /tmp/header
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001b0 00 00 00 00 00 00 00 00 99 56 4c 6b 00 00 80 20 |.........VLk... |
000001c0 21 00 0c eb 17 0c 00 08 00 00 00 20 03 00 00 eb |!.......... ....|
000001d0 17 0c 83 f9 71 05 00 28 03 00 00 d0 3c 00 00 00 |....q..(....<...|
000001e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000

Both of these work well for me. If you don’t see one of these, look up the number you do see here [3]. If it is not a MS-DOS LBA-mapped partition type, you have a problem.

If you made your own SD card and you have the image file handy, you can inspect it to see what it has as follows.

Here is an image I built for myself.

$ ls -l raspbian.raw
-rw-rw-r-- 1 amrith amrith 2147483648 Jul 16 22:26 raspbian.raw

It is a 2GB Raspbian clone that I use for some of my machines. It contains two partitions, the output below shows offsets and sizes in Bytes because of the ‘unit b’ command.

$ sudo parted ./raspbian.raw unit b print
Model: (file)
Disk /home/amrith/images/raspbian.raw: 2147483648B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
 1 1048576B 105906175B 104857600B primary fat32 boot, lba
 2 105906176B 2146435071B 2040528896B primary ext4

Armed with that information, you can now do this.

$ sudo losetup --show -o 1048576 -f ./raspbian.raw
/dev/loop0
$ sudo losetup --show -o 105906176 -f ./raspbian.raw
/dev/loop1
$ mkdir /tmp/boot
$ mkdir /tmp/root
$ sudo mount /dev/loop0 /tmp/boot
$ sudo mount /dev/loop1 /tmp/root

/tmp/boot and /tmp/root now mount the MS-DOS and ext4 partitions in the image using loopback file systems. This is very useful in debugging. You can, of course, do the exact same thing on a MicroSD card and make corrections to the card if you so desire.

Making your own images

The next installment of this blog post will describe how you can build your own images.

References

[1] https://en.wikipedia.org/wiki/Master_boot_record
[2] https://github.com/raspberrypi/noobs/wiki/Standalone-partitioning-explained
[3] http://www.win.tue.nl/~aeb/partitions/partition_types-1.html

Raspberry Pi networking projects

The Raspberry Pi 3 that I have comes standard with two network interfaces; a wired interface that can do 100mbps and a WiFi interface. Older Raspberry Pi’s required that you use a USB dongle for WiFi, I don’t use those units any longer.

So for the purposes of all that follows, I assume Raspberry Pi 3, onboard WiFi and wired ethernet.

By default, these two interfaces are active and software that you run on the Raspberry Pi can connect to the outside world using one, or both.

I’ve found several interesting use-cases for the Raspberry Pi by changing the way these interfaces are configured.

  1. A WiFi satellite location

    piwifi1In this image, three devices (device 1, 2 and 3) are not WiFi enabled and are internet connected using the Raspberry Pi as effectively a wireless network extender.

    This setup is relatively straightforward on a Raspberry Pi.

  • Configure the wireless interface on the Raspberry Pi to connect to the wireless access point.
  • Enable ip forwarding
  • Configure dnsmasq
  • Enable packet forwarding from the wlan0 and eth0 interfaces

    With this setup, the three devices connected to the wired interface will get their DHCP leases from the Raspberry Pi. Packets will get forwarded by the Raspberry Pi between the wired and wireless interfaces.

  1. A WiFi satellite location without DHCP

    piwifi2The above configuration is very useful for some things but not always. I have a printer (quite old) which I have connected to a single parallel port ethernet print server (TP-Link TL-PS110P). I need to be able to access this printer from other wirelessly connected devices and so I need it to have its DHCP lease coming from the WiFi Access Point!

    This setup is similar to (1) above, but no dnsmasq, no NAT, enable proxy ARP.

  1. The Raspberry Pi as a WiFi access point

    piwifi3This is something I’ve just been playing with recently and it appears to work quite well. The Raspberry Pi 3’s WiFi interface can be configured to act as an access point using the hostapd package. The way I have this setup, dnsmasq is enabled and the wirelessly connected devices receive DHCP leases from the Raspberry Pi. Traffic is routed to the internet over the wired interface.

  2. The Raspberry Pi as a secure WiFi access point

    Eventually, this is what I want to get, a Raspberry Pi as a secure WiFi access point; the WiFi interface running in access point mode but all traffic going out of the wired interface is tunneled to a VPN.

    I use OpenVPN, and that works fine on the Raspberry Pi already. Have to put the pieces together and make it a bit more robust; right now, not quite there.

    Equally interesting would be the other configuration like (1) above but where all traffic out of the WiFi interface is tunneled. In that setup, I could, for example connect my laptop to the wired interface and connect to any WiFi access point on the Raspberry Pi. Traffic over the WiFi interface would be tunneled by the Raspberry Pi and this would be an ideal travel setup as the Raspberry Pi would just be powered off the USB port on the laptop 🙂

Raspberry Pi basics

I have been using a Raspberry Pi (I’ve bought a few of these on Amazon, at $50 each, they are a bargain) for some time now and have found them to be excellent for a number of things.

A recent project to set one up as a WiFi access point got me thinking that I should, maybe, share some of these use-cases.

So, here’s a primer on how I setup the Raspberry Pi. I have a new one on order right now so this is an actual first-time setup.

If you have never done this before, don’t worry, it is very simple.

Assembling your Raspberry Pi

New and out of the box, the only things that you have to do are

  • Figure out how to affix the heat-sink on the processor; I always use the largest one that they provide. Do this once, do it carefully and you will have no issues later
  • Figure out how to get the board neatly into the nice clear plastic case.

Formatting your SD card

I don’t purchase the “complete kit” which comes with a Micro-SD card. I usually have a card or three hanging around and set it up using NOOBS (That’s Raspberry Pi’s New Out Of Box Software).

Since I setup the card on a Windows machine, there is one thing I’d like to highlight. The documentation makes it sound hard, they have you download some special format utility and all that stuff. Don’t bother.

Just follow the easy instructions found here.

  1. Launch the disk management utility.

My new 16GB disk drive is the one that shows up as Disk 1.

This was a brand new SD card, if you are reusing an SD card, you may see multiple partitions, delete them all.

If you find that the “Delete Volume” options are greyed out, you will have to use the Windows Command Line. Use the diskpart utility, select the disk, then select each partition in turn and delete it. You will be left with a disk that looks like this.

Observe that now Disk 1 is shown as “Unallocated”. I always make sure I get here and format the disk.

  1. Format the disk
    You do this by simply right clicking on the “Unallocated” disk and choosing “Format”. Be careful to choose FAT32.
  1. Copy NOOBS onto the new SD card
    Download the latest NOOBS zip file and unzip it. Then just drag and drop the whole thing onto your new SD card. Safely eject the SD card, make sure the power is disconnected from the Raspberry Pi and plug the card into the slot. Then … the moment of truth.
  2. Power up the Raspberry Pi for the first time
    If you did everything correctly, you should see a NOOBS screen that comes up and allows you to choose the operating system. I usually enable WiFi at this point (or if the wired network is connected, that works too) , and then I follow the standard NOOBS documentation, setup Raspbian with PIXEL, and then reboot.
  3. On first boot, I enable the SSH server, set the locale, timezone and things like that and from that point the rest of the setup is done from command line.

That’s really all there is to your first time Raspberry Pi setup!