Reflections on the (first annual) OpenDev Conference, SFO

Earlier this week, I attended the OpenDev conference in San Francisco, CA.

The conference was focused on the emerging “edge computing” use cases for the cloud. This is an area that is of particular interest, not just from the obvious applicability to my ‘day job’ at Verizon, but also from the fact that it opens up an interesting new set of opportunities for distributed computing applications.

The highlight(s) of the show were two keynotes by M. Satyanarayanan of CMU. Both sessions were video taped and I’m hoping that the videos will be made available soon.

His team is working on some real cool stuff, and he showed off some of their work. The one that I found most fascinating, which most completely illustrates the value of edge computing is the augmented reality application to playing table tennis (which they call ping pong, and I know that annoys a lot of people :))

It was great to hear a user perspective presented by Andrew Mitry of Walmart. With 11,000 stores and an enormous (2mm??) employees, their edge computing use-case truly represents the scale at which these systems will have to operate, and the benefits that they can bring to the enterprise.

The conference sessions were very interesting and some of my key takeaways were that:

  • Edge Computing means different things to different people, because the term ‘Edge’ means different things to different applications. In some cases the edge device may be in a data center, in other cases in your houses, and in other cases on top of a lamp post at the end of your street.
  • A common API in orchestrating applications across the entirety of the cloud is very important, but different technologies may be better suited to each location in the cloud. There was a lot of discussion of the value (or lack thereof) of having OpenStack at the edge, and whether it made sense for edge devices to be orchestrated by OpenStack (or not).
  • I think an enormous amount of time was spent on debating whether or not OpenStack could be made to fit on a system with limited resources and I found this discussion to be rather tiring. After all, OpenStack runs fine on a little raspberry-pi and for a deployment where there will be relatively few OpenStack operations (instance, volume, security group creation, update, deletetion) the limited resources at the edge should be more than sufficient.
  • There are different use-cases for edge-computing and NFV/VNF are not the only ones, and while they may be the early movers into this space, they may be unrepresentative of the larger market opportunity presented by the edge.

There is a lot of activity going on in the edge computing space and many of the things we’re doing at Verizon fall into that category. There were several sessions that showcased some of the things that we have been doing, AT&T had a couple of sessions describing their initiatives in the space as well.

There was a very interesting discussion of the edge computing use-cases and the etherpad for that session can be found here.

Some others who attended the session also posted summaries on their blogs. This one from Chris Dent provides a good summary.

A conclusion/wrap-up session identified some clear follow-up activities. The etherpad for that session can be found here.

How do you answer this interview question, “what do you make in your current job?”

A couple of months ago, a former co-worker called me and asked if I would provide a reference for her in a job search (which I readily agreed to). Then she went on to ask me this, “This company wants to make me an offer and they called and asked me what I currently make, and asked for a copy of a paystub. What should I do?”

Personally, I find this question stupid. I’ve been asked it many times (including quite recently) and in all instances I’ve been surprised by it (doh!) and I’ve answered in what I now consider to be the wrong way.

Every hiring manager has a range of salaries that they are willing to pay for a position, and they have a range of bonuses, a range of stock options and other incentives. And then there’s the common incentives that everyone gets (401(k), vacation, …). So why even ask the question? Why not make an offer that makes sense and be done with it?

If you are a hiring manager / HR person on the hiring side, do you ask this question?

If you are a candidate, how do you handle this question?

In any event, here’s what I recommended to my friend, answer the question along these lines.

  • I’m sure you are asking me this so you can make me a competitive offer that I’ll accept
  • I’m also sure that you have a range for all the components of the offer that you intend to make to me; base pay, bonus, stock options, …
  • So what I’ll tell you is what I am looking for in an offer and I’ll leave it to you to make me an offer based on the standard ranges that you have
  • I am looking for a take-home pay of $ _____ each month
  • Since you offer a 401(k) plan which I intend to contribute $ _____ to, that means I am looking for a total base pay of $ ______ per year.
  • I am looking for a total annual compensation of $ ______ including bonuses
  • In addition, I am looking for ______ days of vacation each year.

That’s it. When asked for a copy of current pay-stub or anything like that, I recommend that you simply decline to provide it and make it clear that this is not any of their business.

Now, whether one can get away with this answer or not depends on how strong your position is for the opening in question. Some companies have a ‘policy’ that they need this paystub/W-2 stuff.

Not providing last pay information and following their ‘process’ could make the crabby HR person label you ‘not a team player’ or some such bogus thing and put your resume in the ‘special inbox’ which is marked ‘Basura’.

In any event, this all was fine and my friend told me  that she was given a good offer which she accepted.

How do you approach this question?

Running your IRC bouncer on a Raspberry Pi

This blog post is dedicated to Sean McGinnis, the PTL of the OpenStack Cinder project. If you are an OpenStack type, you should really follow him on Twitter and read his blog which he promises to spruce up.


IRC is the lifeblood of communication and collaboration in many open source ecosystems, like OpenStack.

One of the biggest issues in working in this environment is that IRC is inherently an asynchronous communications mechanism and if you aren’t on IRC, you aren’t plugged in.

A bouncer is a simple app that will keep a connection open to the IRC network and allow your device (phone, laptop, tablet) connect and disconnect at will, and easily catch up on what you missed.

There are many other ways to accomplish this, some leave an emacs session running on a machine and connect to it remotely. Whatever floats your boat.

But, you need a place to run this solution, I run mine on a Raspberry Pi.

To install and configure ZNC (my bouncer of choice) on my Raspberry Pi, I use this diskimage-builder element. (https://github.com/amrith/rpi-image-builder/tree/master/elements/rpi-znc)


Using this element is very straightforward, it is a dependency from rpi-generic and you can follow the simple image building tutorial here. Since I enable the log module on ZNC, I run my Raspberry Pi with a read-only MicroSD card and a writable root file system on a USB Flash Drive. Instructions for that setup are found here.


Once you boot the system with this image, you first configure znc.

Simple instructions on how to do this are in the element listed above (in the README file). Those steps are

sudo -u znc /usr/bin/znc –datadir=/var/lib/znc –makeconf

echo “PidFile = /var/run/znc/znc.pid” | sudo su \
– znc -s /bin/bash \
-c “tee -a /var/lib/znc/configs/znc.conf”

That’s it! You can now enable and launch the znc service. In my case, I configured ZNC to run with SSL on port 6697.

root@pi:/var/lib/znc# netstat -nl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp6 0 0 :::6697 :::* LISTEN

[…]

This means that I can get to the webadmin console simply by pointing my browser to

https://<IP ADDRESS>:6697

The only other thing that I do is to enable the log module which will persist IRC logs on the Raspberry Pi. I can search them later, I also setup logrotate to purge them periodically.

Running your Raspberry Pi on a conventional (or SSD) hard disk or a USB thumb drive

A Raspberry Pi (mine is an RPI 3) requires a MicroSD card to boot, but once booted it is not required any longer. This is great because while very convenient, a MicroSD card is not as robust and hardy as a regular SSD card or even a regular USB flash device.

One of the things that I therefore do is to run my RPI’s on either SSD or a USB thumb drive. I’ve also run mine with a 1TB external spinning rust disk with external power. The technique illustrated here works on all of these.

My earlier post described the RPI boot process. The picture here shows a simple MicroSD card image for an RPI. The disk is partitioned into two parts, the first partition is a small FAT32 LBA addressable partition and the second is a larger ext4 partition. The FAT32 partition contains the bootloader and the ext4 partition contains the root filesystem.

The thing that these two together is cmdline.txt which defines the root device with a declaration like:

root=/dev/mmcblk0p2 rootfstype=ext4

Since the RPI always mounts the MicroSD card as /dev/mmcblk0 and the partitions are numbered p1, p2, and so on, this indicates that the root partition is the ext4 partition as shown above.

To move this to a different location is a mere matter of adjusting cmdline.txt (and updating /etc/fstab) as shown below.

Here is my RPI with a USB thumb drive running the root (/) filesystem.

As you can see, I have a USB drive which shows up as /dev/sda and the MicroSD card.

amrith@rpi:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 115.7G 0 disk
└─sda1 8:1 1 115.7G 0 part /
mmcblk0 179:0 0 29G 0 disk
└─mmcblk0p1 179:1 0 100M 0 part /boot

amrith@rpi:~$ blkid
/dev/mmcblk0p1: LABEL="BOOT" UUID="E4F6-9E9D" TYPE="vfat" PARTUUID="01deb70e-01"
/dev/sda1: LABEL="root" UUID="7f4e0807-d745-4d6e-af6f-799d23a6450e" TYPE="ext4" PARTUUID="88578723-01"

I have changed cmdline.txt as shown below.

amrith@rpi:~$ more /boot/cmdline.txt
[...] root=/dev/sda1 rootfstype=ext4 [...]

and updated /etc/fstab (on the USB drive) as shown below.

amrith@rpi:~$ more /etc/fstab
# fstab generated by rpi-base element
proc /proc proc defaults 0 0
LABEL=BOOT /boot vfat defaults,ro 0 2
PARTUUID=88578723-01 / ext4 defaults,noatime 0 1

As you can see, I’ve also marked the MicroSD card (which provides /boot) to be readonly; in the unlikely event that I have to modify it, I can remount it without the ‘ro’ option and make any changes.

This is illustrated below

On the left hand side is the MicroSD card. Note that on the MicroSD card, in the FAT32 partition, cmdline.txt is in ‘/’ (there’s no /boot on the MicroSD card). the cmdline.txt points the root partition to /dev/sda1 which is the USB flash drive.

On the right hand side is the USB flash drive, it has an ext4 partition with an /etc/fstab entry which mounts the MicroSD card’s fat32 partition on /boot and mounts itself on /.

This works just as well with any external disk; just make sure that you have adequate power!