OddBloke's Blog

Last week was entirely spent at an internal engineering sprint in Paris. We discussed a wide variety of things, which will be reflected over the next few weeks and months of work.

I keep notes of the work that I do as a member of Canonical's Ubuntu Server team. This post is a breakdown of the highlights of what I spent my last week doing.


  • Landed MP 372491 to give us additional debugging for an ongoing bug investigation


  • No substantial curtin work this week


  • Code reviews (all for Chad Smith):
    • Reviewed and landed #733
    • Reviewed and landed #738
    • Reviewed #742


  • Rotating Monday triage for me this week, plus regular triage on Tuesday
  • Visiting family in the UK before our sprint the following week, so less work days than usual (plus packing for sprint etc.)
  • Submitted #1034 to multipass, requesting some functionality that would make testing cloud-init using multipass much easier

I keep notes of the work that I do as a member of Canonical's Ubuntu Server team. This post is a breakdown of the highlights of what I spent my last week doing.


  • Validation of the SRU (Stable Release Update) of cloud-init back into xenial, bionic and disco
    • Performed manual boot verification on Oracle Cloud Infrastructure
      • This resulted in the discovery of a minor regression on Oracle, bug 1842752
      • This bug is cosmetic, and causes boot to slow down by <0.1s
      • Under normal circumstances, we would address even this, but Ubuntu images in Oracle Cloud Infrastructure will be migrating to the Oracle data source in the near future, so the impact of this will be substantially limited
    • Wrote the verification script and bug content for bug 1812857
      • This took the best part of a day because I ran into a number of roadblocks; I don't have an OpenStack I can modify network config on, performing validation in a VM would require crafting OpenStack network_data.json for the issue, and bond setup inside xenial containers wasn't working for me
      • Excerpt from my notes:
        • Figured I could probably do better in a VM with a ConfigDrive attached
          • I couldn't
  • Code review


  • No substantial curtin work this week


  • No substantial UA client work this week


  • Back after a week and a day off, so spent the early part of the week just catching up on email etc.
    • Footage of me destroying my email backlog: Footage of me destroying my email backlog
    • (I was at this show during my time off!)
  • Submitted several talk proposals for PyCon Canada 2019, fingers crossed!

I keep notes of the work that I do as a member of Canonical's Ubuntu Server team. This post is a breakdown of the highlights of what I spent my last week doing.


  • Landed MP 371461, a minor doc improvement
  • Landed MP 371403, which allows secondary VNICs in Oracle Cloud Infrastructure to be configured by cloud-init!
    • This only works on Virtual Machines (not Bare Metal Machines)
    • At the moment, this requires configuration in your image/instance to enable
  • Substantial internal discussions about the future of initramfs network configuration in cloud-init
    • To summarise: nothing needs to change, but we would like to drop the “initramfs” NetworkConfigSource and merge its behaviour into the “fallback” NetworkConfigSource
    • (On an iSCSI-root system, for example, the last-resort, fallback network config should be what the initramfs wrote out, not DHCP-on-first-interface.)
  • Completed my inital dig into Dracut network configuration (my notes are here)
  • Landed MP 371673 with the initial refactoring required to introduce Dracut network configuration parsing
    • This doesn't change any behaviour, it just moves klibc from being the source of initramfs network configuration to being a source of initramfs network configuration (albeit the only source, for now)
  • Revisited bug 1834875 to confirm that this issue doesn't lie within cloud-init
  • Code review
  • Landed MP 371350 (see last week for details)
  • Submitted MP 371683 with minor .gitignore updates


  • Dropped building of Python 2 Ubuntu package (python-curtin) in a new upload (19.2-6-g88a1a7ec-0ubuntu1
    • Thanks once again to Ryan Harper for sponsoring the upload
    • This required a minor change upstream (MP 371589) so that the Python 3 tests could be run without the Python 2 tests


  • No substantial UA client work this week


  • Uploaded a new upstream snapshot of simplestreams to eoan (0.1.0-25-gba75825b-0ubuntu1)
    • Thanks to Robie Basak for the sponsorship!
  • Spent some time looking at dropping Python 2 from the simplestreams Ubuntu packaging
    • I have branches for this locally, but we have downstream consumers that depend on the Python 2 packages still, so we'll do it next cycle
  • Investigated and filed a Launchpad bug that was causing our CI and autolander triggers to fail
  • On vacation next week, so won't be back to work until after International Worker's Day (North American Edition)

Note: This blog post is essentially notes from some investigation into dracut in the context of supporting its network configuration in cloud-init. Its intended audience is other cloud-init developers. If you're looking for an accessible introduction to dracut, this isn't it, I'm afraid!

I'm currently trying to work out how we should proceed with the parsing of dracut network configuration, to have feature parity with Ubuntu/initramfs-tools. This blog post is partly reporting on what I've found, and partly starting a discussion around how we should proceed.


All of this was run on an Oracle Linux 7.6 Virtual Machine running in Oracle Cloud Infrastructure. The version of dracut:

$ yum list installed | grep dracut.x
dracut.x86_64             033-554.0.3.el7        @anaconda/7.6  


This is what dracut produces in /run/initramfs:

$ tree /run/initramfs/
├── log
├── net.00:00:17:02:3f:34.did-setup
├── net.00:00:17:02:3f:34.up
├── net.ens3.dhcpopts
├── net.ens3.did-setup
├── net.ens3.gw
├── net.ens3.lease
├── net.ens3.pid
├── net.ens3.resolv.conf
├── net.ens3.up
├── net.ifaces
├── rwtab
└── state
    ├── etc
    │   ├── resolv.conf
    │   └── sysconfig
    │       └── network-scripts
    │           └── ifcfg-ens3
    └── var
        └── lib
            └── dhclient
                └── dhclient-b560099e-e294-4563-bc2f-fe800312c96b-ens3.lease

Note that 00:00:17:02:3f:34 is the MAC address of ens3. Let's run through the top-level entries quickly:

  • log is an empty directory
  • The *.did-setup files are used internally by dracut to track status (it literally means dracut “did setup”)
  • The *.up files indicate the interface is up
  • net.ens3.lease is a dhclient lease file
  • net.ens3.dhcpopts is a shell-sourceable file containing a bunch of DHCP options with new_ prepended (e.g. new_broadcast_address, new_classless_static_routes, new_interface_mtu)
    • These look to be exactly the same values as in net.ens3.lease
  • net.ens3.gw contains an ip command to configure a gateway
    • On this particular instance, it's ip route replace default via dev ens3
  • net.ens3.pid is a PID file
    • Examining the dracut code, this is a dhclient PID file, and that same dhclient invocation used net.ens3.lease as the lease file
  • net.ens3.resolv.conf: nameserver
  • net.ifaces contains just ens3
    • Looking at the dracut code, this is a list of interfaces (with just one entry for our case)
  • rwtab contains two lines, files /etc/sysconfig/network-scripts and files /var/lib/dhclient
    • I believe this is configuration for the read-only root that (I infer) dracut boots with, pulling those two directories into its writable scratch space

Now let's consider the state/ entries:

  • etc/resolv.conf is the same as net.ens3.resolv.conf
  • etc/sysconfig/network-scripts/ifcfg-ens3 is, as you might anticipate, a sysconfig file with network configuration for ens3; we'll break that down more below
  • var/lib/dhclient/dhclient-b560099e-e294-4563-bc2f-fe800312c96b-ens3.lease is identical to net.ens3.lease

The most interesting file of all of the above (from our perspective) is the sysconfig network configuration that dracut writes out (its full path is /run/initramfs/state/etc/sysconfig/network-scripts/ifcfg-ens3). The full content of the file is:

# Generated by dracut initrd

This file is identical to /etc/sysconfig/network-scripts/ifcfg-ens3 in the booted system (including the first, commented line) except for a bunch of repeated NM_CONTROLLED=no lines.

I believe this contains all the information we would need to emit the appropriate cloud-init configuration (i.e. its name and the fact that it's a DHCP interface).

Path Forward

There are two potential paths forward:

  1. implement generic sysconfig network-scripts parsing (that could be used with net-convert, for example) and use that in cloud-init's dracut initramfs code, or

  2. implement parsing in cloud-init's dracut initramfs code which would handle only the dracut network configuration that we see generated.

I am leaning towards (2), implementing the parsing within the dracut code path. My feeling is that a generic parser is overkill for what we actually need to achieve here. If we keep the code contained within the dracut code path, then we can flesh it just as much as we need for the (probably) fairly simple network configurations that dracut can generate, without it seeming deficient when compared to the other network config parsers.

(Perhaps I'm being too conservative, and we can just implement a limited “generic” parser with the understanding that any future uses of it will have to expand it substantially.)

One important note: if implemented as part of the dracut code, the network-scripts parsing code should be structured so that it can easily be used as the basis for a more generic parser in future.

What do people think?

I keep notes of the work that I do as a member of Canonical's Ubuntu Server team. This post is a breakdown of the highlights of what I spent my last week doing.


  • Landed MP 371203 to fix up some confusing variable names
  • Continued conversations about cloud-init logging to the journal
  • Landed MP 371053 to configure secondary VNICs in Oracle Cloud Infrastructure
    • This needs some follow-up work to actually be enabled; we're working out the best path forward later today
  • Submitted MP 371350 to add a GitHub pull request template to inform people that we don't use GitHub PRs for development
    • Thanks to GitHub user @max06 for this suggestion!
  • Spent some time investigating how to handle iSCSI root network configuration on systems that use dracut for their initramfs


  • Uploaded a new upstream snapshot to eoan, the Ubuntu development release
    • Thanks to Ryan Harper for sponsoring the upload!


  • Completed transition to black formatting (#705, #708, #713, #714, #717)
  • Converted a bunch of %-formatting to use .format() instead (#707, #716)
  • Fixed breaking package builds (#709)
  • Promoted mypy failures to fail the build in Travis (#706)
    • This should really have been done a while ago, it was only non-voting so that I could land the beginnings of the mypy work without breaking the build
  • Code review of #688 and #690
  • Filed #715 to capture some internal discussion about improving progress reporting in the client


  • Over the weekend, I wrote a blog post, Using multipass To Create A WriteFreely Dev VM
    • I ran into a few issues with multipass that I filed: #953, #954, #955, #957, #958
    • I need to make some changes to the blog post to reflect some of the responses to those issues, in particular switching to multipass shell to avoid #954
  • cloud-init/curtin triage on Monday/Tuesday
  • Reinstalled my laptop with encrypted root (so I can use it to work outside of the house)
  • Switched from using Synergy to using barrier
    • During setup, I accidentally configured both user accounts on my secondary machine to connect to my primary machine and discovered that this causes a memory leak

I want to get a list of all the One Piece episodes to date (because I want to add them to Todoist, which is probably a bad idea that I will immediately regret and undo).


The Script

The output I'm looking for is <episode number>: <episode title> (<air date>) for each regular episode (i.e. I don't want specials), and I want to emit the episodes ordered by airdate.

import csv
import datetime
import re
import sys

def output_lines(f):
    reader = csv.DictReader(f)

    rows = []
    for row in reader:
        if row["number"].startswith("S"):
            # Skip specials
        row["airdatetime"] = datetime.datetime.strptime(
            row["airdate"], "%d %b %y"
        if "https" in row["title"]:
            # There are some misquoted lines in the CSV, fix those up
            # here
            row["title"] = row["title"].split(",")[0].strip('"')
        # Remove " (25 min)" when present
        row['title'] = re.sub(r' \(\d+ min\)', '', row['title'])
    for line in sorted(rows, key=lambda row: row["airdatetime"]):
            print(f"{line['number']}: {line['title']} ({line['airdate']})")
        except BrokenPipeError:

if __name__ == "__main__":
    with open("onepiece.csv") as f:

And that gives:

$ python3 onepiece.py | head
1: I'm Luffy! The Man Who's Gonna Be King of the Pirates! (20 Oct 99)
2: Enter the Great Swordsman! Pirate Hunter Roronoa Zoro! (17 Nov 99)
3: Morgan versus Luffy! Who's the Mysterious Pretty Girl? (24 Nov 99)
4: Luffy's Past! Enter Red-Haired Shanks! (08 Dec 99)
5: A Terrifying Mysterious Power! Captain Buggy, the Clown Pirate! (15 Dec 99)
6: Desperate Situation! Beast Tamer Mohji vs. Luffy! (29 Dec 99)
7: Epic Showdown! Swordsman Zoro vs. Acrobat Cabaji! (29 Dec 99)
8: Who is the Victor? Devil Fruit Power Showdown! (29 Dec 99)
9: The Honorable Liar? Captain Usopp! (12 Jan 00)
10: The Weirdest Guy Ever! Jango the Hypnotist! (19 Jan 00)


A couple of things I hit semi-regularly tripped me up here:

  • csv.DictReader requires the file it's operating on to be kept open
    • i.e. it, sensibly, doesn't load everything into memory on instantiation
  • the order of parameters to datetime.strptime, which I screw up almost every time I use it
    • This is definitely my fault, because the error message is very clear: ValueError: time data '%d %b %y' does not match format '20 Oct 99'
    • That said, I would generally expect callables to take their most-likely-to-be-static parameters first; this subversion of my expectations is probably why I screw it up every time

There was one thing I hit, as I recall, for the first time ever: BrokenPipeError. I hit this when piping the script's output to head to paste in above, because head closes the pipe once it has all the lines it needs. I'm not really sure why I've never hit this before, because I've certainly piped Python script output before.

I've pushed this script (and the output it produces) to my blog-extras repo.

If you've ended up at this blog post, you're probably like me: you've found a change you'd like to make to WriteFreely, but you can't (or don't want to) install all of its development dependencies on your machine. This guide will show you how to use multipass to set up a WriteFreely development VM so that you don't need to make changes to your host.

Note: If you run into any problems with this guide, feel free to reach out to me on the fediverse, I'm Odd_Bloke@wrestle.town. I can't promise I'll be able to help, but I will at the very least be sympathetic.

Updated (2019/08/19): This guide was updated to use multipass shell <vm> instead of multipass exec <vm> /bin/bash. This is more idiomatic, and avoids some known bugs.


  • A system with multipass installed
    • If you're on a system with snapd (e.g. Ubuntu), this is as simple as sudo snap install --classic --beta multipass
      • For reference, I'm using the Ubuntu development release with the multipass beta snap (v0.8.0).
    • multipass is cross-platform, though, so hopefully you'll be able to follow this guide on Windows or macOS too! You can find installers for these platforms on multipass' GitHub releases page. (Let me know if you hit problems!)
  • Enough RAM to give a VM 4GB of it
    • multipass' default is 1GB, and I ran out of memory during the installation of WriteFreely's Go dependencies when I tried 2GB as well.
    • It's certainly possible you could tune this down some. If you do so successfully, please let me know so I can update this guide!

That's it! We're going to do everything else inside our VM.

Creating our development VM

This is pretty simple:

$ multipass launch bionic --name writefreely-dev --mem 4G

This is going to fetch the latest Ubuntu 18.04 LTS (Bionic Beaver) image and create a VM named writefreely-dev from it, giving that VM 4GB of memory. (If you want to name your VM something different, go ahead! But: don't forget to use that name in later steps.)

(This step can sometimes take a while, if the Ubuntu data centre is under heavy network load (which it often is!). Don't worry if the progress is slow because that is, unfortunately, normal.)

Installing prerequisites in our development VM

First, let's get a shell in our development VM:

$ multipass shell writefreely-dev

<... long MOTD content ...>


Again, nice and simple!

OK, so now we can go and look at the Prerequisites section of the WriteFreely Development Setup guide and see what we need:

  • Go 1.10+
  • Node.js

OK, so let's do these one at a time.

Installing Go 1.10+

We're going to use the go snap to install Go:

$ sudo snap install --classic go
go 1.12.9 from Michael Hudson-Doyle (mwhudson) installed


$ go version
go version go1.12.9 linux/amd64

Great, nailed it! On to the next thing.

Installing node.js

From my testing so far, node.js 8.x is sufficient for WriteFreely so we can just use the Ubuntu packages for it (make sure to install both nodejs and npm; unlike upstream, they don't come bundled together in Ubuntu):

$ sudo apt update
$ sudo apt install -y nodejs npm
$ node -v
$ npm -v

OK, prerequisites sorted, on to getting our WriteFreely development environment set up.

(Side note: apt update might tell you that there are upgrades available; you may as well apply those by running sudo apt upgrade.)

Setting up our WriteFreely development environment

Nobody wants to have to run their IDE or editor inside of their development VM, so we're going to have a WriteFreely tree on our host that we mount into the development VM.

(Note that these instructions diverge from the Development Setup guide because we're splitting things across host and VM.)

On the host

Let's start by cloning the WriteFreely repo. You can put this wherever you want (I use ~/personal_dev), just make sure to make a note of the path to the repo for the next step. Run this command:

$ git clone https://github.com/writeas/writefreely/ \

Now, we want to create the directory for multipass to mount to (multipass should handle this itself, I think, so I'll be filing a bug in the near future):

multipass exec writefreely-dev -- \
    mkdir -p go/src/github.com/writeas/writefreely

Next, we're going to ask multipass to mount the host's directory into the directory we just created in our development VM:

$ multipass mount \
    ~/personal_dev/writefreely \

And we can check that that's worked:

$ multipass exec writefreely-dev -- \
    head -n1 go/src/github.com/writeas/writefreely/CONTRIBUTING.md
# Contributing to WriteFreely

Great! So now we have a WriteFreely tree that we can easily edit from our host, mounted in to our VM so we can actually run WriteFreely there.

(BTW, did you notice that we used multipass exec to run mkdir and head directly within the VM? If you don't need a full shell, use exec.)

Oh, there's one last piece of information we need to grab before we jump into installing and configuring WriteFreely: the IP address of our development VM. You can find this by running multipass list:

$ multipass list
Name                  State           IPv4             Image
writefreely-dev       Running     Ubuntu 18.04 LTS

In my case, the address is Make a note of yours!

In our VM

OK, let's get back in the VM:

$ multipass shell writefreely-dev

And let's change to our mounted directory:

$ cd go/src/github.com/writeas/writefreely/

We're now basically at the point after the first two commands in the “Setting up” section of the Development Setup guide, and we are effectively going to run through those commands in sequence. However, as we aren't running our development server on our host, there are a couple of specific choices we need to make.

First, though, let's install all of our dependencies and compile WriteFreely. To do that, we first need to do a little bit of house-keeping. Go will install binaries into ~/go/bin, and we need to make sure that those binaries are available to run by adding them to our PATH. We'll also modify .bashrc so that we don't have to remember to do this every time we log in:

$ export PATH=$PATH:~/go/bin
$ echo 'export PATH=$PATH:~/go/bin' >> ~/.bashrc

We can then install our dependencies and compile WriteFreely:

$ export GO111MODULE=on
# Save ourselves some time next time we login
$ echo 'export GO111MODULE=on' >> ~/.bashrc

$ make build

This will take a couple of minutes. (GO111MODULE=on is required to be set, and we also add it to our .bashrc so we don't have to remember this every time.)

Once this is complete, we're ready to configure our development WriteFreely instance by running:

$ make install

This is going to prompt you with a few questions. The important answers are:

  • Environment: “Development”
  • Database driver: “SQLite”
  • Public URL: http://<IP ADDRESS>:8080, replacing <IP ADDRESS> with the VM's IP address you made a note of earlier

Other than those, you can just accept the defaults. (Of course, make sure you remember your username/password, you'll need them to log in to your development site once it's running.)

We're very, very close to having a running site! The last change we need to make is to configure the development server to accept connections from our host. (The default configuration will only accept connections from the machine that it's running on, which is a very sensible default if you're running this on your host directly, but doesn't work for us.)

To do this, open up config.ini in your editor of choice (in or out of the VM, either should work fine) and change the bind value to “”. After editing, that line should look like this:

bind                 =

And now, finally, in our VM, we can start the development server:

$ make run

This will print out a bunch of logging with the line Serving on towards the end. Once you see that, you should be able to browse to the Public URL you configured before (http://<IP ADDRESS>:8080, remember?) and see your development instance!

Log in using the username/password you configured during make install and start making WriteFreely even better!

I keep notes of the work that I do as a member of Canonical's Ubuntu Server team. This post is a breakdown of the highlights of what I spent my last week doing.


  • Landed MP 370927 to fix bug 1838794, a doc rewrite for the cc_set_passwords module
  • Worked on and submitted MP 371053 to introduce support for configuring secondary NICs in Oracle Cloud Infrastructure Virtual Machines
  • Helped a couple of Debian users in #cloud-init
  • Submitted and landed MP 371090 to remove the (unused) intersphinx plugin from our Sphinx configuration
    • This was prompted by it causing occasional hangs when I was doing doc builds locally
  • Code reviews
    • Re-reviewed and landed the Exoscale data source (MP 369516)
      • Thanks to Chris Glass for iterating on this with me, and to Mathieu Corbin for the initial implementation!
    • Re-reviewed and landed SSH host key publication (MP 370348)
      • Thanks to Rick Wright!
  • Uploaded a new upstream snapshot of cloud-init to eoan including the above two changes
    • This should mean they are included in the eoan daily cloud images over the weekend
  • Discussed (in bug 1839538) the “plethora of values that are accepted” by cloud-init for true/false flags (e.g. “on”, “true”, “1”, “yes”, true are all “True” values)
    • The conclusion was that we should close that bug Won't Fix and avoid introducing more configuration options that support the values
    • I filed bug 1839659 for discussion of a deprecation plan for the string values


  • No significant curtin work this week.


  • Proposed and landed PR #701 to drop support for authenticating with Ubuntu SSO
    • It was decided against at the UX review at last week's planning sprint
  • Landed PR #697 fixing an issue with contracts with expiry dates
  • After internal discussion (mentioned last week), filed #700 to convert the UA client codebase to black formatting
    • Submitted PR #705 to introduce black config, and apply black formatting to the parts of the codebase that don't have in-flight changes
  • Code reviews


  • Off on Monday for Civic Day, so some extra catch-up early in the week
  • Last week was an internal planning sprint, so more meetings than usual this week to debrief
  • I use Workrave to help manage my RSI, and definitely recommend it
  • cloud-init/curtin bug triage on Tuesday
    • Pretty light day both for bugs and CI issues

Consider this code:

def example(param: str) -> None:
    print("%s" % param)

Seems fine, right? It's just going to print out whatever we pass in:

>>> example("foo")
>>> example(["first", "second"])
['first', 'second']

Yep, it's all going great, time to move on to the next prob— hold on:

>>> example(("first", "second"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in example
TypeError: not all arguments converted during string formatting

What's going on here? Well, %-formatting treats tuples specially, considering them to be a list of arguments to the string template, rather than a single object to be inserted in to the format string. This is an unfortunate relic that can't be changed due to backwards compatibility.

There are a three ways you can address this. You can switch to using the .format() method:

def example_format(param: str) -> None:

or, if you want to make minimal changes and stick with the old-style formatting, you can ensure that there is only a single element in the arguments tuple by explicitly wrapping it:

def example_wrapped(param: str) -> None:
    print("%s" % (param,))

Or, finally, if you only need to target Python 3.6 and later, you can use f-strings:

def example_fstrings(param: str) -> None:

As we can see, in each of these cases, all is well:

>>> example_format(('first', 'second'))
('first', 'second')
>>> example_wrapped(('first', 'second'))
('first', 'second')
>>> example_fstrings(('first', 'second'))
('first', 'second')