Distracting adventures in ZFS upgrades

Sep 04 2015

Last week I wanted to play around with some software packages for logging and charting of environmental measurements and events (specifically, two packages, openhab, and emoncms)
Wanting to save time (sweet irony!), rather than building up a VM and manually configuring the tools, I figured I’d use docker. Except that the workstation I wanted to use was running Debian Squeeze was still on kernel 3.2, which doesn’t support docker. Oh, and a ZfsOnLinux (ZoL) zraid for the root filesystem.
So the steps to get to docker involved upgrading the kernel, ZFS, and by the way, the nvidia drivers.
Mistake #1. I should have just built a Xubuntu 14.04 VM and run docker inside that!

Before upgrading the kernel from Debian backports, I decided to ensure ZfsOnLinux was updated. I (correctly, confirmed) anticipated the most problems with ZoL. Anyway, I knew that upgrading ZoL would be fraught with danger so I read all the documentation, and upgrade advice, and so on, and took all the recommended precautions.

But of course, after going through two cycles of apt-get and dpkg-reconfigure and  rebuilding the initramfs and so on, after rebooting, BAM! A variant of the dreaded “failed to mount the root filesystem” error. Reported close by was a missing kernel module error for something called zcommon.

After a bit of digging and breaking the virtual glass on an emergency boot partition I worked out that I had missed upgrading one of the packages required for ZoL. Why it was not an automatic dependency I don’t know, but after installing something called “libvnpair” the system booted further. And then stopped again.

This one would take rather a bit more work to track down. Semi-helpfully, the entire error message was:

Manually import the root pool at the command prompt and then exit.
Hint: Try: zpool import -R / -N ${ZFS_RPOOL}

At this point, the initramfs was dropping my system to a rescue shell, and via the above message advising me to import the ZFS pool containing the root filesystem. So I tried its helpful suggestion to execute the ‘zpool import’ command, which actually succeeded, and after some more fiddling manually mount various file systems proceeded to boot the system. However, this manual process only got me out of trouble once, and still needed to be resolved.

To get further I had to instrument the initramfs file scripts/zfs with a bunch of echo statements and rebuild the initramfs. (The script files bundled in when rebuilding initramfs on Debian are located under /usr/share/initramfs-tools/scripts) This let me reboot and work out where the zpool import was failing (or not even being called at all.)

As it turns out, zpool was not being called, at all, in a way that would work for my partitioning scheme. The logic in scripts/zfs runs a whole bunch of permutations trying to locate the pool, but if a variable called ROOT is empty it skips executing zpool as required. The solution, as it turns out, was to update my grub with ‘root=zfs:AUTO‘ – previously, my kernel did not require this kernel argument, but now, having upgraded ZoL, from 0.6.2 to 0.6.4, it did.

So, what caused this? There were a lot of year or so old threads discussing upgrade errors related to ZfsOnLinux but none of them quite matched my specific scenario.

One possibility is this:
* I run a separate boot filesystem from the usual /boot, containing a hand crafted grub, which can execute various tools such as Gparted, various minimal linux installs for rescue purposes, memcheckx86 and other tools.
* Whenever I upgrade the kernel on this system I need to copy over the vmlinux and initramfs files to this originating boot filesystem from /boot (which is never used by my grub)
* I wonder if ZoL  may have added the root=zfs:AUTO option to the Debian grub update facility, but I neglected to check for changes to the generated /boot/grub/grub.cfg and apply any changes to  my real grub.cfg. And wham!

However, I couldn’t find any references to zfs in /etc/grub.d, so this hypothesis may well be wrong. Via occams razor, perhaps its just that my setup on this particular workstation is more complex or unusual than most users of ZfsOnLinux. Anyway, onward and upwards.

I’ shortly to decide on which of OpenHAB or EmonCMS I’ll be using for my Hackaday Prize finals entry. Stay tuned!

No responses yet

Using an i2c RTC with the Carambola2 (or any OpenWRT modified router)

Apr 27 2015

Using i2c with a modded router is simple enough, if you have two spare GPIO then the module package kmod-i2c-gpio-custom allows selected GPIO pins to be bound to SCL and SDA respectively when the module is loaded.

However for inexplicable reasons the ability to bind an i2c RTC module to the Linux hardware clock is disabled by default by the OpenWRT configuration mechanism for ar71xx and other consumer router architectures, and there is no way to turn it on without patching!

Regardless, here is how to use an i2c RTC with the Carambola2 or any other ar71xx architecture router (e.g. WRTnode, etc.)

  • Patch the file target/linux/ar71xx/generic/target.mk as follows:
  • Patch the kernel configuration target/linux/ar71xx/config-3.xx where (xx depends on your version of OpenWRT) as follows:
  • Note: the kernel configuration can be modified via the kernel build system using the command make kernel_menuconfig
  • Note: add other kernel i2c RTC modules as required
  • Add the module to your image:
  • If you have previously built OpenWRT then remove the tmp/ directory, or the change ‘+rtc’ will be ignored and the DS1307 module will not be included in your image
  • Run make defconfig
  • Build your image: make -j2

If everything worked, then the the file /lib/modules/3.xx…/rtc-ds1307.ko should be in the resulting image

Following is an aggregation of information I was already able to find elsewhere on the net.

  • Ensure that i2c-tools package is installed as well. This may require the ‘oldpackages’ feed.
  • Configure the module as follows by creating a file /etc/modules.d/99gpio-i2c-rtc
  • You can also put this file into files/etc/modules.d/99gpio-i2c-rtc for it to be automatically added to your image
  • Create the following content, where in this example 18 == SDA pin id and 19 == SCL pin id
  • There are additional arguments controlling delays, etc.; refer to package/kernel/i2c-gpio-custom/src/i2c-gpio-custom.c
  • Create a script, /etc/init.d/rtc-driver to load the device driver and set the time.
  • Create a symlink…
  • Note, if you are running ntp that will take over anyway, but for system with an intermittent or no network connection, or if the network is down on boot, the RTC will provide a better time than 1 Jan 2012 or whatever…

You can test the above out before scripting it by booting the system and manually stepping through:


PS Dont forget pullup presistors, and take care interfacing between 5V and 3.3V systems and peripherals…

No responses yet

Why is the Arduino IDE so stupid?

Apr 23 2015

If I perform the following actions:

  • File, New
    Opens a new editor window. Reasonable enough, although I would have preferred a default single-window GUI model like QtCreator or even gedit.
  • File, Save
    Opens a save as dialog. In spite of the Arduino ‘sketchbook’ directory, it opens in my home directory.
  • New Folder
    Creates a directory New Folder, but doesn’t shift the focus to it, leaving you confused when this is done in a directory with a lot of files…
  • Click on ‘New Folder’ and rename it, say, Test123
  • Navigate into Test123/
  • Type in a filename for the project, say, TestTest1
  • Hit save.
    So now Arduino IDE dutifully ignores what I typed and proceeds to create a tab called ‘Test123’.
    It will even do this if ‘Test123/’ already existed.
  • File, Save As.
    It forgets where you where in the hierarchy and starts in the home directory again(!)
  • Navigate to Test123/ intending to use it as a container for multiple projects
  • Type in a filename, say Hello, then hit Save
  • The sketch is _still_ called Test123.

So insanely enough, it seems you essentially create a director and thats where the sketch gets its name.  Within that directory it creates a file with the same name with the extension ‘.ino’

Lets try something else:

  • From the shell, create a directory, Test456 and create a readme.txt file, and a directory Test456a and a file Test456a/readme2.txt
  • File New
  • File save
  • Navigate into Test456
  • Type in helloworld for the name
  • Again, the project gets called Test456
  • But take a look in the directory Test456: the contents are now gone (all, including the sub directory Test456a) and replaced with Test123.ino

Luckily I discovered this in a directory in a git working copy with no modifications so I didn’t lose anything important.

Testing done using Arduino1.5.8 amd64 for Digispark. So its a little out of date but not exactly the oldest either.

I have used Arduino before and to be honest I don’t recall it being this stupid, but maybe I just got lucky.

One difference is this time I got sick of the massive latency opening the windows and tried a few different Java JRE (openjdk6, openjdk7, gcc4.7-jre) before discovering that with gcc4.7-jre the menus are as snappy as the openbox right click menu, or even a (*sharp intake of breath*) brand new Windows 7 corporate desktop… maybe there is some API implementation difference between the JRE’s that affects the file save dialog functionality.

I don’t seem to have any issues opening projects.

So my workflow for creating a new project now consists of:

  • From the shell, create a directory in the relevant part of the git working copy I am using
  • Create a new empty .ino text file with the same name as the directory (or copy a template I made)
  • Open it with the IDE and start working


No responses yet

Challenge for 2015: hackaday prize competition

Mar 29 2015

So the 2015 Hackaday prize is happening, until at least August.

Somehow I’ve currently ended up involved with not one, but two entries!  The good thing is that with four months to go until the first round submission, I have been careful not to bite off more that can be chewed in the time available on weekends, or after the kids go to bed, etc. with other commitments. Along the way though it should be educational and fun, and with any luck I might at least win a T-shirt or something (some electronics test gear would be nice) … I’m under no illusion we will get anywhere near winning a trip to space!

The themes this year are is “Build Something that Matters”, around environment, agriculture and energy, with the related facet of solving a problem, and not necessarily a world-scale problem.

So my first project, of which I am making good progress, is a farm crop monitoring system for Australian conditions.  This utilises the ESP8266 wifi module and will exercise its deep sleep mode, and solar power, along with a yet to be determined Linux module for a local base station, and hopefully ISM band telemetry over long distances. I will also be helped by my neighbour who is a farmer who can use this system.

The second project, which is not my idea but that of a close friend, (but for which I am presently responsible for maintaining the hackaday.io page), is an Algorithmic Composting machine built out of repurposed parts and cheap electronics.  I’ll probably end up assisting with the embedded electronics, as well as keeping the documentation up to date.

I wont be posting here in a lot of detail as the contest progresses, as there is a project log built into the hackaday.io site intended for that purpose.  So follow along at http://hackaday.io/project/4758 and http://hackaday.io/project/4991  instead! (And please like our projects if you have a hackaday account!)


No responses yet

hacksa2015 – we won!

Mar 14 2015

The previous week we were informed that we were the winners of the first hacksa competition!

This was really awesome, we had put in a bit of work that week and it validates my ideas about how to approach a hackathon I blogged about previously . You can see our proof of concept web application at http://phaze.space

I think what helped us over the line was that we had a working web application that actually ‘did something’, or a ‘minimum viable product’ in the parlance: we demonstrated the primary user experience ( generate musical playlists when you don’t know what to choose ) along with various potential features illustrated by button placeholders.

There was a cash prize, and some music, and headphones, and a membership in a co-working space which we donated to the runner-up because we all have day jobs and wouldn’t be able to use it.  For me though the best prize was tickets to the NetWorkPlay conference held in Adelaide last week.  This was a completely different scene, this was a media industry conference (mostly documentary film-makers, and a mix of other film industry and media) and I met some different and quite interesting people.

One takeaway from NetWorkPlay as a software guy was research showing that most younger people directly use youtube as a search engine instead of google when searching for media. This was interesting, my first instinct (habit?)  is using google or other ‘traditional’ search engines even when searching for videos that ends up with me on YouTube anyway. A learned quite a few other interesting things, and more importantly had to move out of my comfort zone and had a good time interacting with people I would never have likely crossed paths with.

So thanks to my team members (you know who you are) for an awesome effort, and I’m looking forward to govhack 2015!

I’d also like to thank the competition organisers, including madeinkatana.com , SA Music Development Office, Musitec and Flinders New Ventures Institute, and the sponsors for the generous prizes.

No responses yet

Older posts »