Raspberry Pi Virtual Machine Automation

Aug 23 2014

Several months ago now I was doing some development for Raspberry Pi. I guess that shows how busy with life things have been. (I have a backlog of lots of things I would like to blog about that didn’t get blogged yet, pardon my grammar!)

Now the Pi runs on an SD card and it some things start to get very tedious after a while, write performance is not exactly fast, and doing extensive work would probably start to wear cards a bit fast.

So I looked into running a Raspberry Pi Qemu virtual machine, which would let me do builds on my main workstation without needing to setup a cross compiling buildroot. This has the further advantage in that I could test automating full installations for real Raspberry Pis.

Overview

Because I really dislike having to lots of manual steps I automated the whole process. The scripts I used are on GitHub (https://github.com/pastcompute/pi_magic), use at your own risk etc.

There are three scripts: pi_qemu_build.sh which generates a pretend SD card image suitable for use by Qemu; pi_qemu_setup.sh which will SSH into a fresh image and perform second stage customisation; and pi_qemu_run.sh which launches the actual Qemu VM.

My work is based on that described at http://xecdesign.com/qemu-emulating-raspberry-pi-the-easy-way/.

Image Generation

The image generation works as follows:

  • start with an unzipped Raspbian SD card image downloaded from http://www.raspbian.org/
  • convert the image to a Qemu qcow2 image
  • extend the size out to 4GB to match the SD card
  • mount using Qemu NBD and extend the ext2 partition to the rest of the disk
  • Patch the image to work with Qemu quirks (more on that in a moment)
  • Create a snapshot so that changes made later inside Qemu can be rolled back if required

Now Raspbian is setup to load a DLL /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so, which needs to be disabled for Qemu; also it is set up to search for an MMC device so we need to make a symlink to access /dev/mmc* as /dev/sda inside Qemu instead.

Qemu Execution

The stock Raspberry Pi kernel wont work inside Qemu, so we need to launch with a different kernel.

I haven’t yet had time to include producing one in the build process.  Instead, I launch Qemu using the kernel downloadable from XEC Design.

Qemu is invoked using qemu-system-arm -kernel path/to/kernel -cpu arm1176 -m 256 -M versatilepb ..., so that the processor matches that used by the Raspberry Pi. The script also maps HTTP and SSH through to the VM so you can connect in from localhost, and runs using the snapshot.

This was tested on Debian Wheezy with qemu 1.7 from backports.

PI VM screenshot

PI VM screenshot

No responses yet

Unleashed GovHack – an Adelaide Adventure in Open Data

Aug 19 2014

Last month I attended Unleashed Govhack, our local contribution to the Australian GovHack hackathon.

Unleashed

Essentially GovHack is a chance for makers, hackers, designers, artists, and researchers to team up with government ‘data custodians’ and build proof of concept applications (web or mobile), software tools, video productions or presentations (data journalism) in a way that best illustrates how the vast amount of publically sourced data held by the government in trust for the community can be ‘unleashed’ for the benefit of the wider community. There was also the Machinery of Data competition, where participants combine data with physical manifestations.

Entrants form teams, and across the weekend define a concept and produce an application or a presentation, and in all cases a video to showcase the idea. Prizes are awarded both nationally, and for Adelaide and Mount Gambier specifically, for entries meeting various criteria such as “More Informed Adelaideians”, “Open Society Unleashed”, “Best entry by a University team, etc. A second aim is also to promote the small business / “startup” sector, so there are prizes related to business mentoring, for example.

Just a very, very small example of available data sets include: geospatial data of significant trees and also the Waite Arboretum, locations of recycle stations, air quality measurements, historic photos and images, City of Adelaide public wifi hotspots, locations of sightings of endangered species, to name but a few.

A new experience

I brought along with me my son who is now 13. He aspires to be an Author when he grows up, and also has a natural ability in maths as well as written language. I am encouraging him to have a backup plan, so at least he has something to write about! I thought this would be a wonderful opportunity to introduce him to the community side of the “digital revolution” – the Internet is about more than games, and computers about more than just Minecraft!

He charmed the volunteers, who were excited at the idea of having a school youth element attending. Together we both learned a lot, this was an especially intense weekend and by the Sunday afternoon during video production things got a bit tense at times; I need to learn to chill a bit, because he is quite capable of producing video! He has been creating presentations at school and some extra-curricular activities for a few years now, so a taste of data journalism was something he actually thought quite interesting.

In the end, he was fortunate enough to receive one of the weekend “Spirit” prizes, he did very well in talking to lots of people and generally participating, he gracefully received help from some members of Adelaide Hackerspace who helped him sort data on the Saturday,  and as we found out on the weekend, he was also awarded a commendation for the “Data Journalism” presentation that he created, where he examined correlations between endangered animal sightings and Australian natural disasters to inform policy makers when allocating funds related to endangered species protection. (His presentation was from a 13yo perspective, the previous paragraph is obviously my words!) You can see the final on the GovHack Hackerspace.

A crash course, or what not to do if you want to get the most out of the weekend!

For my part, I was really along for the ride, this time at least, having not participated in such an event before.  I am a professional software developer, but that is no help if you only leave yourself a day to attempt to write any real code!  I formed a team of my own with a friend from Adelaide Hackerspace, as well as having to be on my sons team as his guardian, but for various reasons it took us until halfway through Saturday to decide on a theme, and we also spent a lot of time on Sunday producing video, having to help Zachary as well as do our own.

From observations of other, successful teams, as well as the pain of experience, I have compiled a list of lessons fir first time / future participants:

  1. Most of important of all: decide what to build on Friday night! We dithered until way into Saturday so we were effectively a day behind other teams.
  2. Almost as important: don’t get carried away. Start small, stay focused, and then when you pull of the first concept you should have time to incrementally expand. It is easy to get very excited about all the possibilites, which is fine, but save the embellishments for the video so you get something finished.
  3. Become very familiar with your chosen video editor before the Govhack weekend
  4. If you intend to deploy a web application, become very familiar with your provider. Govhack actually supplied AWS vouchers, and luckily AWS was one area where I did have some recent experience so for my team at least, this was one area that did not cause frustration.
  5. Avoid large teams, so you avoid analysis paralysis.  It may be tempting to do a large group brainstorm, but then you do get carried away and cant keep to #2 above.  This also happened to us, which is why we didn’t really define our projects until Saturday when we split into smaller groups.
  6. Don’t bite off more than you can chew! Stay with your teams skillsets. See also #2
  7. For working with data ensure you (or some team members) have good familiarity with tools such as CSV conversion scripting, graphical tools such as QGIS, and scripting languages and tools for pulling data from PDF files
  8. Aim to have familiarity with SQL database systems, this can make life easier if you want to use a tool like PostGIS for geographic analysis
  9. If like to build web applications, get back up to speed before the weekend with a good rapid prototyping framework
  10. Decide how to host beforehand if you are planning a web app. CPanel will limit you to PHP or PERL so if you want to use Postgresql for geo features, or a Python backend you will probably need a VPS.

Obviously the later items in the list are more specific to programming skills.  Some of the items are more applicable if you have already organised a team before the night.  If you form a team late it can be hard to plan skill set coverage, in which case you may be better off aiming for a comprehensive and illuminating data journalism entry rather than an app.  It does depend on your specific  circumstances.

All in all it was a good weekend, both myself and my son met a lot of awesome new people, learned new skills and experiences and in a small way contributed to the community.

Finally a massive thanks to the volunteers who organised the whole thing!

There are photos up on Twitter or over on the official Facebook page.

No responses yet

Avoiding your own Goto Fail in a pinch

Jun 08 2014

(Note – this article particularly applies to C, C++ and Java like languages.)

Recently I came across some code that had the same kind of latent defect as was responsible for the now (in)famous ‘Goto Fail’ event [1]. So it was an excuse to write a blog article, I have been a bit slack busy lately.

OK. So ideally your project has decent revision control, with source code guidelines, code review and tools to make it easier to produce high quality code. This should make it easier to reduce the odds of your very own ‘Goto Fail’ event, right?

Well, here is what often happens in the real world:

  • there will be times the project is “experimental”, or done in a hurry and not intended to last (and just how many good ideas started off like that, indeed?)
  • very often, you have to maintain a legacy codebase
  • you need to “ship something yesterday” and haven’t been provided with budget to keep it properly maintained; sometimes the accountants win…

With Goto Fail in particular, part of the problem was not really the use of the much maligned goto statement in C, but rather lack of adherence to coding standards. In particular, a lack of braces that limit scope!

Many observers of course jumped on the ‘never use goto’ meme [2], but one thing I have come to learn in life, almost nothing is black and white. The use of goto is like any tool, in this case, one reserved for special situations. To see valid scenarios where goto has both readability and performance benefits take a look at the Linux kernel [3]; a well placed goto can remove a tangle of nested if/else/etc as well as provide a key performance improvement, when used properly.

Noting that the Linux kernel is not a normal application; the need for goto will still be extremely unlikely in most mundane applications.

Summary

Here is a quick and dirty method for auditing a code-base for possible “goto fail”-type defects.

Requirement

We are looking for the following code fragments:

These are bad because inadvertent edits can produce:

This should be a big no-no!

Instead, we treat either of the following as OK:

This is actually tighter than the Google Style Guide [4]; consider it wise to plan for multiple developers editing future code, some who may be inexperienced or “just passing through” – on the theory of if they expend the least energy, if the braces are there, they will use them!

Method #1 – from a Linux / Unix shell

Extend the -name clause as required.

This works by finding all lines where the first non-whitespace is the if statement, and then reports if there is NO opening brace anywhere after on the same line.  It handles any whitespace combination.

Method #2 – the following regular expression may work in Microsoft Visual Studio

Caveat: I haven’t had a chance to test this ye, so if it is wrong, please let me know!

Notes

Now, these wont catch everything: if anything ‘creative’ is happening inside a preprocessor macro for example; also this may trigger false positives if you have a string constant that includes the text " if (". It is also probably not resilient against nested statements on the same line, if your guidelines allow it (e.g. if (blah) { for (;;) would not be caught without modifying the expression.

The above expressions, as-is, will also produce many false positives, if your coding guideline allows the following:

I’ll leave it as an exercise for the reader to adjust the regular expression to suit… it should extend to more complex situations if required. You could also use sed or pcregrep or even perl -efor example.

if you have to remediate a very large code base, you can redirect the output to a file and measure progression over time. You could script out any false positives as well.

[1] http://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-goto-fail-apples-ssl-bug-explained-plus-an-unofficial-patch/

[2] http://en.wikipedia.org/wiki/Considered_Harmful

[3] https://web.archive.org/web/20130410044210/http://kerneltrap.org/node/553/2131

[4] http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Conditionals

No responses yet

Booting a Windows7/Vista Recovery partition when Windows is broken

Apr 22 2014

Even though I am “pretty much” an open source advocate, I still have to use Windows professionally when required, and of course am the IT support for extended family :-) In this case, I needed to rebuild a laptop for my mum from scratch.

The laptop in question, a Benq Joybook A52 had previously been my dads, and been through incarnations of Windows Vista, downgraded to XP then back up to Vista, and I decided it would be safer to start with a clean slate and perform a factory restore, apply the service packs and all the recent security updates anew.

This laptop had been previously been cleansed of the usual ‘crapware’ and other default programs, including the factory recovery icon. I had early on installed the very useful tool EasyBCD from http://neosmart.net/EasyBCD/ to dual boot Vista and XP. (Aside: EasyBCD used to be free, it seems you can still get it free for Non-Commercial use but you have to dig a bit.) Using a Linux bootable USB I was able to detect that the recovery partition, a FAT32 primary partition labelled ‘PQSERVICE’ but set to type 0xde (Dell Utility) was still present (luckily). However regardless of which settings I tried I was unable to immediately get the factory recovery partition to start, either from the rescue USB or via EasyBCD. Being an mum & dads for dinner I didn’t have a lot of time to get into nuts and bolts.

Surprisingly, a quick search for PQSERVICE, booting benq recovery partition or various other combinations didn’t really bring up anything that useful. So for the moment /dev/sda4 remained stubbornly inaccessible to the Windows boot machinery.

A couple of weeks later, now having the laptop in my possession to deal with this, I took an image so that I could experiment. Luckily the drive was only 80GB, such an expansive size from circa 2006!

First thing I did was create an image to play with in qemu. Figuring that the important parts were simply the recovery partition and the boot sector, I managed this as follows:

(Aside – these instructions may or may not also work on other flavours of laptop of this vintage!)

Firstly, upon mounting the recovery partition, you can see what appears to be a bog standard cut down Windows filesystem:

Examine the partition layout:

Results:

The recovery partition is #4. Note that ‘diag’ is actually type 0xde when checked using fdisk.

Second, assemble a fresh experimental disk:

As a check the size of test.bin and laptop.img should be identical.

Now, attempt to boot the image in QEMU.

This is achievable using Grub2, and in this case I chose SuperGub2Disk, http://www.supergrubdisk.org.

After the boot screen starts, choose ‘c’ for a command line, and use the following:

For once, this worked first time, starting the Powerquest recovery software.

So now I simply had to repeat this process, using a USB key with Grub2 on the laptop itself.

Prologue

For some reason the BIOS in this laptop did not understand my favourite bootable USB with Grub2 so I ended up burning a CDR for the first time in a little while.

I also had to wipe the other partitions out of the partition table; it seems the recovery program just unpacks a partition to C: rather than rebuilding the partition table! Things to note include, remember to set the partition type of what will become C: (/dev/sda1) as bootable and NTFS; and for good measure zero the first sectors of that partition.

No responses yet

OpenWRT WDS between legacy WRT54G and recent TP-Link devices

Mar 30 2014

For a while now I had a multiple wifi routers all providing access points, and a connection to each other, using a feature called WDS. All of the routers run OpenWRT. Recently one of them died and everything kind of stopped working properly. I actually had the following configuration:

TP-LINK <--wired,bridged--> ASUS WL500G <--wireless,WDS,bridged--> Linksys WRT54G

Now this all worked because the conventional wisdom appears to be that WDS works when the device at each end is the same, i.e. Broadcom to Broadcom (which I had in the case of WL500G and WRT54G) and is problematic at best otherwise.

Documentation alluding to this includes:

http://wiki.openwrt.org/doc/recipes/broadcomwds

http://wiki.openwrt.org/doc/recipes/atheroswds

http://wiki.openwrt.org/doc/howto/clientmode

So with the WL500G out of the picture, the TP-Link and WRT54G were no longer able to see each other, and I experienced a loss of connectivity from my workshop where the WRT54G was physically located.

However if you look under the covers a bit, it seems this configuration can be made to work, as suggested by the folk over at DD-WRT.

So with a bit of tweaking I was able to get WDS working between the TP-LINK and WRT54G as follows.

TP-LINK Router (WR1043ND, OpenWRT 10.03 Backfire)

Content of /etc/config/wireless:

WRT54G Router (WRT54G, OpenWRT 10.03 with legacy 2.4 kernel for broadcom wireless)

Content of /etc/config/wireless:

Notes

  • xx:xx:xx:xx:xx:xx is the MAC address of the WR1043ND, and yy:yy:yy:yy:yy:yy the MAC address of the WRT54G
  • Both devices have to be on the same channel
  • On the secondary router (WRT54G) you actually connect on the ssid LOCALSSID but all DHCP, etc. comes from the TP-LINK

Note also you need to ensure that DHCP on the LAN is disabled on the second router!

I am unsure why this configuration has apparently been problematic for some – perhaps it is quite router specific as to whether it works.

No responses yet

« Newer posts Older posts »