Evaluating the security of OpenWRT (part 1)

Sep 18 2014

I have recently spent  some time pondering the state of embedded system security.
I have been a long time user of OpenWRT, partly based on the premise that it should be “more secure” out of the box (provided attention is paid to routine hardening); but as the Infosec world keeps  on evolving, I thought I would take a closer look at the state of play.

OpenWRT – an embedded Linux distribution.

OpenWRT is a Linux distribution targeted as a replacement firmware for consumer home routers, having been named after the venerable Linksys WRT54g, but it is gaining a growing user base in the so-called Internet of Things space, as evidenced by devices such as the Carambola2, VoCore and WRTnode.

Now there are many, many areas that could be the focus of security related attention.  One useful reference is the Arch Linux security guide , which covers a broad array of mitigations, but for some reason I felt like diving into the deeper insides of the operating system. I decided it would be worthwhile to compare the security performance of the core of OpenWRT, specifically the Linux kernel plus the userspace toolchain, against other “mainstream” distributions.

The theory is that ensuring a baseline level of protection in the toolchain and the kernel will cut of a large number of potential root escalation vulnerabilities before they can get started.  To get started, take a look at the Gentoo Linux toolchain hardening guide.  This lists mitigations such as: Stack Smashing Protection (SSP), used to make it much harder for malware to take advantage of stack overrun situations; position independent code (PIC or PIE) that randomises the address of a program in memory when it is loaded; and so-on to other features such as Address Space Layout Randomisation (ASLR) in the kernel, all of these designed to halt entire classes of exploits in their tracks.

I wont repeat all the pros and cons of the various methods here.  It should be noted that these methods are not foolproof; new techniques such as Return Oriented Programming (ROP) can be used to circumvent these mitigations.  But defense-in-depth would appear to be an increasingly important strategy.  If a device can be made secure using these techniques it is probably ahead of 95% of the embedded market and more likely to avoid the larger number of automated attacks; perhaps this a bit like the old joke:

The first zebra said to the second zebra: “Why are you bothering to run? The cheetah is faster than us!”

To which the second zebra replied, “I only have to run faster than you!”

Analysis of just a few aspects.

There is an excellent tool “checksec.sh”, that can scan binaries in a Linux system and report their status against a variety of  security criteria.
The original author has retained his website but the script was last updated in November 2011. I found a more recently maintained version on GitHub featuring various enhancements (the maintainer even accepted a patch from me addressing a cosmetic bugfix.)

Initially I started using the tool to report on just the status of the following elements in binaries and libraries: RELRO, non-executable stack, SSP and PIC/PIE.

The basic procedure for using the tool against an OpenWRT build is straightforward enough:

  1. Configure the OpenWRT build and ensure the tar.gz target is enabled ( CONFIG_TARGET_ROOTFS_TARGZ=y )
  2. To expedite the testing I build OpenWRT with a pretty minimal set of packages
  3. Build the image as a tar.gz
  4. Unpack the tar.gz image to a temporary directory
  5. Run the tool

When repeating a test I made sure I cleaned out the toolchain and the target binaries, because I ahve been bitten by spuriuos results caused by changes to the configuration not propagating through all components.  This includes manually removing the staging_dir which may be missed by <code>make clean</code> . This made each build take around 20+ minutes on my quad core Phenom machine.

I used the following base configuration, only changing the target:

I repeated the test for three platform configurations initially (note, these are mutually exclusive choices) – the Carambola2 and rt305x are MIPS platforms.

There are several directories most likely to have binaries of interest, so of course I scanned them using a script, essentially consisting of:

The results were interesting.

(to be continued)

No responses yet

Raspberry Pi Virtual Machine Automation

Aug 23 2014

Several months ago now I was doing some development for Raspberry Pi. I guess that shows how busy with life things have been. (I have a backlog of lots of things I would like to blog about that didn’t get blogged yet, pardon my grammar!)

Now the Pi runs on an SD card and it some things start to get very tedious after a while, write performance is not exactly fast, and doing extensive work would probably start to wear cards a bit fast.

So I looked into running a Raspberry Pi Qemu virtual machine, which would let me do builds on my main workstation without needing to setup a cross compiling buildroot. This has the further advantage in that I could test automating full installations for real Raspberry Pis.


Because I really dislike having to lots of manual steps I automated the whole process. The scripts I used are on GitHub (https://github.com/pastcompute/pi_magic), use at your own risk etc.

There are three scripts: pi_qemu_build.sh which generates a pretend SD card image suitable for use by Qemu; pi_qemu_setup.sh which will SSH into a fresh image and perform second stage customisation; and pi_qemu_run.sh which launches the actual Qemu VM.

My work is based on that described at http://xecdesign.com/qemu-emulating-raspberry-pi-the-easy-way/.

Image Generation

The image generation works as follows:

  • start with an unzipped Raspbian SD card image downloaded from http://www.raspbian.org/
  • convert the image to a Qemu qcow2 image
  • extend the size out to 4GB to match the SD card
  • mount using Qemu NBD and extend the ext2 partition to the rest of the disk
  • Patch the image to work with Qemu quirks (more on that in a moment)
  • Create a snapshot so that changes made later inside Qemu can be rolled back if required

Now Raspbian is setup to load a DLL /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so, which needs to be disabled for Qemu; also it is set up to search for an MMC device so we need to make a symlink to access /dev/mmc* as /dev/sda inside Qemu instead.

Qemu Execution

The stock Raspberry Pi kernel wont work inside Qemu, so we need to launch with a different kernel.

I haven’t yet had time to include producing one in the build process.  Instead, I launch Qemu using the kernel downloadable from XEC Design.

Qemu is invoked using qemu-system-arm -kernel path/to/kernel -cpu arm1176 -m 256 -M versatilepb ..., so that the processor matches that used by the Raspberry Pi. The script also maps HTTP and SSH through to the VM so you can connect in from localhost, and runs using the snapshot.

This was tested on Debian Wheezy with qemu 1.7 from backports.

PI VM screenshot

PI VM screenshot

No responses yet

Unleashed GovHack – an Adelaide Adventure in Open Data

Aug 19 2014

Last month I attended Unleashed Govhack, our local contribution to the Australian GovHack hackathon.


Essentially GovHack is a chance for makers, hackers, designers, artists, and researchers to team up with government ‘data custodians’ and build proof of concept applications (web or mobile), software tools, video productions or presentations (data journalism) in a way that best illustrates how the vast amount of publically sourced data held by the government in trust for the community can be ‘unleashed’ for the benefit of the wider community. There was also the Machinery of Data competition, where participants combine data with physical manifestations.

Entrants form teams, and across the weekend define a concept and produce an application or a presentation, and in all cases a video to showcase the idea. Prizes are awarded both nationally, and for Adelaide and Mount Gambier specifically, for entries meeting various criteria such as “More Informed Adelaideians”, “Open Society Unleashed”, “Best entry by a University team, etc. A second aim is also to promote the small business / “startup” sector, so there are prizes related to business mentoring, for example.

Just a very, very small example of available data sets include: geospatial data of significant trees and also the Waite Arboretum, locations of recycle stations, air quality measurements, historic photos and images, City of Adelaide public wifi hotspots, locations of sightings of endangered species, to name but a few.

A new experience

I brought along with me my son who is now 13. He aspires to be an Author when he grows up, and also has a natural ability in maths as well as written language. I am encouraging him to have a backup plan, so at least he has something to write about! I thought this would be a wonderful opportunity to introduce him to the community side of the “digital revolution” – the Internet is about more than games, and computers about more than just Minecraft!

He charmed the volunteers, who were excited at the idea of having a school youth element attending. Together we both learned a lot, this was an especially intense weekend and by the Sunday afternoon during video production things got a bit tense at times; I need to learn to chill a bit, because he is quite capable of producing video! He has been creating presentations at school and some extra-curricular activities for a few years now, so a taste of data journalism was something he actually thought quite interesting.

In the end, he was fortunate enough to receive one of the weekend “Spirit” prizes, he did very well in talking to lots of people and generally participating, he gracefully received help from some members of Adelaide Hackerspace who helped him sort data on the Saturday,  and as we found out on the weekend, he was also awarded a commendation for the “Data Journalism” presentation that he created, where he examined correlations between endangered animal sightings and Australian natural disasters to inform policy makers when allocating funds related to endangered species protection. (His presentation was from a 13yo perspective, the previous paragraph is obviously my words!) You can see the final on the GovHack Hackerspace.

A crash course, or what not to do if you want to get the most out of the weekend!

For my part, I was really along for the ride, this time at least, having not participated in such an event before.  I am a professional software developer, but that is no help if you only leave yourself a day to attempt to write any real code!  I formed a team of my own with a friend from Adelaide Hackerspace, as well as having to be on my sons team as his guardian, but for various reasons it took us until halfway through Saturday to decide on a theme, and we also spent a lot of time on Sunday producing video, having to help Zachary as well as do our own.

From observations of other, successful teams, as well as the pain of experience, I have compiled a list of lessons fir first time / future participants:

  1. Most of important of all: decide what to build on Friday night! We dithered until way into Saturday so we were effectively a day behind other teams.
  2. Almost as important: don’t get carried away. Start small, stay focused, and then when you pull of the first concept you should have time to incrementally expand. It is easy to get very excited about all the possibilites, which is fine, but save the embellishments for the video so you get something finished.
  3. Become very familiar with your chosen video editor before the Govhack weekend
  4. If you intend to deploy a web application, become very familiar with your provider. Govhack actually supplied AWS vouchers, and luckily AWS was one area where I did have some recent experience so for my team at least, this was one area that did not cause frustration.
  5. Avoid large teams, so you avoid analysis paralysis.  It may be tempting to do a large group brainstorm, but then you do get carried away and cant keep to #2 above.  This also happened to us, which is why we didn’t really define our projects until Saturday when we split into smaller groups.
  6. Don’t bite off more than you can chew! Stay with your teams skillsets. See also #2
  7. For working with data ensure you (or some team members) have good familiarity with tools such as CSV conversion scripting, graphical tools such as QGIS, and scripting languages and tools for pulling data from PDF files
  8. Aim to have familiarity with SQL database systems, this can make life easier if you want to use a tool like PostGIS for geographic analysis
  9. If like to build web applications, get back up to speed before the weekend with a good rapid prototyping framework
  10. Decide how to host beforehand if you are planning a web app. CPanel will limit you to PHP or PERL so if you want to use Postgresql for geo features, or a Python backend you will probably need a VPS.

Obviously the later items in the list are more specific to programming skills.  Some of the items are more applicable if you have already organised a team before the night.  If you form a team late it can be hard to plan skill set coverage, in which case you may be better off aiming for a comprehensive and illuminating data journalism entry rather than an app.  It does depend on your specific  circumstances.

All in all it was a good weekend, both myself and my son met a lot of awesome new people, learned new skills and experiences and in a small way contributed to the community.

Finally a massive thanks to the volunteers who organised the whole thing!

There are photos up on Twitter or over on the official Facebook page.

No responses yet

Avoiding your own Goto Fail in a pinch

Jun 08 2014

(Note – this article particularly applies to C, C++ and Java like languages.)

Recently I came across some code that had the same kind of latent defect as was responsible for the now (in)famous ‘Goto Fail’ event [1]. So it was an excuse to write a blog article, I have been a bit slack busy lately.

OK. So ideally your project has decent revision control, with source code guidelines, code review and tools to make it easier to produce high quality code. This should make it easier to reduce the odds of your very own ‘Goto Fail’ event, right?

Well, here is what often happens in the real world:

  • there will be times the project is “experimental”, or done in a hurry and not intended to last (and just how many good ideas started off like that, indeed?)
  • very often, you have to maintain a legacy codebase
  • you need to “ship something yesterday” and haven’t been provided with budget to keep it properly maintained; sometimes the accountants win…

With Goto Fail in particular, part of the problem was not really the use of the much maligned goto statement in C, but rather lack of adherence to coding standards. In particular, a lack of braces that limit scope!

Many observers of course jumped on the ‘never use goto’ meme [2], but one thing I have come to learn in life, almost nothing is black and white. The use of goto is like any tool, in this case, one reserved for special situations. To see valid scenarios where goto has both readability and performance benefits take a look at the Linux kernel [3]; a well placed goto can remove a tangle of nested if/else/etc as well as provide a key performance improvement, when used properly.

Noting that the Linux kernel is not a normal application; the need for goto will still be extremely unlikely in most mundane applications.


Here is a quick and dirty method for auditing a code-base for possible “goto fail”-type defects.


We are looking for the following code fragments:

These are bad because inadvertent edits can produce:

This should be a big no-no!

Instead, we treat either of the following as OK:

This is actually tighter than the Google Style Guide [4]; consider it wise to plan for multiple developers editing future code, some who may be inexperienced or “just passing through” – on the theory of if they expend the least energy, if the braces are there, they will use them!

Method #1 – from a Linux / Unix shell

Extend the -name clause as required.

This works by finding all lines where the first non-whitespace is the if statement, and then reports if there is NO opening brace anywhere after on the same line.  It handles any whitespace combination.

Method #2 – the following regular expression may work in Microsoft Visual Studio

Caveat: I haven’t had a chance to test this ye, so if it is wrong, please let me know!


Now, these wont catch everything: if anything ‘creative’ is happening inside a preprocessor macro for example; also this may trigger false positives if you have a string constant that includes the text " if (". It is also probably not resilient against nested statements on the same line, if your guidelines allow it (e.g. if (blah) { for (;;) would not be caught without modifying the expression.

The above expressions, as-is, will also produce many false positives, if your coding guideline allows the following:

I’ll leave it as an exercise for the reader to adjust the regular expression to suit… it should extend to more complex situations if required. You could also use sed or pcregrep or even perl -efor example.

if you have to remediate a very large code base, you can redirect the output to a file and measure progression over time. You could script out any false positives as well.

[1] http://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-goto-fail-apples-ssl-bug-explained-plus-an-unofficial-patch/

[2] http://en.wikipedia.org/wiki/Considered_Harmful

[3] https://web.archive.org/web/20130410044210/http://kerneltrap.org/node/553/2131

[4] http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Conditionals

No responses yet

Booting a Windows7/Vista Recovery partition when Windows is broken

Apr 22 2014

Even though I am “pretty much” an open source advocate, I still have to use Windows professionally when required, and of course am the IT support for extended family :-) In this case, I needed to rebuild a laptop for my mum from scratch.

The laptop in question, a Benq Joybook A52 had previously been my dads, and been through incarnations of Windows Vista, downgraded to XP then back up to Vista, and I decided it would be safer to start with a clean slate and perform a factory restore, apply the service packs and all the recent security updates anew.

This laptop had been previously been cleansed of the usual ‘crapware’ and other default programs, including the factory recovery icon. I had early on installed the very useful tool EasyBCD from http://neosmart.net/EasyBCD/ to dual boot Vista and XP. (Aside: EasyBCD used to be free, it seems you can still get it free for Non-Commercial use but you have to dig a bit.) Using a Linux bootable USB I was able to detect that the recovery partition, a FAT32 primary partition labelled ‘PQSERVICE’ but set to type 0xde (Dell Utility) was still present (luckily). However regardless of which settings I tried I was unable to immediately get the factory recovery partition to start, either from the rescue USB or via EasyBCD. Being an mum & dads for dinner I didn’t have a lot of time to get into nuts and bolts.

Surprisingly, a quick search for PQSERVICE, booting benq recovery partition or various other combinations didn’t really bring up anything that useful. So for the moment /dev/sda4 remained stubbornly inaccessible to the Windows boot machinery.

A couple of weeks later, now having the laptop in my possession to deal with this, I took an image so that I could experiment. Luckily the drive was only 80GB, such an expansive size from circa 2006!

First thing I did was create an image to play with in qemu. Figuring that the important parts were simply the recovery partition and the boot sector, I managed this as follows:

(Aside – these instructions may or may not also work on other flavours of laptop of this vintage!)

Firstly, upon mounting the recovery partition, you can see what appears to be a bog standard cut down Windows filesystem:

Examine the partition layout:


The recovery partition is #4. Note that ‘diag’ is actually type 0xde when checked using fdisk.

Second, assemble a fresh experimental disk:

As a check the size of test.bin and laptop.img should be identical.

Now, attempt to boot the image in QEMU.

This is achievable using Grub2, and in this case I chose SuperGub2Disk, http://www.supergrubdisk.org.

After the boot screen starts, choose ‘c’ for a command line, and use the following:

For once, this worked first time, starting the Powerquest recovery software.

So now I simply had to repeat this process, using a USB key with Grub2 on the laptop itself.


For some reason the BIOS in this laptop did not understand my favourite bootable USB with Grub2 so I ended up burning a CDR for the first time in a little while.

I also had to wipe the other partitions out of the partition table; it seems the recovery program just unpacks a partition to C: rather than rebuilding the partition table! Things to note include, remember to set the partition type of what will become C: (/dev/sda1) as bootable and NTFS; and for good measure zero the first sectors of that partition.

No responses yet

Older posts »