Evaluating the security of OpenWRT (part 2)

Sep 20 2014

In my last post I covered how I setup an OpenWRT build, to examine  a small subset of indicators of security of the firmware.

In this follow-up post we will examine in detail the analysis results of one of the indicators: specifically, the RELRO flag.

A first look – what are the defaults?

The analysis here is specific to the Barrier Breaker release of OpenWRT, but it should be noted that during experiments with the OpenWRT development trunk the results are much the same.

Before diving into RELRO, lets take a look at the overall default situation.

Here is the checksec report for the Carambola2 device (MIPS processor) build.
It is a sea of red…

Default checksec report for OpenWRT x86 buildThe ‘run as root’ errors can be ignored: those programs are actually absolute symbolic links which do not resolve in the host system. Relative symbolic links resolve correctly but are filtered out of the analysis.

The x86 build paints a similar picture:

owrt_cs_report_0_x86(Notably for x86, the NX flag is correctly set, but that is a topic for another time.)

Note, the rest of this post describes how to modify OpenWRT to enable RELRO.  There may be perfectly valid reasons to not enable the flag (for example, using RELRO may have a performance impact, and for a given system the adverse security risk may be judged low), so I have ensured that the suggested mitigation if applied remains a choice in the configuration menu of the system.  For the moment my patch also retains backward compatibility by defaulting to off.

Inside the OpenWRT build system

After a brief look at the build logs, the reason is obvious: the typical gcc linker command is missing the flags needed to enable RELRO:  -Wl,-z,relro -Wl,-z,now (or the direct linker equivalents, -z relro -z now)

What could be done to address this?

OpenWRT provides a hook for appending to the global compiler CFLAGS but there is no similar hook for the linker stage. We could add those flags to the global CFLAGS and they can in fact flow through to the linker for many programs, but that would also be redundant as the flags are irrelevant to the compiler.  In the end I decided I would modify the OpenWRT source to add a new global CONFIG option, which adds -Wl,-z,relro -Wl,-z,now to the global LDFLAGS instead.

The following patch achieves that (note, I have left out some of the help for brevity):

Having attched OpenWRT, and enabled the new flag, lets rebuild everything again and run another checksec scan.

owrt_cs_report_1_x86What a difference!

The results shown above are for x86, the picture is similar for the Carambola2 MIPS image.

The new results indicate that the RELRO flag is present on some binaries but not all of them. From this we can predict that some packages do not fully honour the global OpenWRT build system linker flags. I soon confirmed this**; the implication is that the new flag CONFIG_SECURITY_FORCE_RELRO  is useful, however, a caveat in the Kconfig help is required. In particular, a statement to the effect that the efficacy depends on proper coding of OpenWRT packages (with ideally all packages maintained by the project being fixed to honour the flag.)

** For example: the package that builds libnl-tiny.so does not pass LDFLAGS through to the linker; this and some other base system packages needed patching to get complete coverage.  it is likely that there are other packages that I did not have selected that may also need tweaking.

Another notable package is busybox.  Busybox it turns out uses ld directly for linking, instead of indirectly via gcc, and thus requires the flags in the pure form -z relro -z now. (The busybox OpenWRT package Makefile also happens to treat the global TARGET_LDFLAGS differently from the TARGET_CFLAGS although I am unsure if this is a bug; but that turned out to be a red-herring.) Oddly, this solution worked for MIPS when I tried it previously, but is presently not successful for the x86 build, so further investigation is needed here; possibly I incorrectly noted the fix in previous experiments.

Fun and Games with uClibc and busybox

The other recalcitrant is the uClibc library. I spent quite a bit time trying to work out why this was not working, especially having  confirmed with verbose logging that the flags are being applied as expected. Along the way I learned that uClibc already has its own apply RELRO config item, which was already enabled. Even more oddly, RELRO is present on some uClibc libraries and not others, that as far as I could tell were being linked with identical linker flag sets.

After some digging I discovered hints of bugs related to RELRO in various versions of binutils, so I further patched OpenWRT to use the very latest binutils release.   However that made no difference.  At this point I took a big diversion and spent some time building the latest uClibc externally, where I discovered that it built fine using the native toolchain of Debian Wheezy (including a much older binutils!)  After some discussion on the uClibc mailing list I have come to the conclusion that there may be a combination of problems, including the fact that uClibc in OpenWRT is a couple of years old (and additionally has a set of OpenWRT specific patches.)   I could go further and patch OpenWRT to use the trunk uClibc but then I would have to work through refreshing the set of patches which I really don’t have time or inclination to do, so for the moment I have deferred working on resolving this conundrum.  Eventually someone at OpenWRT may realise that uClibc has undergone a flurry of development in recent times and may bump to the more recent version.


Along the way, I discovered that Debian actually runs security scans across all packages in the distribution – take a look at https://lintian.debian.org/tags/hardening-no-relro.html.

It is worth noting that whenever changing any build-related flag it is worth cleaning and rebuilding the toolchain as well as the target packages and kernel; I found without doing this, flag changes such as the RELRO flag don’t fully take effect as expected.

For maximum verboseness, run with make V=csw although I had to dig through the code to find this out.

I was going to repeat all the testing against a third target, another  MIPS-based SOC the  RALINK 3530 but at this point I don’t really have the time or inclination, I am sure the results will be quite similar.  It would probably be useful to try with an ARM-based target as well.

I should also try repeating this experiment with MUSL, which is an alternative C library that OpenWRT can be built with.


Out of the box, OpenWRT has very limited coverage of the RELRO security mitigation in a standard firmware build.  By applying the suggested patches it is possible to bring OpenWRT up to a level of coverage, for RELRO, to that approaching a hardened Gentoo or Ubuntu distribution, with only a small subset of binaries missing the flag.


My Github account includes the repository openwrt-barrier-breaker-hardening. The following branch include the completed series of patches  mentioned above:  owrt_analysis_relro_everywhere  I hope it will remain possible to apply these changes against the official release for a while yet.

The patch that enables the latest binutils is not in that branch, but in this commit.

No responses yet

Evaluating the security of OpenWRT (part 1)

Sep 18 2014

I have recently spent  some time pondering the state of embedded system security.
I have been a long time user of OpenWRT, partly based on the premise that it should be “more secure” out of the box (provided attention is paid to routine hardening); but as the Infosec world keeps  on evolving, I thought I would take a closer look at the state of play.

OpenWRT – an embedded Linux distribution.

OpenWRT is a Linux distribution targeted as a replacement firmware for consumer home routers, having been named after the venerable Linksys WRT54g, but it is gaining a growing user base in the so-called Internet of Things space, as evidenced by devices such as the Carambola2, VoCore and WRTnode.

Now there are many, many areas that could be the focus of security related attention.  One useful reference is the Arch Linux security guide , which covers a broad array of mitigations, but for some reason I felt like diving into the deeper insides of the operating system. I decided it would be worthwhile to compare the security performance of the core of OpenWRT, specifically the Linux kernel plus the userspace toolchain, against other “mainstream” distributions.

The theory is that ensuring a baseline level of protection in the toolchain and the kernel will cut of a large number of potential root escalation vulnerabilities before they can get started.  To get started, take a look at the Gentoo Linux toolchain hardening guide.  This lists mitigations such as: Stack Smashing Protection (SSP), used to make it much harder for malware to take advantage of stack overrun situations; position independent code (PIC or PIE) that randomises the address of a program in memory when it is loaded; and so-on to other features such as Address Space Layout Randomisation (ASLR) in the kernel, all of these designed to halt entire classes of exploits in their tracks.

I wont repeat all the pros and cons of the various methods here.  It should be noted that these methods are not foolproof; new techniques such as Return Oriented Programming (ROP) can be used to circumvent these mitigations.  But defense-in-depth would appear to be an increasingly important strategy.  If a device can be made secure using these techniques it is probably ahead of 95% of the embedded market and more likely to avoid the larger number of automated attacks; perhaps this a bit like the old joke:

The first zebra said to the second zebra: “Why are you bothering to run? The cheetah is faster than us!”

To which the second zebra replied, “I only have to run faster than you!”

Analysis of just a few aspects.

There is an excellent tool “checksec.sh”, that can scan binaries in a Linux system and report their status against a variety of  security criteria.
The original author has retained his website but the script was last updated in November 2011. I found a more recently maintained version on GitHub featuring various enhancements (the maintainer even accepted a patch from me addressing a cosmetic bugfix.)

Initially I started using the tool to report on just the status of the following elements in binaries and libraries: RELRO, non-executable stack, SSP and PIC/PIE.

The basic procedure for using the tool against an OpenWRT build is straightforward enough:

  1. Configure the OpenWRT build and ensure the tar.gz target is enabled ( CONFIG_TARGET_ROOTFS_TARGZ=y )
  2. To expedite the testing I build OpenWRT with a pretty minimal set of packages
  3. Build the image as a tar.gz
  4. Unpack the tar.gz image to a temporary directory
  5. Run the tool

When repeating a test I made sure I cleaned out the toolchain and the target binaries, because I ahve been bitten by spuriuos results caused by changes to the configuration not propagating through all components.  This includes manually removing the staging_dir which may be missed by make clean. This made each build take around 20+ minutes on my quad core Phenom machine.

I used the following base configuration, only changing the target:

I repeated the test for three platform configurations initially (note, these are mutually exclusive choices) – the Carambola2 and rt305x are MIPS platforms.

There are several directories most likely to have binaries of interest, so of course I scanned them using a script, essentially consisting of:

The results were interesting.

(to be continued)

Further reading

[1] http://wiki.openwrt.org/doc/howto/secure.access
[2] https://wiki.archlinux.org/index.php/Security
[3] http://wiki.gentoo.org/wiki/Hardened/Toolchain
[4] http://blog.siphos.be/2011/07/high-level-explanation-on-some-binary-executable-security/
[5] http://www.trapkit.de/tools/checksec.html

No responses yet

Raspberry Pi Virtual Machine Automation

Aug 23 2014

Several months ago now I was doing some development for Raspberry Pi. I guess that shows how busy with life things have been. (I have a backlog of lots of things I would like to blog about that didn’t get blogged yet, pardon my grammar!)

Now the Pi runs on an SD card and it some things start to get very tedious after a while, write performance is not exactly fast, and doing extensive work would probably start to wear cards a bit fast.

So I looked into running a Raspberry Pi Qemu virtual machine, which would let me do builds on my main workstation without needing to setup a cross compiling buildroot. This has the further advantage in that I could test automating full installations for real Raspberry Pis.


Because I really dislike having to lots of manual steps I automated the whole process. The scripts I used are on GitHub (https://github.com/pastcompute/pi_magic), use at your own risk etc.

There are three scripts: pi_qemu_build.sh which generates a pretend SD card image suitable for use by Qemu; pi_qemu_setup.sh which will SSH into a fresh image and perform second stage customisation; and pi_qemu_run.sh which launches the actual Qemu VM.

My work is based on that described at http://xecdesign.com/qemu-emulating-raspberry-pi-the-easy-way/.

Image Generation

The image generation works as follows:

  • start with an unzipped Raspbian SD card image downloaded from http://www.raspbian.org/
  • convert the image to a Qemu qcow2 image
  • extend the size out to 4GB to match the SD card
  • mount using Qemu NBD and extend the ext2 partition to the rest of the disk
  • Patch the image to work with Qemu quirks (more on that in a moment)
  • Create a snapshot so that changes made later inside Qemu can be rolled back if required

Now Raspbian is setup to load a DLL /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so, which needs to be disabled for Qemu; also it is set up to search for an MMC device so we need to make a symlink to access /dev/mmc* as /dev/sda inside Qemu instead.

Qemu Execution

The stock Raspberry Pi kernel wont work inside Qemu, so we need to launch with a different kernel.

I haven’t yet had time to include producing one in the build process.  Instead, I launch Qemu using the kernel downloadable from XEC Design.

Qemu is invoked using qemu-system-arm -kernel path/to/kernel -cpu arm1176 -m 256 -M versatilepb ..., so that the processor matches that used by the Raspberry Pi. The script also maps HTTP and SSH through to the VM so you can connect in from localhost, and runs using the snapshot.

This was tested on Debian Wheezy with qemu 1.7 from backports.

PI VM screenshot

PI VM screenshot

No responses yet

Unleashed GovHack – an Adelaide Adventure in Open Data

Aug 19 2014

Last month I attended Unleashed Govhack, our local contribution to the Australian GovHack hackathon.


Essentially GovHack is a chance for makers, hackers, designers, artists, and researchers to team up with government ‘data custodians’ and build proof of concept applications (web or mobile), software tools, video productions or presentations (data journalism) in a way that best illustrates how the vast amount of publically sourced data held by the government in trust for the community can be ‘unleashed’ for the benefit of the wider community. There was also the Machinery of Data competition, where participants combine data with physical manifestations.

Entrants form teams, and across the weekend define a concept and produce an application or a presentation, and in all cases a video to showcase the idea. Prizes are awarded both nationally, and for Adelaide and Mount Gambier specifically, for entries meeting various criteria such as “More Informed Adelaideians”, “Open Society Unleashed”, “Best entry by a University team, etc. A second aim is also to promote the small business / “startup” sector, so there are prizes related to business mentoring, for example.

Just a very, very small example of available data sets include: geospatial data of significant trees and also the Waite Arboretum, locations of recycle stations, air quality measurements, historic photos and images, City of Adelaide public wifi hotspots, locations of sightings of endangered species, to name but a few.

A new experience

I brought along with me my son who is now 13. He aspires to be an Author when he grows up, and also has a natural ability in maths as well as written language. I am encouraging him to have a backup plan, so at least he has something to write about! I thought this would be a wonderful opportunity to introduce him to the community side of the “digital revolution” – the Internet is about more than games, and computers about more than just Minecraft!

He charmed the volunteers, who were excited at the idea of having a school youth element attending. Together we both learned a lot, this was an especially intense weekend and by the Sunday afternoon during video production things got a bit tense at times; I need to learn to chill a bit, because he is quite capable of producing video! He has been creating presentations at school and some extra-curricular activities for a few years now, so a taste of data journalism was something he actually thought quite interesting.

In the end, he was fortunate enough to receive one of the weekend “Spirit” prizes, he did very well in talking to lots of people and generally participating, he gracefully received help from some members of Adelaide Hackerspace who helped him sort data on the Saturday,  and as we found out on the weekend, he was also awarded a commendation for the “Data Journalism” presentation that he created, where he examined correlations between endangered animal sightings and Australian natural disasters to inform policy makers when allocating funds related to endangered species protection. (His presentation was from a 13yo perspective, the previous paragraph is obviously my words!) You can see the final on the GovHack Hackerspace.

A crash course, or what not to do if you want to get the most out of the weekend!

For my part, I was really along for the ride, this time at least, having not participated in such an event before.  I am a professional software developer, but that is no help if you only leave yourself a day to attempt to write any real code!  I formed a team of my own with a friend from Adelaide Hackerspace, as well as having to be on my sons team as his guardian, but for various reasons it took us until halfway through Saturday to decide on a theme, and we also spent a lot of time on Sunday producing video, having to help Zachary as well as do our own.

From observations of other, successful teams, as well as the pain of experience, I have compiled a list of lessons fir first time / future participants:

  1. Most of important of all: decide what to build on Friday night! We dithered until way into Saturday so we were effectively a day behind other teams.
  2. Almost as important: don’t get carried away. Start small, stay focused, and then when you pull of the first concept you should have time to incrementally expand. It is easy to get very excited about all the possibilites, which is fine, but save the embellishments for the video so you get something finished.
  3. Become very familiar with your chosen video editor before the Govhack weekend
  4. If you intend to deploy a web application, become very familiar with your provider. Govhack actually supplied AWS vouchers, and luckily AWS was one area where I did have some recent experience so for my team at least, this was one area that did not cause frustration.
  5. Avoid large teams, so you avoid analysis paralysis.  It may be tempting to do a large group brainstorm, but then you do get carried away and cant keep to #2 above.  This also happened to us, which is why we didn’t really define our projects until Saturday when we split into smaller groups.
  6. Don’t bite off more than you can chew! Stay with your teams skillsets. See also #2
  7. For working with data ensure you (or some team members) have good familiarity with tools such as CSV conversion scripting, graphical tools such as QGIS, and scripting languages and tools for pulling data from PDF files
  8. Aim to have familiarity with SQL database systems, this can make life easier if you want to use a tool like PostGIS for geographic analysis
  9. If like to build web applications, get back up to speed before the weekend with a good rapid prototyping framework
  10. Decide how to host beforehand if you are planning a web app. CPanel will limit you to PHP or PERL so if you want to use Postgresql for geo features, or a Python backend you will probably need a VPS.

Obviously the later items in the list are more specific to programming skills.  Some of the items are more applicable if you have already organised a team before the night.  If you form a team late it can be hard to plan skill set coverage, in which case you may be better off aiming for a comprehensive and illuminating data journalism entry rather than an app.  It does depend on your specific  circumstances.

All in all it was a good weekend, both myself and my son met a lot of awesome new people, learned new skills and experiences and in a small way contributed to the community.

Finally a massive thanks to the volunteers who organised the whole thing!

There are photos up on Twitter or over on the official Facebook page.

No responses yet

Avoiding your own Goto Fail in a pinch

Jun 08 2014

(Note – this article particularly applies to C, C++ and Java like languages.)

Recently I came across some code that had the same kind of latent defect as was responsible for the now (in)famous ‘Goto Fail’ event [1]. So it was an excuse to write a blog article, I have been a bit slack busy lately.

OK. So ideally your project has decent revision control, with source code guidelines, code review and tools to make it easier to produce high quality code. This should make it easier to reduce the odds of your very own ‘Goto Fail’ event, right?

Well, here is what often happens in the real world:

  • there will be times the project is “experimental”, or done in a hurry and not intended to last (and just how many good ideas started off like that, indeed?)
  • very often, you have to maintain a legacy codebase
  • you need to “ship something yesterday” and haven’t been provided with budget to keep it properly maintained; sometimes the accountants win…

With Goto Fail in particular, part of the problem was not really the use of the much maligned goto statement in C, but rather lack of adherence to coding standards. In particular, a lack of braces that limit scope!

Many observers of course jumped on the ‘never use goto’ meme [2], but one thing I have come to learn in life, almost nothing is black and white. The use of goto is like any tool, in this case, one reserved for special situations. To see valid scenarios where goto has both readability and performance benefits take a look at the Linux kernel [3]; a well placed goto can remove a tangle of nested if/else/etc as well as provide a key performance improvement, when used properly.

Noting that the Linux kernel is not a normal application; the need for goto will still be extremely unlikely in most mundane applications.


Here is a quick and dirty method for auditing a code-base for possible “goto fail”-type defects.


We are looking for the following code fragments:

These are bad because inadvertent edits can produce:

This should be a big no-no!

Instead, we treat either of the following as OK:

This is actually tighter than the Google Style Guide [4]; consider it wise to plan for multiple developers editing future code, some who may be inexperienced or “just passing through” – on the theory of if they expend the least energy, if the braces are there, they will use them!

Method #1 – from a Linux / Unix shell

Extend the -name clause as required.

This works by finding all lines where the first non-whitespace is the if statement, and then reports if there is NO opening brace anywhere after on the same line.  It handles any whitespace combination.

Method #2 – the following regular expression may work in Microsoft Visual Studio

Caveat: I haven’t had a chance to test this ye, so if it is wrong, please let me know!


Now, these wont catch everything: if anything ‘creative’ is happening inside a preprocessor macro for example; also this may trigger false positives if you have a string constant that includes the text " if (". It is also probably not resilient against nested statements on the same line, if your guidelines allow it (e.g. if (blah) { for (;;) would not be caught without modifying the expression.

The above expressions, as-is, will also produce many false positives, if your coding guideline allows the following:

I’ll leave it as an exercise for the reader to adjust the regular expression to suit… it should extend to more complex situations if required. You could also use sed or pcregrep or even perl -efor example.

if you have to remediate a very large code base, you can redirect the output to a file and measure progression over time. You could script out any false positives as well.

[1] http://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-goto-fail-apples-ssl-bug-explained-plus-an-unofficial-patch/

[2] http://en.wikipedia.org/wiki/Considered_Harmful

[3] https://web.archive.org/web/20130410044210/http://kerneltrap.org/node/553/2131

[4] http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Conditionals

No responses yet

« Newer posts Older posts »