Evaluating the security of OpenWRT (part 3) adventures in NOEXECSTACK’land

Oct 04 2014

To recap our experiments to date, out of the box OpenWRT, with further digging, may appear to give the impression to have sporadic coverage of various Linux hardening measures without doing a bit of extra work. This in fact can be a false impression – see update – but for the uninitiated it could take a bit of digging to check!
One metric of interest not closely examined to date is the NOEXECSTACK attribute on executable binaries and libraries. When coupled with Kernel support, if enabled this disallows execution of code in the stack memory area of a program, thus preventing an entire class of vulnerabilities from working.  I mentioned NOEXECSTACK in passing previously; from the checksec report we saw that the x86 build has 100% coverage of NOEXECSTACK, whereas the MIPS build was almost completely lacking.

For a quick introduction to NOEXECSTACK, see http://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart.

Down the Toolchain Rabbit Hole

As far as a challenging detective exercise, this one was a bit of a doosy, for me at least.  Essentially  I had to turn the OpenWRT build system inside out to understand how it worked, and then the same with uClibc, so that I could learn where to begin to start.  After rather a few false starts, the culprit turned out to be somewhere completely different.

First, OpenWRT by default uses uClibc as the C library, which is the bedrock upon which the rest of the user space is built.  The C library is not just a standard package however. OpenWRT, like the majority of typical embedded Linux systems employs a  “toolchain” or “buildroot” architecture.  Simply put, the combines packages together the C/C++ compiler, the assembler, linker, the C library and various other core components in a way that the remainder of the firmware can be built without having knowledge of how this layer is put together.

This is a Good Thing as it turns out, especially when cross-compiling, i.e. when building the OpenWRT firmware for a CPU or platform (the TARGET) that is different from that where the firmware build happens (the HOST.)  Especially where everything is bootstrapped from source, as OpenWRT is.

The toolchain is combined from the following components:

  • binutils — provides the linker (ld) and the assembler and code for manipulating ELF files (Linux binaries) and object libraries
  • gcc — provides the C/C++ compiler and, often overlooked, libgcc, a library various “intrinsic” functions such as optimisations for certain C library functions, amongst others
  • A C library — in this case, uClibc
  • gdb — a debugger

Now all these elements need to be built in concert, and installed to the correct locations, and to complicate matters, the toolchain actually has multiple categories of output:

  • Programs that run on the HOST that produce programs and libraries that run on the TARGET (such as the cross compiler)
  • Programs that run on the on the TARGET (e.g. ldd, used for scanning dependencies)
  • Programs and libraries that run on the HOST to perform various tasks related to the above
  • Header files that are needed to build other programs that run on the HOST to perform various tasks related to the above
  • Header files that are needed to build programs and libraries that run on the TARGET
  • Even, programs that run on the on the TARGET  to produce programs and libraries that run on the TARGET (a target-hosted C compiler!)

Confused yet?

All this magic is managed by the OpenWRT build system in the following way:

  • The toolchain programs are unpacked and built individually under the build_dir/toolchain directory
  • The results of the toolchain build designed to run on the host under the staging_dir/toolchain
  • The partial toolchain under staging_dir is used to build the remaining items under build_dir which are finally installed to staging_dir/target/blah-rootfs
  • (this is an approximation, maybe build OpenWRT for yourself to find out all the accurate naming conventions )
  • The kernel headers are an intrinsic part of this because of the C library, so along the way a pass over the target Linux kernel source is required as well.

OpenWRT is flexible enough to allow the C compiler to be changed (e.g. between stock gcc 4.6 and LInaro gcc 4.8) , and the binutils version, and even switch the C library between different project implementations ( uClibc vs eglibc vs MUSL.)

OpenWRT fetches the sources for all these things, then applies a number of local patches, before building.

We will need to refer to this later.

Confirming the Problem and Fishing for Red Herrings.

The first thing to note is that x86 has no problem, but MIPS does, and I want to run OpenWRT on various embedded devices with MIPS SOC.  Without that I may never have bothered digging deeper!

Of course I dived in initially and took the naive brute force approach.  I patched OpenWRT to apply the override flag to the linker: -Wl,-z,noexecstack. This was a bit unthinking, after all x86 did not need this.

Doing this gave partial success. In fact most programs gained NOEXECSTACK, except for a chunk of the uClibc components, busybox, and tellingly as it turned out, libgcc_s.so. That is, core components used by nearly everything. Of course.

(Spoiler: modern Linux toolchain implementations actually enable NOEXECSTACK by DEFAULT for C code! Which was an important fact I forgot at this point! Silly me.)

At this point,  I managed to overlook libgcc_s.so and decided to focus on uClibc. This decision would greatly expand my knowledge of OpenWRT and uClibc and embedded built systems, and do nothing to solve the problem the proper way!

OpenWRT builds uClibc as a host package, which basically means it abuses Makefiles to generate a uClibc configuration file partly derived from the OpenWRT config file settings, and eventually call the uClibc top level makefile to build uClibc. This can only be done after building binutils and two of three stages of gcc.

At this point I still did not fully understand how NOEXECSTACK really should be employed, which is probably an artefact of working on this stuff late at night and not reading everything as carefully as I might have.  So I did the obvious and incorrect thing and worked out how to patch uClibc further to push the force -Wl,-z,noexecstack through it. What I had to do to do that could almost take another blog article, so I’ll skip it for brevity.  Anyway, this did not solve the problem.

Finally I turned on all the debug and examined the build:

make V=csw toolchain/uClibc/compile

(Aside: the documentation for OpenWRT mentions using V=s to turn on some verboseness, but to get the actual compiler and linker commands of the toolchain build you need the extra flags. I should probably try and update the OpenWRT wiki but I have invested so much time in this that I might have to leave that as an exercise for the reader)

All the libraries were being linked using the -Wl,-z,noexecstack flag, yet some still failed checksec. Argh!

I should also note that repeating this process over and over gets tedious, taking about 20 minutes to built the toolchain plus minimal target firmware on my quad core AMD Phenom. Dont delete the build_dir/host and staging_dir/host  directories or it doubles!

So something else was going on.

Upgrades and trampolines, or not.

I sought help from the uClibc developers mailing list, who suggested I first try using up to date software. This was a fair point, as OpenWRT is using a 2 year old release of uClibc and 1 year old release of binutils, etc.

This of course entailed having to learn how to patch OpenWRT to give me that choice.

So another week later, around work and family and life, I found some time to do this, and discovered that the problem persisted.

At this point I revisited the Gentoo hardening guide.  After some detective work I discovered that several MIPS assembler files inside of uClibc did not actually have the recommended code fragments. Aha! I thought. Incorrectly again, as I should have realised; uClibc has already thought of this and when NOEXECSTACK is configured, as it is for OpenWRT, uClibc passes a different flag to the assembler that has the effect of fixing NOEXECSTACK for assembler files. And of course after I patched about 17 .S files and waited another half hour, the checksec scan was still unsuccessful. Red herring again!

I started to get desperate when I read about some compiler systems that use something called a ‘trampolline’. So I went back to the mailing uClibc  list.

At this point I would like to thank the uClibc developers for being so patient with me, as the solution was now near at hand.

Cutting edge patches and a wrinkle in time.

One of the uClibc developers pointed me to a patch posted on the gcc mailing list. As fate would happen, dated 10 September 2014, which was _after_ I started on these investigations.  This actually went full circle back to libgcc_s.so which was the small library I passed over to focus on uClibc.  This target library itself has some assembly files, which were neither built with the noexecstack option nor including the Gentoo-suggested assembly pragma. This patch also applies on gcc, not on uClibc, and of course was outside of binutils which was the other component I had to upgrade.  The fact that libgcc_s.so was not clean should maybe have pointed me to look at gcc, and it did cross my mind. But we all have to learn somehow.  Without all the above I would be the poorer for my knowledge of the internals of all these systems.

So I applied this patch, and finally, bliss, a sea of green NX enabled flags. Without in the end any modifications actually required to the older version of uClibc used inside OpenWRT.

This even fixed busybox NX status without modification.  So empirically this confirms what I read previously and also overlooked to my detriment, that being NOEXECSTACK is aggregated from all linked libraries: if one is missing it pollutes the lot.

Fixed NOEXECSTACK on uClibc


Now I have to package the patch so that it can be submitted to OpenWRT, initially against Barrier Breaker given that has just been released!

Then I will probably need to repeat it against all the supported gcc versions and submit separate patches for those. That will get a bit tedious, thankfully I can start a test build then go away…

No responses yet

Evaluating the security of OpenWRT (part 2) – bugfix

Sep 23 2014

I had a bug applying the RELRO flag to busybox, this is fixed in GitHub now.

For some reason the build links the busybox binary a second time and I missed the flag.

Also an omission from my prior blog entry: uClibc has RELRO turned on in its configuration already in OpenWRT, so does not need flags passing through to it. However, it is failing to build its libraries with RELRO in all cases, in spite of the flag. This problem doesn’t happen in a standalone uClibc build from the latest uClibc trunk, but I haven’t scoped how to get uClibc trunk into OpenWRT. This may have been unclear they way I described it.

No responses yet

Evaluating the security of OpenWRT (part 2)

Sep 20 2014

In my last post I covered how I setup an OpenWRT build, to examine  a small subset of indicators of security of the firmware.

In this follow-up post we will examine in detail the analysis results of one of the indicators: specifically, the RELRO flag.

A first look – what are the defaults?

The analysis here is specific to the Barrier Breaker release of OpenWRT, but it should be noted that during experiments with the OpenWRT development trunk the results are much the same.

Before diving into RELRO, lets take a look at the overall default situation.

Here is the checksec report for the Carambola2 device (MIPS processor) build.
It is a sea of red…

Default checksec report for OpenWRT x86 buildThe ‘run as root’ errors can be ignored: those programs are actually absolute symbolic links which do not resolve in the host system. Relative symbolic links resolve correctly but are filtered out of the analysis.

The x86 build paints a similar picture:

owrt_cs_report_0_x86(Notably for x86, the NX flag is correctly set, but that is a topic for another time.)

Note, the rest of this post describes how to modify OpenWRT to enable RELRO.  There may be perfectly valid reasons to not enable the flag (for example, using RELRO may have a performance impact, and for a given system the adverse security risk may be judged low), so I have ensured that the suggested mitigation if applied remains a choice in the configuration menu of the system.  For the moment my patch also retains backward compatibility by defaulting to off.

Inside the OpenWRT build system

After a brief look at the build logs, the reason is obvious: the typical gcc linker command is missing the flags needed to enable RELRO:  -Wl,-z,relro -Wl,-z,now (or the direct linker equivalents, -z relro -z now)

What could be done to address this?

OpenWRT provides a hook for appending to the global compiler CFLAGS but there is no similar hook for the linker stage. We could add those flags to the global CFLAGS and they can in fact flow through to the linker for many programs, but that would also be redundant as the flags are irrelevant to the compiler.  In the end I decided I would modify the OpenWRT source to add a new global CONFIG option, which adds -Wl,-z,relro -Wl,-z,now to the global LDFLAGS instead.

The following patch achieves that (note, I have left out some of the help for brevity):

Having attched OpenWRT, and enabled the new flag, lets rebuild everything again and run another checksec scan.

owrt_cs_report_1_x86What a difference!

The results shown above are for x86, the picture is similar for the Carambola2 MIPS image.

The new results indicate that the RELRO flag is present on some binaries but not all of them. From this we can predict that some packages do not fully honour the global OpenWRT build system linker flags. I soon confirmed this**; the implication is that the new flag CONFIG_SECURITY_FORCE_RELRO  is useful, however, a caveat in the Kconfig help is required. In particular, a statement to the effect that the efficacy depends on proper coding of OpenWRT packages (with ideally all packages maintained by the project being fixed to honour the flag.)

** For example: the package that builds libnl-tiny.so does not pass LDFLAGS through to the linker; this and some other base system packages needed patching to get complete coverage.  it is likely that there are other packages that I did not have selected that may also need tweaking.

Another notable package is busybox.  Busybox it turns out uses ld directly for linking, instead of indirectly via gcc, and thus requires the flags in the pure form -z relro -z now. (The busybox OpenWRT package Makefile also happens to treat the global TARGET_LDFLAGS differently from the TARGET_CFLAGS although I am unsure if this is a bug; but that turned out to be a red-herring.) Oddly, this solution worked for MIPS when I tried it previously, but is presently not successful for the x86 build, so further investigation is needed here; possibly I incorrectly noted the fix in previous experiments.

Fun and Games with uClibc and busybox

The other recalcitrant is the uClibc library. I spent quite a bit time trying to work out why this was not working, especially having  confirmed with verbose logging that the flags are being applied as expected. Along the way I learned that uClibc already has its own apply RELRO config item, which was already enabled. Even more oddly, RELRO is present on some uClibc libraries and not others, that as far as I could tell were being linked with identical linker flag sets.

After some digging I discovered hints of bugs related to RELRO in various versions of binutils, so I further patched OpenWRT to use the very latest binutils release.   However that made no difference.  At this point I took a big diversion and spent some time building the latest uClibc externally, where I discovered that it built fine using the native toolchain of Debian Wheezy (including a much older binutils!)  After some discussion on the uClibc mailing list I have come to the conclusion that there may be a combination of problems, including the fact that uClibc in OpenWRT is a couple of years old (and additionally has a set of OpenWRT specific patches.)   I could go further and patch OpenWRT to use the trunk uClibc but then I would have to work through refreshing the set of patches which I really don’t have time or inclination to do, so for the moment I have deferred working on resolving this conundrum.  Eventually someone at OpenWRT may realise that uClibc has undergone a flurry of development in recent times and may bump to the more recent version.


Along the way, I discovered that Debian actually runs security scans across all packages in the distribution – take a look at https://lintian.debian.org/tags/hardening-no-relro.html.

It is worth noting that whenever changing any build-related flag it is worth cleaning and rebuilding the toolchain as well as the target packages and kernel; I found without doing this, flag changes such as the RELRO flag don’t fully take effect as expected.

For maximum verboseness, run with make V=csw although I had to dig through the code to find this out.

I was going to repeat all the testing against a third target, another  MIPS-based SOC the  RALINK 3530 but at this point I don’t really have the time or inclination, I am sure the results will be quite similar.  It would probably be useful to try with an ARM-based target as well.

I should also try repeating this experiment with MUSL, which is an alternative C library that OpenWRT can be built with.


Out of the box, OpenWRT has very limited coverage of the RELRO security mitigation in a standard firmware build.  By applying the suggested patches it is possible to bring OpenWRT up to a level of coverage, for RELRO, to that approaching a hardened Gentoo or Ubuntu distribution, with only a small subset of binaries missing the flag.


My Github account includes the repository openwrt-barrier-breaker-hardening. The following branch include the completed series of patches  mentioned above:  owrt_analysis_relro_everywhere  I hope it will remain possible to apply these changes against the official release for a while yet.

The patch that enables the latest binutils is not in that branch, but in this commit.

No responses yet

Evaluating the security of OpenWRT (part 1)

Sep 18 2014

I have recently spent  some time pondering the state of embedded system security.
I have been a long time user of OpenWRT, partly based on the premise that it should be “more secure” out of the box (provided attention is paid to routine hardening); but as the Infosec world keeps  on evolving, I thought I would take a closer look at the state of play.

OpenWRT – an embedded Linux distribution.

OpenWRT is a Linux distribution targeted as a replacement firmware for consumer home routers, having been named after the venerable Linksys WRT54g, but it is gaining a growing user base in the so-called Internet of Things space, as evidenced by devices such as the Carambola2, VoCore and WRTnode.

Now there are many, many areas that could be the focus of security related attention.  One useful reference is the Arch Linux security guide , which covers a broad array of mitigations, but for some reason I felt like diving into the deeper insides of the operating system. I decided it would be worthwhile to compare the security performance of the core of OpenWRT, specifically the Linux kernel plus the userspace toolchain, against other “mainstream” distributions.

The theory is that ensuring a baseline level of protection in the toolchain and the kernel will cut of a large number of potential root escalation vulnerabilities before they can get started.  To get started, take a look at the Gentoo Linux toolchain hardening guide.  This lists mitigations such as: Stack Smashing Protection (SSP), used to make it much harder for malware to take advantage of stack overrun situations; position independent code (PIC or PIE) that randomises the address of a program in memory when it is loaded; and so-on to other features such as Address Space Layout Randomisation (ASLR) in the kernel, all of these designed to halt entire classes of exploits in their tracks.

I wont repeat all the pros and cons of the various methods here.  It should be noted that these methods are not foolproof; new techniques such as Return Oriented Programming (ROP) can be used to circumvent these mitigations.  But defense-in-depth would appear to be an increasingly important strategy.  If a device can be made secure using these techniques it is probably ahead of 95% of the embedded market and more likely to avoid the larger number of automated attacks; perhaps this a bit like the old joke:

The first zebra said to the second zebra: “Why are you bothering to run? The cheetah is faster than us!”

To which the second zebra replied, “I only have to run faster than you!”

Analysis of just a few aspects.

There is an excellent tool “checksec.sh”, that can scan binaries in a Linux system and report their status against a variety of  security criteria.
The original author has retained his website but the script was last updated in November 2011. I found a more recently maintained version on GitHub featuring various enhancements (the maintainer even accepted a patch from me addressing a cosmetic bugfix.)

Initially I started using the tool to report on just the status of the following elements in binaries and libraries: RELRO, non-executable stack, SSP and PIC/PIE.

The basic procedure for using the tool against an OpenWRT build is straightforward enough:

  1. Configure the OpenWRT build and ensure the tar.gz target is enabled ( CONFIG_TARGET_ROOTFS_TARGZ=y )
  2. To expedite the testing I build OpenWRT with a pretty minimal set of packages
  3. Build the image as a tar.gz
  4. Unpack the tar.gz image to a temporary directory
  5. Run the tool

When repeating a test I made sure I cleaned out the toolchain and the target binaries, because I ahve been bitten by spuriuos results caused by changes to the configuration not propagating through all components.  This includes manually removing the staging_dir which may be missed by make clean. This made each build take around 20+ minutes on my quad core Phenom machine.

I used the following base configuration, only changing the target:

I repeated the test for three platform configurations initially (note, these are mutually exclusive choices) – the Carambola2 and rt305x are MIPS platforms.

There are several directories most likely to have binaries of interest, so of course I scanned them using a script, essentially consisting of:

The results were interesting.

(to be continued)

Further reading

[1] http://wiki.openwrt.org/doc/howto/secure.access
[2] https://wiki.archlinux.org/index.php/Security
[3] http://wiki.gentoo.org/wiki/Hardened/Toolchain
[4] http://blog.siphos.be/2011/07/high-level-explanation-on-some-binary-executable-security/
[5] http://www.trapkit.de/tools/checksec.html

No responses yet

Raspberry Pi Virtual Machine Automation

Aug 23 2014

Several months ago now I was doing some development for Raspberry Pi. I guess that shows how busy with life things have been. (I have a backlog of lots of things I would like to blog about that didn’t get blogged yet, pardon my grammar!)

Now the Pi runs on an SD card and it some things start to get very tedious after a while, write performance is not exactly fast, and doing extensive work would probably start to wear cards a bit fast.

So I looked into running a Raspberry Pi Qemu virtual machine, which would let me do builds on my main workstation without needing to setup a cross compiling buildroot. This has the further advantage in that I could test automating full installations for real Raspberry Pis.


Because I really dislike having to lots of manual steps I automated the whole process. The scripts I used are on GitHub (https://github.com/pastcompute/pi_magic), use at your own risk etc.

There are three scripts: pi_qemu_build.sh which generates a pretend SD card image suitable for use by Qemu; pi_qemu_setup.sh which will SSH into a fresh image and perform second stage customisation; and pi_qemu_run.sh which launches the actual Qemu VM.

My work is based on that described at http://xecdesign.com/qemu-emulating-raspberry-pi-the-easy-way/.

Image Generation

The image generation works as follows:

  • start with an unzipped Raspbian SD card image downloaded from http://www.raspbian.org/
  • convert the image to a Qemu qcow2 image
  • extend the size out to 4GB to match the SD card
  • mount using Qemu NBD and extend the ext2 partition to the rest of the disk
  • Patch the image to work with Qemu quirks (more on that in a moment)
  • Create a snapshot so that changes made later inside Qemu can be rolled back if required

Now Raspbian is setup to load a DLL /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so, which needs to be disabled for Qemu; also it is set up to search for an MMC device so we need to make a symlink to access /dev/mmc* as /dev/sda inside Qemu instead.

Qemu Execution

The stock Raspberry Pi kernel wont work inside Qemu, so we need to launch with a different kernel.

I haven’t yet had time to include producing one in the build process.  Instead, I launch Qemu using the kernel downloadable from XEC Design.

Qemu is invoked using qemu-system-arm -kernel path/to/kernel -cpu arm1176 -m 256 -M versatilepb ..., so that the processor matches that used by the Raspberry Pi. The script also maps HTTP and SSH through to the VM so you can connect in from localhost, and runs using the snapshot.

This was tested on Debian Wheezy with qemu 1.7 from backports.

PI VM screenshot

PI VM screenshot

No responses yet

« Newer posts Older posts »