Evaluating the security of OpenWRT (part 2) – bugfix

Sep 23 2014 Published by under blog

I had a bug applying the RELRO flag to busybox, this is fixed in GitHub now.

For some reason the build links the busybox binary a second time and I missed the flag.

Also an omission from my prior blog entry: uClibc has RELRO turned on in its configuration already in OpenWRT, so does not need flags passing through to it. However, it is failing to build its libraries with RELRO in all cases, in spite of the flag. This problem doesn’t happen in a standalone uClibc build from the latest uClibc trunk, but I haven’t scoped how to get uClibc trunk into OpenWRT. This may have been unclear they way I described it.

No responses yet

Evaluating the security of OpenWRT (part 2)

Sep 20 2014 Published by under infosec

In my last post I covered how I setup an OpenWRT build, to examine  a small subset of indicators of security of the firmware.

In this follow-up post we will examine in detail the analysis results of one of the indicators: specifically, the RELRO flag.

A first look – what are the defaults?

The analysis here is specific to the Barrier Breaker release of OpenWRT, but it should be noted that during experiments with the OpenWRT development trunk the results are much the same.

Before diving into RELRO, lets take a look at the overall default situation.

Here is the checksec report for the Carambola2 device (MIPS processor) build.
It is a sea of red…

Default checksec report for OpenWRT x86 buildThe ‘run as root’ errors can be ignored: those programs are actually absolute symbolic links which do not resolve in the host system. Relative symbolic links resolve correctly but are filtered out of the analysis.

The x86 build paints a similar picture:

owrt_cs_report_0_x86(Notably for x86, the NX flag is correctly set, but that is a topic for another time.)

Note, the rest of this post describes how to modify OpenWRT to enable RELRO.  There may be perfectly valid reasons to not enable the flag (for example, using RELRO may have a performance impact, and for a given system the adverse security risk may be judged low), so I have ensured that the suggested mitigation if applied remains a choice in the configuration menu of the system.  For the moment my patch also retains backward compatibility by defaulting to off.

Inside the OpenWRT build system

After a brief look at the build logs, the reason is obvious: the typical gcc linker command is missing the flags needed to enable RELRO:  -Wl,-z,relro -Wl,-z,now (or the direct linker equivalents, -z relro -z now)

What could be done to address this?

OpenWRT provides a hook for appending to the global compiler CFLAGS but there is no similar hook for the linker stage. We could add those flags to the global CFLAGS and they can in fact flow through to the linker for many programs, but that would also be redundant as the flags are irrelevant to the compiler.  In the end I decided I would modify the OpenWRT source to add a new global CONFIG option, which adds -Wl,-z,relro -Wl,-z,now to the global LDFLAGS instead.

The following patch achieves that (note, I have left out some of the help for brevity):

Having attched OpenWRT, and enabled the new flag, lets rebuild everything again and run another checksec scan.

owrt_cs_report_1_x86What a difference!

The results shown above are for x86, the picture is similar for the Carambola2 MIPS image.

The new results indicate that the RELRO flag is present on some binaries but not all of them. From this we can predict that some packages do not fully honour the global OpenWRT build system linker flags. I soon confirmed this**; the implication is that the new flag CONFIG_SECURITY_FORCE_RELRO  is useful, however, a caveat in the Kconfig help is required. In particular, a statement to the effect that the efficacy depends on proper coding of OpenWRT packages (with ideally all packages maintained by the project being fixed to honour the flag.)

** For example: the package that builds libnl-tiny.so does not pass LDFLAGS through to the linker; this and some other base system packages needed patching to get complete coverage.  it is likely that there are other packages that I did not have selected that may also need tweaking.

Another notable package is busybox.  Busybox it turns out uses ld directly for linking, instead of indirectly via gcc, and thus requires the flags in the pure form -z relro -z now. (The busybox OpenWRT package Makefile also happens to treat the global TARGET_LDFLAGS differently from the TARGET_CFLAGS although I am unsure if this is a bug; but that turned out to be a red-herring.) Oddly, this solution worked for MIPS when I tried it previously, but is presently not successful for the x86 build, so further investigation is needed here; possibly I incorrectly noted the fix in previous experiments.

Fun and Games with uClibc and busybox

The other recalcitrant is the uClibc library. I spent quite a bit time trying to work out why this was not working, especially having  confirmed with verbose logging that the flags are being applied as expected. Along the way I learned that uClibc already has its own apply RELRO config item, which was already enabled. Even more oddly, RELRO is present on some uClibc libraries and not others, that as far as I could tell were being linked with identical linker flag sets.

After some digging I discovered hints of bugs related to RELRO in various versions of binutils, so I further patched OpenWRT to use the very latest binutils release.   However that made no difference.  At this point I took a big diversion and spent some time building the latest uClibc externally, where I discovered that it built fine using the native toolchain of Debian Wheezy (including a much older binutils!)  After some discussion on the uClibc mailing list I have come to the conclusion that there may be a combination of problems, including the fact that uClibc in OpenWRT is a couple of years old (and additionally has a set of OpenWRT specific patches.)   I could go further and patch OpenWRT to use the trunk uClibc but then I would have to work through refreshing the set of patches which I really don’t have time or inclination to do, so for the moment I have deferred working on resolving this conundrum.  Eventually someone at OpenWRT may realise that uClibc has undergone a flurry of development in recent times and may bump to the more recent version.


Along the way, I discovered that Debian actually runs security scans across all packages in the distribution – take a look at https://lintian.debian.org/tags/hardening-no-relro.html.

It is worth noting that whenever changing any build-related flag it is worth cleaning and rebuilding the toolchain as well as the target packages and kernel; I found without doing this, flag changes such as the RELRO flag don’t fully take effect as expected.

For maximum verboseness, run with make V=csw although I had to dig through the code to find this out.

I was going to repeat all the testing against a third target, another  MIPS-based SOC the  RALINK 3530 but at this point I don’t really have the time or inclination, I am sure the results will be quite similar.  It would probably be useful to try with an ARM-based target as well.

I should also try repeating this experiment with MUSL, which is an alternative C library that OpenWRT can be built with.


Out of the box, OpenWRT has very limited coverage of the RELRO security mitigation in a standard firmware build.  By applying the suggested patches it is possible to bring OpenWRT up to a level of coverage, for RELRO, to that approaching a hardened Gentoo or Ubuntu distribution, with only a small subset of binaries missing the flag.


My Github account includes the repository openwrt-barrier-breaker-hardening. The following branch include the completed series of patches  mentioned above:  owrt_analysis_relro_everywhere  I hope it will remain possible to apply these changes against the official release for a while yet.

The patch that enables the latest binutils is not in that branch, but in this commit.

No responses yet

Evaluating the security of OpenWRT (part 1)

Sep 18 2014 Published by under infosec

I have recently spent  some time pondering the state of embedded system security.
I have been a long time user of OpenWRT, partly based on the premise that it should be “more secure” out of the box (provided attention is paid to routine hardening); but as the Infosec world keeps  on evolving, I thought I would take a closer look at the state of play.

OpenWRT – an embedded Linux distribution.

OpenWRT is a Linux distribution targeted as a replacement firmware for consumer home routers, having been named after the venerable Linksys WRT54g, but it is gaining a growing user base in the so-called Internet of Things space, as evidenced by devices such as the Carambola2, VoCore and WRTnode.

Now there are many, many areas that could be the focus of security related attention.  One useful reference is the Arch Linux security guide , which covers a broad array of mitigations, but for some reason I felt like diving into the deeper insides of the operating system. I decided it would be worthwhile to compare the security performance of the core of OpenWRT, specifically the Linux kernel plus the userspace toolchain, against other “mainstream” distributions.

The theory is that ensuring a baseline level of protection in the toolchain and the kernel will cut of a large number of potential root escalation vulnerabilities before they can get started.  To get started, take a look at the Gentoo Linux toolchain hardening guide.  This lists mitigations such as: Stack Smashing Protection (SSP), used to make it much harder for malware to take advantage of stack overrun situations; position independent code (PIC or PIE) that randomises the address of a program in memory when it is loaded; and so-on to other features such as Address Space Layout Randomisation (ASLR) in the kernel, all of these designed to halt entire classes of exploits in their tracks.

I wont repeat all the pros and cons of the various methods here.  It should be noted that these methods are not foolproof; new techniques such as Return Oriented Programming (ROP) can be used to circumvent these mitigations.  But defense-in-depth would appear to be an increasingly important strategy.  If a device can be made secure using these techniques it is probably ahead of 95% of the embedded market and more likely to avoid the larger number of automated attacks; perhaps this a bit like the old joke:

The first zebra said to the second zebra: “Why are you bothering to run? The cheetah is faster than us!”

To which the second zebra replied, “I only have to run faster than you!”

Analysis of just a few aspects.

There is an excellent tool “checksec.sh”, that can scan binaries in a Linux system and report their status against a variety of  security criteria.
The original author has retained his website but the script was last updated in November 2011. I found a more recently maintained version on GitHub featuring various enhancements (the maintainer even accepted a patch from me addressing a cosmetic bugfix.)

Initially I started using the tool to report on just the status of the following elements in binaries and libraries: RELRO, non-executable stack, SSP and PIC/PIE.

The basic procedure for using the tool against an OpenWRT build is straightforward enough:

  1. Configure the OpenWRT build and ensure the tar.gz target is enabled ( CONFIG_TARGET_ROOTFS_TARGZ=y )
  2. To expedite the testing I build OpenWRT with a pretty minimal set of packages
  3. Build the image as a tar.gz
  4. Unpack the tar.gz image to a temporary directory
  5. Run the tool

When repeating a test I made sure I cleaned out the toolchain and the target binaries, because I ahve been bitten by spuriuos results caused by changes to the configuration not propagating through all components.  This includes manually removing the staging_dir which may be missed by make clean. This made each build take around 20+ minutes on my quad core Phenom machine.

I used the following base configuration, only changing the target:

I repeated the test for three platform configurations initially (note, these are mutually exclusive choices) – the Carambola2 and rt305x are MIPS platforms.

There are several directories most likely to have binaries of interest, so of course I scanned them using a script, essentially consisting of:

The results were interesting.

(to be continued)

Further reading

[1] http://wiki.openwrt.org/doc/howto/secure.access
[2] https://wiki.archlinux.org/index.php/Security
[3] http://wiki.gentoo.org/wiki/Hardened/Toolchain
[4] http://blog.siphos.be/2011/07/high-level-explanation-on-some-binary-executable-security/
[5] http://www.trapkit.de/tools/checksec.html

No responses yet

OpenWRT WDS between legacy WRT54G and recent TP-Link devices

Mar 30 2014 Published by under howto

For a while now I had a multiple wifi routers all providing access points, and a connection to each other, using a feature called WDS. All of the routers run OpenWRT. Recently one of them died and everything kind of stopped working properly. I actually had the following configuration:

TP-LINK <--wired,bridged--> ASUS WL500G <--wireless,WDS,bridged--> Linksys WRT54G

Now this all worked because the conventional wisdom appears to be that WDS works when the device at each end is the same, i.e. Broadcom to Broadcom (which I had in the case of WL500G and WRT54G) and is problematic at best otherwise.

Documentation alluding to this includes:




So with the WL500G out of the picture, the TP-Link and WRT54G were no longer able to see each other, and I experienced a loss of connectivity from my workshop where the WRT54G was physically located.

However if you look under the covers a bit, it seems this configuration can be made to work, as suggested by the folk over at DD-WRT.

So with a bit of tweaking I was able to get WDS working between the TP-LINK and WRT54G as follows.

TP-LINK Router (WR1043ND, OpenWRT 10.03 Backfire)

Content of /etc/config/wireless:

WRT54G Router (WRT54G, OpenWRT 10.03 with legacy 2.4 kernel for broadcom wireless)

Content of /etc/config/wireless:


  • xx:xx:xx:xx:xx:xx is the MAC address of the WR1043ND, and yy:yy:yy:yy:yy:yy the MAC address of the WRT54G
  • Both devices have to be on the same channel
  • On the secondary router (WRT54G) you actually connect on the ssid LOCALSSID but all DHCP, etc. comes from the TP-LINK

Note also you need to ensure that DHCP on the LAN is disabled on the second router!

I am unsure why this configuration has apparently been problematic for some – perhaps it is quite router specific as to whether it works.

No responses yet

Achievement unlocked – OpenWRT device porting – and a quick and dirty level conversion

Apr 26 2013 Published by under howto, tech

Porting a Linux Router

So I just posted what I hope this time are properly formatted patches to the OpenWRT developers mailing list, a set of patches that add support to OpenWRT for the D-link DIR-632-A1 router.

You can see some of my progression on the OpenWRT forum.

This was quite a bit of a journey, as I had a few false starts and along the way crossed a number of different areas in the Linux kernel. The best thing was debugging bit-banging of device registers, which reminds me of hacking on my Commodore 64 many years ago. I have of course had occasion to do such at times professionally, but I have rediscovered my desire to do more low level hacking, and having recently received a Raspberry Pi as a gift and finding use for my leostick from the 2012 linux conf I expect to have plenty of opportunity when I manage to find some more spare time.

Along the way I extended significantly my knowledge of how to work with OpenWRT and build customised images, etc. I added a build guide for this router to the wiki pending acceptance of patches into the main stream; I think this cuts through the useful but verbose documentation elsewhere in the wiki.

When you need a USB-TTL RS232…

and you live 45m from Jaycar, and its 9pm Friday night anyway and you certainly don’t want to wit 3days for Internet delivery from your friendly Aussie electronics suppliers (hello Core electronics!) or a month for some Chinese no-brand on ebay… and you want to hack now!

A crucial part of hacking a router is access to the serial port.  These are always at logic level, and in the case of the DIR-632 at 3v3 not 5V, further complicating things.  However all I had handy was a USB-RS232 dongle running at proper RS-232 levels, of +/- 12V – so connecting this would be sure to let the smoke out of something…

Now it turns out I have a shedful of what most people would term junk.  But usefully this included a tube of MAX232 chips I acquired some years ago.  So first, find a handful of resistors and capacitors, and a birdnest later, one logic level to RS232 converter:


There is a leostick there, but it is being used to provide 5V :-) Originally I attempted to use it as a serial relay, but apparently there is a bug in the firmware in the model I have that causes problems with the d0/d1 serial port and thus all I could ever see was junk… (thanks to MarkJ for letting me know)

The circuit is basically as described here (1), but I used 2x 2u2 capactitors in parallel, not having enough 1uF (* I really get into going the whole bush-mechanic-electronics thing!)

However, this doesnt fix the whole problem – the MAX232 is powered by 5V, although I also later found out it may have worked at 3v3 anyway.

I suspect many of the younger maker crowd might not be aware of some trick that can be employed when necessary: you can of course buy a 2cm sized 3v3 to 5v level converter form Jaycar, or from China on ebay, but again I didnt want to wait, and I had a pile of BC547 transistors handy and basic knoweldge of how logic circuits are actually built.

Thus, two transistors and a few resistors later, a logic level converter:

              -+-----------------+- 5V
               |                 |
               |                 Z
               |                 Z
               |                 Z
               |                 Z        __
               |                 |     __|  |__
               Z                 |
               Z                 +--------->
     __        Z   __    __      |
  __|  |__     Z     |__|      | /
               |               |/
               |               |\
             | /               | \
     4k7     |/                  V
---/\/\/\----|                   |
             |\                  |
             | \                 |
               v                 |
               |                 |

(ASCII art FTW!)

Going he other way, I just ran three signal 1N4148 diodes in series, this dropped the line to about 3v5 which managed to not blow anything up with!


And several rather late nights^H^H^H early mornings later:


(1) http://www.scienceprog.com/alternatives-of-max232-in-low-budget-projects

No responses yet

Older posts »