2015 catchup #1

Jan 26 2015

At the conference in Auckland I had two presentations.

For the first time I managed to get a main conference talk accepted, actually it was a tutorial which goes for 90 minutes! It was a bit daunting beforehand, but after I finished, I realised I prefer the tutorial format over having to deliver a talk. I enjoy the interaction with the audience and the sharing of knowledge, and also not being the sole focus (and not having to remember exactly what to say so much!)

My tutorial was on Reverse Engineering with Radare2; the video (Youtube) and slides are linked from the conference presentation, and have the slides up on my personal landing page as well. Thanks to James for helping with a final practice run, its always good to have a typical candidate audience perspective beforehand.

I also did a shorter talk at the Open Hardware mini-conference, on hardening embedded Linux, using OpenWRT on devices like the carambola2 as an example. The video of the mini-conferences is a bit less polished due to resourcing, here I am on about 2/3 the way through. I was somewhat more flustered in my delivery due to late changes to some slides (see earlier blog article) and a problem with my laptop deciding to have thermal issues an hour before the talk. I managed to resolve these (thanks AndyK for your help!) but it put me off my mojo a bit unfortunately. The live demo I was quite happy with, it worked without issue, so perhaps the demo gods were appeased by my earlier mishaps… The final slides are here.

No responses yet – so much all the things!

Jan 21 2015

Well, I’ve said it before and not followed through, but I am intending to blog about various stuff from last weeks LCA over the next month or two.

One things about LCA of course is how much you learn. Especially when you stand up in front of a room to share something and discover errors in your own understanding! In my own case, I had a talk at the Open Hardware miniconf about some security things related to embedded devices. Literally an hour before I had a ping on twitter alterting me to a factual error in my blog, which was also loudly proclaimed in the talk I was about to deliver. Luckily it was only one slide, and the misunderstanding did not impact the rest of the talk (or for that matter, most of the offending blog article.) So I have updated the original blog article with a correction.

No responses yet

Experiments with hardening OpenWRT: applying the grsecurity patches

Dec 14 2014

A well known set of security enhancements to the Linux kernel is the grsecurity patch.  The grsecurity patch is a (large) patch that applies cleanly against selected supported stock Linux kernel versions. It brings with it PAX, which protects against various well known memory exploits, plus  a number of other hardening features including logging time and mount changes. In particular it enables features such as Non-executable stack (NX) on platforms that do not provide NX in hardware, such as MIPS devices and older x86.
UPDATE Unfortunately, NX protection for MIPS 32-bit devices is not in fact supported in software. This would be very useful. Whilst I was teaching myself I managed to mix things up, so be aware when reading the rest of this blog entry. Otherwise, the usefulness of grsecurity and the mechanism for patching into OpenWRT is still valid.

Note also, a more detailed procedure you can use to rebase the patches is at .

OpenWRT hardening

OpenWRT is a widely adopted embedded / router Linux distribution. It would benefit greatly from including grsecurity, in particular given most MIPS platforms do not support NX protection in hardware. However for a long time the differences between the OpenWRT kernel and the kernel revisions that grsecurity is supported on have been significant and would likely have taken an extreme effort to get working, let alone get working securely.

This is a shame, because there is malware targeted at consumer embedded routers, and it must only be a matter of time before OpenWRT is targeted.  OpenWRT is widely regarded as relatively secure compared to many consumer devices, at least if configured properly,  but eventually some bug will allow a remote binary to be dropped. It would be helpful if the system can be hardened and stay one step ahead of things.

The OpenWRT development trunk (destined to become the next release, ‘Chaos Calmer’ in due course) has recently migrated most devices to the 3.14 kernel tree.  Serendipidously this aligns with the long term supported grsecurity revision 3.14.  When I noticed this I figured I’d take a look at whether it was feasible to deploy grsecurity with OpenWRT.

Applying grsecurity – patch

In late November I pulled the latest OpenWRT sources and the kernel version was 3.14.25, which I noticed matched the current grsecurity stable branch 3.14.25

The grsecurity patch applies cleanly against a stock kernel, and OpenWRT starts with a stock kernel and then applies a series of patches designed to extend hardware support to many obscure embedded things not present in the mainline kernel, along with patches that reduce the memory footprint. Some of the general patches are pushed upstream but may not yet have been accepted, and some could be backports from later kernels.  Examples of generic patches  include a simplified crash report.

Anyway, I had two choices, and tried them both: apply grsecurity, then the OpenWRT patches; or start with the OpenWRT patched kernel.  In both cases there were a number of rejects, but there seemed to be less when I applied grsecurity last. I also decided this would be easier for me to support for myself going forward, a decision later validated successfully.

OpenWRT kernel patches are stored in two locations; generic patches applying against any platform, then platform specific patches.  My work is tested against the Carambola2, an embedded MIPS board supported by the ‘ar71xx’ platform in OpenWRT, so for my case, there were ar71xx patches.

To make life easy I wrote a script that would take a directory of OpenWRT kernel patches, apply to a git kernel repository and auto-commit. This allowed me to use gitg and git difftool to examine things efficiently.  It also worked well with using an external kernel tree to OpenWRT so I didnt have to worry yet about integrating patches into OpenWRT. This script is on github, it should be easily adaptable for other experiments.

(Note: to use an external tree, managed by git, use config options like the following:

There were four primary rejects that required fixing.  This involved inspecting each case and working out what OpenWRT had changed in the way. Generally, this was caused because one or the other had modified the end of the same structure or macro, but luckily it turned out nothing significant and I was able to easily reconcile things. The hardest was because OpenWRT modifies vmstat.c for MIPS and the same code was modified by grsecurity to add extra memory protections.  At this point I attempted to build the system, and discovered three other minor cases that broke the build. These mispatches essentially were due to movements in one or two lines, or new code using internal kernel API modified by grsecurity, and were also easily repaired.  The most difficult mispatch to understand was where OpenWRT rewrites the kernel module loader code, apparently to make better use of MIPS memory structures and it took me a little while to understand how to try and fix things.

The end result is on github at

Applying grsecurity – OpenWRT quirks

One strange bug that had to be worked around was some new dependency in the kernel build process, where extra tools that grsecurity adds were not being built in the correct order with other kernel prerequisites.

In the end I had to patch how OpenWRT builds the kernel to perform an extra ‘make olddefconfig‘ to sort things out.

I also had to run ‘make kernel_menuconfig‘ and turn on grsecurity.

As the system built, I eventually hit another problem area: building packages. This was a bit of an ‘OH-NO’ moment as I thought it had the potential to become a big rabbit hole. Luckily as it turned out, only one package was affected in the end: compat-wireless.  This package builds some extra user space tools and wifi drivers, and used a macro, ACCESS_ONCE, that was changed by grsecurity to be more secure; and required use of a new macro to make everything work again, ACCESS_ONE_RW. There were rather a number of calls to this macro, but luckily it turned out to be fixable using sed!

Booting OpenWRT with grsecurity – modules not loading

I was able to then complete an INITRAMFS image that I TFTP’d into my carambola2 via uboot.

Amazingly the system booted and provided me with a prompt.

I then discovered that no kernel modules were loading. A bit of digging and it turns out that a grsecurity option, CONFIG_GRKERNSEC_RANDSTRUCT  will auto-enable CONFIG_MODVERSIONS. One thing I learned at this point is that OpenWRT does not support CONFIG_MODVERSIONS=y, due to the way it packages modules with its packaging system. So an iteration later with the setting disabled, and everything appeared to be “working”

Testing OpenWRT with grsecurity

Of course, all this work is moot if we cant prove it works.

Easy to check is auditing. For example, we now had these messages:

However, the acid test would be enforcement of the NX flag. Here I used the code from to test incorrect memory protections. Result:


Revisiting Checksec, and tweaking PAX

In an earlier blog I wrote about experimenting with checksec.  Here I used it to double-check that the binaries were built with NX protection. MOst were, due to a patch I previously submitted to OpenWRT for MIPS. However, openssl was missing NX. It turns out that OpenSSL amongst everything else it has been discussed for this year, uses assembler in parts of the encryption code! I was able to fix this by adding the relevant linker ‘.note.GNU-stack‘ directive.

The PAX component can be tweaked using the paxctl command, so I had to build that with the OpenWRT toolchain to try it out. I discovered that it doesnt work for files on the JFFS2 partition, only in the ramdisk. Further to enable soft mode, you need to add a kernel boot command line argument. To do this for OpenWRT, edit a file called target/linux/$KERNEL_PLATFORM/generic/config-default where in my case, $KERNEL_PLATFORM is ar71xx

Moving Targets

Right in the middle of all this, OpenWRT bumped the kernel to 3.14.26. So I had to exercise a workflow in keeping the patch current.  As it happened the grsecuroty patch was also updated to 3.14.26 so I presume this made life easier.

After downloading the stock kernel and pulling the latest OpenWRT, I again re-created the patch series, then applied grsecurity 3.14.26.  The same four rejects were present again, so fingers crossed I cherry-picked all my work from 3.14.25 onto 3.14.26. As luck would have it this was one smooth rebase!

Recap of OpenWRT grsecurity caveats

  • CONFIG_GRKERNSEC_RANDSTRUCT is not compatible with the OpenWRT build system; using it will prevent modules loading
  • Some packages may need to be modified to support NX – generally, if these use assembly language and don’t use the proper linker directive.
  • For some reason paxctl only seems to work on files in /tmp not in the JFFS overlay. This is probably only a problem when debugging
  • Your experience with the debugger gdb will probably be sub-optimal unless you put the debug target on /tmp and use paxctl to mark it with exceptions


After concluding the above, I converted the change set from my local Linux working copy into a set of additional patches on OpenWRT and rebuilt everything to double check.

The branch ‘ar71xx-3.14.26-grsecurity’ in has all the work, along with some extra minor fixes I made to some other packages related to checksec scan results.

THIS MAY EXPLODE YOUR COMPUTER AND GET YOU POWNED! This has been working for me on one device with minimal testing and is just a proof of concept.

No responses yet

Running FreeBSD on the carambola2

Nov 30 2014

The carambola2 is a small module built around the Atheros AR9330 SOC. Manufactured by 8devices, it has 64MB RAM, 16MB flash, two Ethernet ports and a host of GPIO pins, some of which can be configured as i2c, SPI or i2s. The carambola2 is shipped with OpenWRT a Linux distribution targeted at small devices and as a replacement firmware for consumer routers.

I have previously presented on using the carambola2 at the Sysadmin Miniconference at LCA2014 (slides here, video here)

To try something different, I thought I’d take FreeBSD for a run on this board. This became an extensive learning exercise as I knew absolutely nothing about any of the *BSD distributions other than their unix heritage and that they use BSD type licenses instead of GPL for the kernel and most of the userland.


As with any of these things, there are a bunch of perceived or actual pros/cons between OpenWRT and FreeBSD.
Some of these I only discovered during this process.

My requirements included:

  • Being able to do a complete firmware build from source, which is possible for both OpenWRT and FreeBSD
  • Easy access to LED and GPIOs
  • Run the image from a RAM filesystem

Some pros/cons of either include:

  • The build system used is actually the standard build system for FreeBSD. You could probably build OpenWRT under OpenWRT but you usually dont.
  • The build system when used for cross compiling is functional but not as elegant as OpenWRT
  • OpenWRT builds actually take significantly longer from scratch for some reason
  • FreeBSD may be regarded as more secure under some circumstances, for some definition of security. But see below…
  • FreeBSD ships with two firewalls: pf, and ipfw. This adds quite a learning curve when doing a bottom up build like this.
  • Many common packages (the BSD “ports” system) do not cross-build correctly for mips under FreeBSD
  • FreeBSD 10.x ships with llvm as the default compiler but falls back to gcc for cross-building mips. But the gcc supplied with FreeBSD is only 4.2      O_o      Apparently this is for licensing reasons. This can be worked around  but I haven’t had time to try it yet. Ramifications of this likely include weaker security.

The carambola2 and the mips platform in general is actually reasonably well supported by FreeBSD, although it is treated as a ‘beta’. As to be expected, to build a firmware for FreeBSD requires a host FreeBSD system (at least this would be the path of least resistance!)

I built a virtual machine using kvm and was able to install FreeBSD 10.0 with minimum of hassle. I was pleasantly surprised at how easy it was to get up and running to OpenBox. FreeBSD has ‘pkg’ as a binary package manager and it worked similarly enough to ‘apt-get’, or ‘yum’ that I had a build machine up in about half an hour.
I did need to install bash and vim and gedit, some things are just too hard to give up!

Build Process

There appeared to be more than one way to cross-build, including the use of qemu as a build host inside FreeBSD, but rather than chasing turtles on this occasion I went with a tool called ‘freebsd-wifi-build‘. This was actually quite straightforward and produced me a working firmware out of the box, with some caveats. The firmware includes only binaries from the FreeBSD base userland, and only a limited subset at that. Initially it also wanted to build as the root user, which was both an annoyance and a shock to discover, although I soon resolved that problem; I hope to soon have patches accepted into the project to change the default to build as user!

In general, constructing a firmware using FreeBSD is more manual than OpenWRT, as it lacks the all-encompassing configuration of packages and the packaging infrastructure provided by OpenWRT opkg. It is more  similar to the Linux buildroot or even Gentoo.

The end result is a build script that automates the process I used to customise things, this is published at , for use as you see fit.

To flash the firmware, I used scp to copy the image to my host machine then using minicom to connect to the board, flash via tftp.
freebsd-wifi-build produces separate kernel and filesystem images, I was able to combine them into one file to simplify flashing.

Easy Wins

  • Network worked, with caveats
  • I was able to toggle the LED using ‘/dev/led’, although overall Linux has much better access to GPIO / LED hardware

Tweaking required along the way

  • FreeBSD swaps the ethernet ports relative to OpenWRT, and also by default configured them in switched mode instead of independently routed. I resolved this by rebuilding the firmware with the latest FreeBSD kernel from -CURRENT, which made the ethernet PHY configuration configurable.
  • As part of resolving that, I by chance discovered I could built the FreeBSD-release-10.1.0 userland and the bleeding edge FreeBSD-CURRENT kernel and have them cooperate together!
  • Only some of the FreeBSD ports easily build with the default cross compiler configuration. This limits the software that can be installed (at least, if built using the ports infrastructure)
  • Defaulting to gcc-4.2 means various important security measures, such as -fstack-protector, are disabled
  • I also had to tweak the default FreeBSD kernel configuration provided for the carambola2, to turn on the FAT filesystem (for USB transfer) and to enable additional GPIO
  • FreeBSD ignores uboot environment and arguments on the ar71xx platform, I managed to patch the kernel to support that


I’ll keep using OpenWRT on most of my devices for the forseeable future. But I will have a couple of FreeBSD gadgets thrown into the mix, just so I can keep learning new things, and also because ironically FreeBSD supports another router I have, the dir-632 ( I blogged about this device previously) which is not officially supported in the mainline OpenWRT and probably wont be anytime soon, but does work in a FreeBSD fork, zrouter.
It will also be interesting to compare the performance of pf against iptables.

Potential future exploration ideas: running Debian kFreeBSD on the carambola2.

No responses yet

(UPDATE) Evaluating the security of OpenWRT (part 3) adventures in NOEXECSTACK’land

Oct 04 2014

Of course, there are more things I had known but not fully internalised yet. Of course.

Many  MIPS architectures, and specifically, most common router architectures, don’t have hardware support for NX.


Yet. It surely wont be long.


My own feeling in this age of Heartbleed and Shellshock is we should get ahead of the curve if we can – if a distribution supports NX in the toolchain then when a newer SOC arrives there is one less thing that we can forget about.

If I had bothered to keep reading instead of hacking I may have rediscovered sooner. But then I would know significantly less about embedded toolchains and how to debug and patch them.  Anyway, a determined user could also cherry-pick emulated NX protection from PAX.
When they Google this problem they will at least find my work.


How else to  learn?  :-)

No responses yet

Older posts »