Experiments with hardening OpenWRT: applying the grsecurity patches

Dec 14 2014 Published by under infosec

A well known set of security enhancements to the Linux kernel is the grsecurity patch.  The grsecurity patch is a (large) patch that applies cleanly against selected supported stock Linux kernel versions. It brings with it PAX, which protects against various well known memory exploits, plus  a number of other hardening features including logging time and mount changes. In particular it enables features such as Non-executable stack (NX) on platforms that do not provide NX in hardware, such as MIPS devices and older x86.

OpenWRT hardening

OpenWRT is a widely adopted embedded / router Linux distribution. It would benefit greatly from including grsecurity, in particular given most MIPS platforms do not support NX protection in hardware. However for a long time the differences between the OpenWRT kernel and the kernel revisions that grsecurity is supported on have been significant and would likely have taken an extreme effort to get working, let alone get working securely.

This is a shame, because there is malware targeted at consumer embedded routers, and it must only be a matter of time before OpenWRT is targeted.  OpenWRT is widely regarded as relatively secure compared to many consumer devices, at least if configured properly,  but eventually some bug will allow a remote binary to be dropped. It would be helpful if the system can be hardened and stay one step ahead of things.

The OpenWRT development trunk (destined to become the next release, ‘Chaos Calmer’ in due course) has recently migrated most devices to the 3.14 kernel tree.  Serendipidously this aligns with the long term supported grsecurity revision 3.14.  When I noticed this I figured I’d take a look at whether it was feasible to deploy grsecurity with OpenWRT.

Applying grsecurity – patch

In late November I pulled the latest OpenWRT sources and the kernel version was 3.14.25, which I noticed matched the current grsecurity stable branch 3.14.25

The grsecurity patch applies cleanly against a stock kernel, and OpenWRT starts with a stock kernel and then applies a series of patches designed to extend hardware support to many obscure embedded things not present in the mainline kernel, along with patches that reduce the memory footprint. Some of the general patches are pushed upstream but may not yet have been accepted, and some could be backports from later kernels.  Examples of generic patches  include a simplified crash report.

Anyway, I had two choices, and tried them both: apply grsecurity, then the OpenWRT patches; or start with the OpenWRT patched kernel.  In both cases there were a number of rejects, but there seemed to be less when I applied grsecurity last. I also decided this would be easier for me to support for myself going forward, a decision later validated successfully.

OpenWRT kernel patches are stored in two locations; generic patches applying against any platform, then platform specific patches.  My work is tested against the Carambola2, an embedded MIPS board supported by the ‘ar71xx’ platform in OpenWRT, so for my case, there were ar71xx patches.

To make life easy I wrote a script that would take a directory of OpenWRT kernel patches, apply to a git kernel repository and auto-commit. This allowed me to use gitg and git difftool to examine things efficiently.  It also worked well with using an external kernel tree to OpenWRT so I didnt have to worry yet about integrating patches into OpenWRT. This script is on github, it should be easily adaptable for other experiments.

(Note: to use an external tree, managed by git, use config options like the following:

There were four primary rejects that required fixing.  This involved inspecting each case and working out what OpenWRT had changed in the way. Generally, this was caused because one or the other had modified the end of the same structure or macro, but luckily it turned out nothing significant and I was able to easily reconcile things. The hardest was because OpenWRT modifies vmstat.c for MIPS and the same code was modified by grsecurity to add extra memory protections.  At this point I attempted to build the system, and discovered three other minor cases that broke the build. These mispatches essentially were due to movements in one or two lines, or new code using internal kernel API modified by grsecurity, and were also easily repaired.  The most difficult mispatch to understand was where OpenWRT rewrites the kernel module loader code, apparently to make better use of MIPS memory structures and it took me a little while to understand how to try and fix things.

The end result is on github at https://github.com/pastcompute/openwrt-cc-linux-3.14.x-grsecurity

Applying grsecurity – OpenWRT quirks

One strange bug that had to be worked around was some new dependency in the kernel build process, where extra tools that grsecurity adds were not being built in the correct order with other kernel prerequisites.

In the end I had to patch how OpenWRT builds the kernel to perform an extra ‘make olddefconfig‘ to sort things out.

I also had to run ‘make kernel_menuconfig‘ and turn on grsecurity.

As the system built, I eventually hit another problem area: building packages. This was a bit of an ‘OH-NO’ moment as I thought it had the potential to become a big rabbit hole. Luckily as it turned out, only one package was affected in the end: compat-wireless.  This package builds some extra user space tools and wifi drivers, and used a macro, ACCESS_ONCE, that was changed by grsecurity to be more secure; and required use of a new macro to make everything work again, ACCESS_ONE_RW. There were rather a number of calls to this macro, but luckily it turned out to be fixable using sed!

Booting OpenWRT with grsecurity – modules not loading

I was able to then complete an INITRAMFS image that I TFTP’d into my carambola2 via uboot.

Amazingly the system booted and provided me with a prompt.

I then discovered that no kernel modules were loading. A bit of digging and it turns out that a grsecurity option, CONFIG_GRKERNSEC_RANDSTRUCT  will auto-enable CONFIG_MODVERSIONS. One thing I learned at this point is that OpenWRT does not support CONFIG_MODVERSIONS=y, due to the way it packages modules with its packaging system. So an iteration later with the setting disabled, and everything appeared to be “working”

Testing OpenWRT with grsecurity

Of course, all this work is moot if we cant prove it works.

Easy to check is auditing. For example, we now had these messages:

However, the acid test would be enforcement of the NX flag. Here I used the code from http://wiki.gentoo.org/wiki/Hardened/PaX_Quickstart to test incorrect memory protections. Result:

Success!

Revisiting Checksec, and tweaking PAX

In an earlier blog I wrote about experimenting with checksec.  Here I used it to double-check that the binaries were built with NX protection. MOst were, due to a patch I previously submitted to OpenWRT for MIPS. However, openssl was missing NX. It turns out that OpenSSL amongst everything else it has been discussed for this year, uses assembler in parts of the encryption code! I was able to fix this by adding the relevant linker ‘.note.GNU-stack‘ directive.

The PAX component can be tweaked using the paxctl command, so I had to build that with the OpenWRT toolchain to try it out. I discovered that it doesnt work for files on the JFFS2 partition, only in the ramdisk. Further to enable soft mode, you need to add a kernel boot command line argument. To do this for OpenWRT, edit a file called target/linux/$KERNEL_PLATFORM/generic/config-default where in my case, $KERNEL_PLATFORM is ar71xx

Moving Targets

Right in the middle of all this, OpenWRT bumped the kernel to 3.14.26. So I had to exercise a workflow in keeping the patch current.  As it happened the grsecuroty patch was also updated to 3.14.26 so I presume this made life easier.

After downloading the stock kernel and pulling the latest OpenWRT, I again re-created the patch series, then applied grsecurity 3.14.26.  The same four rejects were present again, so fingers crossed I cherry-picked all my work from 3.14.25 onto 3.14.26. As luck would have it this was one smooth rebase!

Recap of OpenWRT grsecurity caveats

  • CONFIG_GRKERNSEC_RANDSTRUCT is not compatible with the OpenWRT build system; using it will prevent modules loading
  • Some packages may need to be modified to support NX – generally, if these use assembly language and don’t use the proper linker directive.
  • For some reason paxctl only seems to work on files in /tmp not in the JFFS overlay. This is probably only a problem when debugging
  • Your experience with the debugger gdb will probably be sub-optimal unless you put the debug target on /tmp and use paxctl to mark it with exceptions

Summary

After concluding the above, I converted the change set from my local Linux working copy into a set of additional patches on OpenWRT and rebuilt everything to double check.

The branch ‘ar71xx-3.14.26-grsecurity’ in https://github.com/pastcompute/openwrt-cc-ar71xx-hardened has all the work, along with some extra minor fixes I made to some other packages related to checksec scan results.

THIS MAY EXPLODE YOUR COMPUTER AND GET YOU POWNED! This has been working for me on one device with minimal testing and is just a proof of concept.

No responses yet

(UPDATE) Evaluating the security of OpenWRT (part 3) adventures in NOEXECSTACK’land

Oct 04 2014 Published by under infosec, linux

Of course, there are more things I had known but not fully internalised yet. Of course.

Many  MIPS architectures, and specifically, most common router architectures, don’t have hardware support for NX.

 

Yet. It surely wont be long.

 

My own feeling in this age of Heartbleed and Shellshock is we should get ahead of the curve if we can – if a distribution supports NX in the toolchain then when a newer SOC arrives there is one less thing that we can forget about.

If I had bothered to keep reading instead of hacking I may have rediscovered sooner. But then I would know significantly less about embedded toolchains and how to debug and patch them.  Anyway, a determined user could also cherry-pick emulated NX protection from PAX.
When they Google this problem they will at least find my work.

 

How else to  learn?  :-)

No responses yet

Evaluating the security of OpenWRT (part 3) adventures in NOEXECSTACK’land

Oct 04 2014 Published by under infosec, linux

To recap our experiments to date, out of the box OpenWRT, with further digging, may appear to give the impression to have sporadic coverage of various Linux hardening measures without doing a bit of extra work. This in fact can be a false impression – see update – but for the uninitiated it could take a bit of digging to check!
One metric of interest not closely examined to date is the NOEXECSTACK attribute on executable binaries and libraries. When coupled with Kernel support, if enabled this disallows execution of code in the stack memory area of a program, thus preventing an entire class of vulnerabilities from working.  I mentioned NOEXECSTACK in passing previously; from the checksec report we saw that the x86 build has 100% coverage of NOEXECSTACK, whereas the MIPS build was almost completely lacking.

For a quick introduction to NOEXECSTACK, see http://wiki.gentoo.org/wiki/Hardened/GNU_stack_quickstart.

Down the Toolchain Rabbit Hole

As far as a challenging detective exercise, this one was a bit of a doosy, for me at least.  Essentially  I had to turn the OpenWRT build system inside out to understand how it worked, and then the same with uClibc, so that I could learn where to begin to start.  After rather a few false starts, the culprit turned out to be somewhere completely different.

First, OpenWRT by default uses uClibc as the C library, which is the bedrock upon which the rest of the user space is built.  The C library is not just a standard package however. OpenWRT, like the majority of typical embedded Linux systems employs a  “toolchain” or “buildroot” architecture.  Simply put, the combines packages together the C/C++ compiler, the assembler, linker, the C library and various other core components in a way that the remainder of the firmware can be built without having knowledge of how this layer is put together.

This is a Good Thing as it turns out, especially when cross-compiling, i.e. when building the OpenWRT firmware for a CPU or platform (the TARGET) that is different from that where the firmware build happens (the HOST.)  Especially where everything is bootstrapped from source, as OpenWRT is.

The toolchain is combined from the following components:

  • binutils — provides the linker (ld) and the assembler and code for manipulating ELF files (Linux binaries) and object libraries
  • gcc — provides the C/C++ compiler and, often overlooked, libgcc, a library various “intrinsic” functions such as optimisations for certain C library functions, amongst others
  • A C library — in this case, uClibc
  • gdb — a debugger

Now all these elements need to be built in concert, and installed to the correct locations, and to complicate matters, the toolchain actually has multiple categories of output:

  • Programs that run on the HOST that produce programs and libraries that run on the TARGET (such as the cross compiler)
  • Programs that run on the on the TARGET (e.g. ldd, used for scanning dependencies)
  • Programs and libraries that run on the HOST to perform various tasks related to the above
  • Header files that are needed to build other programs that run on the HOST to perform various tasks related to the above
  • Header files that are needed to build programs and libraries that run on the TARGET
  • Even, programs that run on the on the TARGET  to produce programs and libraries that run on the TARGET (a target-hosted C compiler!)

Confused yet?

All this magic is managed by the OpenWRT build system in the following way:

  • The toolchain programs are unpacked and built individually under the build_dir/toolchain directory
  • The results of the toolchain build designed to run on the host under the staging_dir/toolchain
  • The partial toolchain under staging_dir is used to build the remaining items under build_dir which are finally installed to staging_dir/target/blah-rootfs
  • (this is an approximation, maybe build OpenWRT for yourself to find out all the accurate naming conventions )
  • The kernel headers are an intrinsic part of this because of the C library, so along the way a pass over the target Linux kernel source is required as well.

OpenWRT is flexible enough to allow the C compiler to be changed (e.g. between stock gcc 4.6 and LInaro gcc 4.8) , and the binutils version, and even switch the C library between different project implementations ( uClibc vs eglibc vs MUSL.)

OpenWRT fetches the sources for all these things, then applies a number of local patches, before building.

We will need to refer to this later.

Confirming the Problem and Fishing for Red Herrings.

The first thing to note is that x86 has no problem, but MIPS does, and I want to run OpenWRT on various embedded devices with MIPS SOC.  Without that I may never have bothered digging deeper!

Of course I dived in initially and took the naive brute force approach.  I patched OpenWRT to apply the override flag to the linker: -Wl,-z,noexecstack. This was a bit unthinking, after all x86 did not need this.

Doing this gave partial success. In fact most programs gained NOEXECSTACK, except for a chunk of the uClibc components, busybox, and tellingly as it turned out, libgcc_s.so. That is, core components used by nearly everything. Of course.

(Spoiler: modern Linux toolchain implementations actually enable NOEXECSTACK by DEFAULT for C code! Which was an important fact I forgot at this point! Silly me.)

At this point,  I managed to overlook libgcc_s.so and decided to focus on uClibc. This decision would greatly expand my knowledge of OpenWRT and uClibc and embedded built systems, and do nothing to solve the problem the proper way!

OpenWRT builds uClibc as a host package, which basically means it abuses Makefiles to generate a uClibc configuration file partly derived from the OpenWRT config file settings, and eventually call the uClibc top level makefile to build uClibc. This can only be done after building binutils and two of three stages of gcc.

At this point I still did not fully understand how NOEXECSTACK really should be employed, which is probably an artefact of working on this stuff late at night and not reading everything as carefully as I might have.  So I did the obvious and incorrect thing and worked out how to patch uClibc further to push the force -Wl,-z,noexecstack through it. What I had to do to do that could almost take another blog article, so I’ll skip it for brevity.  Anyway, this did not solve the problem.

Finally I turned on all the debug and examined the build:

make V=csw toolchain/uClibc/compile

(Aside: the documentation for OpenWRT mentions using V=s to turn on some verboseness, but to get the actual compiler and linker commands of the toolchain build you need the extra flags. I should probably try and update the OpenWRT wiki but I have invested so much time in this that I might have to leave that as an exercise for the reader)

All the libraries were being linked using the -Wl,-z,noexecstack flag, yet some still failed checksec. Argh!

I should also note that repeating this process over and over gets tedious, taking about 20 minutes to built the toolchain plus minimal target firmware on my quad core AMD Phenom. Dont delete the build_dir/host and staging_dir/host  directories or it doubles!

So something else was going on.

Upgrades and trampolines, or not.

I sought help from the uClibc developers mailing list, who suggested I first try using up to date software. This was a fair point, as OpenWRT is using a 2 year old release of uClibc and 1 year old release of binutils, etc.

This of course entailed having to learn how to patch OpenWRT to give me that choice.

So another week later, around work and family and life, I found some time to do this, and discovered that the problem persisted.

At this point I revisited the Gentoo hardening guide.  After some detective work I discovered that several MIPS assembler files inside of uClibc did not actually have the recommended code fragments. Aha! I thought. Incorrectly again, as I should have realised; uClibc has already thought of this and when NOEXECSTACK is configured, as it is for OpenWRT, uClibc passes a different flag to the assembler that has the effect of fixing NOEXECSTACK for assembler files. And of course after I patched about 17 .S files and waited another half hour, the checksec scan was still unsuccessful. Red herring again!

I started to get desperate when I read about some compiler systems that use something called a ‘trampolline’. So I went back to the mailing uClibc  list.

At this point I would like to thank the uClibc developers for being so patient with me, as the solution was now near at hand.

Cutting edge patches and a wrinkle in time.

One of the uClibc developers pointed me to a patch posted on the gcc mailing list. As fate would happen, dated 10 September 2014, which was _after_ I started on these investigations.  This actually went full circle back to libgcc_s.so which was the small library I passed over to focus on uClibc.  This target library itself has some assembly files, which were neither built with the noexecstack option nor including the Gentoo-suggested assembly pragma. This patch also applies on gcc, not on uClibc, and of course was outside of binutils which was the other component I had to upgrade.  The fact that libgcc_s.so was not clean should maybe have pointed me to look at gcc, and it did cross my mind. But we all have to learn somehow.  Without all the above I would be the poorer for my knowledge of the internals of all these systems.

So I applied this patch, and finally, bliss, a sea of green NX enabled flags. Without in the end any modifications actually required to the older version of uClibc used inside OpenWRT.

This even fixed busybox NX status without modification.  So empirically this confirms what I read previously and also overlooked to my detriment, that being NOEXECSTACK is aggregated from all linked libraries: if one is missing it pollutes the lot.

Fixed NOEXECSTACK on uClibc

Postscript

Now I have to package the patch so that it can be submitted to OpenWRT, initially against Barrier Breaker given that has just been released!

Then I will probably need to repeat it against all the supported gcc versions and submit separate patches for those. That will get a bit tedious, thankfully I can start a test build then go away…

No responses yet

Evaluating the security of OpenWRT (part 2) – bugfix

Sep 23 2014 Published by under blog

I had a bug applying the RELRO flag to busybox, this is fixed in GitHub now.

For some reason the build links the busybox binary a second time and I missed the flag.

Also an omission from my prior blog entry: uClibc has RELRO turned on in its configuration already in OpenWRT, so does not need flags passing through to it. However, it is failing to build its libraries with RELRO in all cases, in spite of the flag. This problem doesn’t happen in a standalone uClibc build from the latest uClibc trunk, but I haven’t scoped how to get uClibc trunk into OpenWRT. This may have been unclear they way I described it.

No responses yet

Evaluating the security of OpenWRT (part 2)

Sep 20 2014 Published by under infosec

In my last post I covered how I setup an OpenWRT build, to examine  a small subset of indicators of security of the firmware.

In this follow-up post we will examine in detail the analysis results of one of the indicators: specifically, the RELRO flag.

A first look – what are the defaults?

The analysis here is specific to the Barrier Breaker release of OpenWRT, but it should be noted that during experiments with the OpenWRT development trunk the results are much the same.

Before diving into RELRO, lets take a look at the overall default situation.

Here is the checksec report for the Carambola2 device (MIPS processor) build.
It is a sea of red…

Default checksec report for OpenWRT x86 buildThe ‘run as root’ errors can be ignored: those programs are actually absolute symbolic links which do not resolve in the host system. Relative symbolic links resolve correctly but are filtered out of the analysis.

The x86 build paints a similar picture:

owrt_cs_report_0_x86(Notably for x86, the NX flag is correctly set, but that is a topic for another time.)

Note, the rest of this post describes how to modify OpenWRT to enable RELRO.  There may be perfectly valid reasons to not enable the flag (for example, using RELRO may have a performance impact, and for a given system the adverse security risk may be judged low), so I have ensured that the suggested mitigation if applied remains a choice in the configuration menu of the system.  For the moment my patch also retains backward compatibility by defaulting to off.

Inside the OpenWRT build system

After a brief look at the build logs, the reason is obvious: the typical gcc linker command is missing the flags needed to enable RELRO:  -Wl,-z,relro -Wl,-z,now (or the direct linker equivalents, -z relro -z now)

What could be done to address this?

OpenWRT provides a hook for appending to the global compiler CFLAGS but there is no similar hook for the linker stage. We could add those flags to the global CFLAGS and they can in fact flow through to the linker for many programs, but that would also be redundant as the flags are irrelevant to the compiler.  In the end I decided I would modify the OpenWRT source to add a new global CONFIG option, which adds -Wl,-z,relro -Wl,-z,now to the global LDFLAGS instead.

The following patch achieves that (note, I have left out some of the help for brevity):

Having attched OpenWRT, and enabled the new flag, lets rebuild everything again and run another checksec scan.

owrt_cs_report_1_x86What a difference!

The results shown above are for x86, the picture is similar for the Carambola2 MIPS image.

The new results indicate that the RELRO flag is present on some binaries but not all of them. From this we can predict that some packages do not fully honour the global OpenWRT build system linker flags. I soon confirmed this**; the implication is that the new flag CONFIG_SECURITY_FORCE_RELRO  is useful, however, a caveat in the Kconfig help is required. In particular, a statement to the effect that the efficacy depends on proper coding of OpenWRT packages (with ideally all packages maintained by the project being fixed to honour the flag.)

** For example: the package that builds libnl-tiny.so does not pass LDFLAGS through to the linker; this and some other base system packages needed patching to get complete coverage.  it is likely that there are other packages that I did not have selected that may also need tweaking.

Another notable package is busybox.  Busybox it turns out uses ld directly for linking, instead of indirectly via gcc, and thus requires the flags in the pure form -z relro -z now. (The busybox OpenWRT package Makefile also happens to treat the global TARGET_LDFLAGS differently from the TARGET_CFLAGS although I am unsure if this is a bug; but that turned out to be a red-herring.) Oddly, this solution worked for MIPS when I tried it previously, but is presently not successful for the x86 build, so further investigation is needed here; possibly I incorrectly noted the fix in previous experiments.

Fun and Games with uClibc and busybox

The other recalcitrant is the uClibc library. I spent quite a bit time trying to work out why this was not working, especially having  confirmed with verbose logging that the flags are being applied as expected. Along the way I learned that uClibc already has its own apply RELRO config item, which was already enabled. Even more oddly, RELRO is present on some uClibc libraries and not others, that as far as I could tell were being linked with identical linker flag sets.

After some digging I discovered hints of bugs related to RELRO in various versions of binutils, so I further patched OpenWRT to use the very latest binutils release.   However that made no difference.  At this point I took a big diversion and spent some time building the latest uClibc externally, where I discovered that it built fine using the native toolchain of Debian Wheezy (including a much older binutils!)  After some discussion on the uClibc mailing list I have come to the conclusion that there may be a combination of problems, including the fact that uClibc in OpenWRT is a couple of years old (and additionally has a set of OpenWRT specific patches.)   I could go further and patch OpenWRT to use the trunk uClibc but then I would have to work through refreshing the set of patches which I really don’t have time or inclination to do, so for the moment I have deferred working on resolving this conundrum.  Eventually someone at OpenWRT may realise that uClibc has undergone a flurry of development in recent times and may bump to the more recent version.

Comments

Along the way, I discovered that Debian actually runs security scans across all packages in the distribution – take a look at https://lintian.debian.org/tags/hardening-no-relro.html.

It is worth noting that whenever changing any build-related flag it is worth cleaning and rebuilding the toolchain as well as the target packages and kernel; I found without doing this, flag changes such as the RELRO flag don’t fully take effect as expected.

For maximum verboseness, run with make V=csw although I had to dig through the code to find this out.

I was going to repeat all the testing against a third target, another  MIPS-based SOC the  RALINK 3530 but at this point I don’t really have the time or inclination, I am sure the results will be quite similar.  It would probably be useful to try with an ARM-based target as well.

I should also try repeating this experiment with MUSL, which is an alternative C library that OpenWRT can be built with.

Conclusion

Out of the box, OpenWRT has very limited coverage of the RELRO security mitigation in a standard firmware build.  By applying the suggested patches it is possible to bring OpenWRT up to a level of coverage, for RELRO, to that approaching a hardened Gentoo or Ubuntu distribution, with only a small subset of binaries missing the flag.

References

My Github account includes the repository openwrt-barrier-breaker-hardening. The following branch include the completed series of patches  mentioned above:  owrt_analysis_relro_everywhere  I hope it will remain possible to apply these changes against the official release for a while yet.

The patch that enables the latest binutils is not in that branch, but in this commit.

No responses yet

Older posts »