Feb 06 2015

In the middle of last year I attended Unleashed/Govhack 2014, I blogged about it here.

Barely over a week ago by pure chance I stumbled across another hackathon, this time being hacksa. This was an Adelaide only event that was designed to tie in with Entrepreneurs Week here. It seemed like an excellent opportunity to practice for the next GovHack (and a chance to meet some new people) so I entered a team! Also, I discovered that one of the people running the event was a friend of mine from uni over from Sydney for the event, so it was a good chance to catch up.

Hacksa was also going to be a small event, being first time, held at short notice, in one city, and also more focused to a specific domain (music industry) compared to GovHack. So I figured that if I managed to put into practice some of what I think I learned at Unleashed in 2014 we might stand a chance of earning some conference tickets (one of the prizes is entry to a conference called NetWorkPlay) or maybe if lucky enough cash!

Having entered a team, I had to find some teammates. So I press-ganged Mr 13, and then convinced a friend from linux.conf.au who was in another team at Unleashed, and another friend of mine who knows a bit about business, to join in, and we set to work.

Spread Out in Time

Unlike GovHack, hacksa released the API (well, some of them) the Friday before the event. So we had the weekend to work on it. Actually we only had the weekend because the hacksa event proper was strangely held on a Wednesday, and we all had work or school.  But this was OK, we only had to show up in the evening to finalise things and do the video interview. This was another difference (and I think improvement), we didn’t have to shoot our own video.

Actually we really only had Sunday, because of life and stuff, so I put in a late one on the Saturday evening pulling together a VPS and web infrastructure for our entry, along with libraries (I decided to use twitter bootstrap to get moving in  hurry, along with Python web.py)  I’d had two ideas, one for a visualusation/infographic and the other a web app, and learning from Unleashed I intentionally stopped thinking too hard at that point and went with the web app, which was to be a mash up based initially on the V-channel chart APIs. Ultimately I think going for a web app will prove to be a good idea, we’ll see in due course.

On the Sunday we congregated at a local library and polished off the prototype. Mr 13 put together a snazzy landing page for us using weebly, and a request page using launch rock. Its pretty handy being able to divvy out tasks and keep in involved and motivated!


The hackathon event was held at a place called St Pauls Creative Center in the CBD.  It was actually a pretty nice venue for this event. We got there about 6pm which gave us a couple more hours to refine Sundays efforts and then do the video interview. Of course we got pizza which was good too because Mr 13 was pretty hungry by then :-)

Overall it felt good to actually pull something together that was essentially a working ‘minimal viable product’ (in the parlance), and we also had a good ‘story’ to tell, thanks to my friends.

I’ll probably blog more about the app itself and the API feeds a bit later.

Even if we don’t win anything, I think I’ll finish it off and we’ll go live as an experiment to see if anyone actually uses it and maybe make enough to pay for the VPS hosting for a few cups of coffee from google ads!


No responses yet

Linux.conf.au 2015 : working backwards from the end… and things that go bang!

Feb 06 2015

Some of the most awesome things about LCA are events that are not part of the official programme.  These include the affectionately named BOFs, and also various things happening before and after the conference proper.

On the Saturday just after the conference, I was lucky enough that I had sufficient time before my flight home to be able to tag along to a hobbyist rocket launch meet, and watch the friendly locals, as well as the well known to the open source community rocket enthusiasts Bdale Garbee and Keith Packard, send a variety of projectiles high up into the sky! I jumped at the chance to go because my son who is 13 did a launch of a small rocket through scouts, so I thought I’d better get some pictures and video for him :-)

The next shot gives an idea of the scale of an assembled rocket:

The rockets launch from the middle of a paddock.

The rocketry club is really into safety, for obvious reasons. They have to get licenses to be allowed to launch, and there is a ritual each time a rocket is launched – yelling out “Sky Is Clear”, “Range Is Clear” before a launch can proceed.

This video I shot is of a small rocket launching.

One of the other linux.conf.au’ers Augur posted some other vides to YouTube: https://www.youtube.com/channel/UCcFwcKbYU3Q66PIUck_Haew

Watching things go bang is always fun, but these guys take it to a whole new level. If you listen carefully to the video a few minutes after the launch, notice the Android phone and the laptop actually speak the telemetry data…

This works roughly as follows: there is a computer built on Altus Metrum components in the rocket, this sends position and other data back to a receiver connected to an antenna that the flyer (is that the right word?) is holding and pointing in the general direction of the rocket… the data is then relayed via bluetooth to the phone or laptop and reported via speech synthesis. “Range one thousand two hundred seventy metres. Bearing thirty degrees south west elevation forty two degrees.”

I believe one of the rockets made it over eight kilometers high (I didn’t manage to record the exact amount!)

Eventually they run out of puff and start falling back. A parachute then deploys bringing it to ground, and a couple of km hike ensues.

I believe the following rocket is known as the Pink Freak. My daughter loved the shoes when I showed her the pics…

Alongside everything else, there was a quadcopter. When I first arrived, this was keeping a couple of the younger kids amused. But once the meet started moving, I realised it had a HD camera, and was used to monitor the rocket launches – and the pilot (this must be the right word!) had a VR headset, not an Occulus, but a new one I had not heard of, a SkyZone(?), to fly the thing way up and record the launch! It sounds like a mosquito in the videos.

I had a lot of other photos and video but many had people and I haven’t managed to make contact with the club to check if it is OK to post them, so for now I haven’t.

No responses yet

Linux.conf.au 2015 catchup #1

Jan 26 2015

At the conference in Auckland I had two presentations.

For the first time I managed to get a main conference talk accepted, actually it was a tutorial which goes for 90 minutes! It was a bit daunting beforehand, but after I finished, I realised I prefer the tutorial format over having to deliver a talk. I enjoy the interaction with the audience and the sharing of knowledge, and also not being the sole focus (and not having to remember exactly what to say so much!)

My tutorial was on Reverse Engineering with Radare2; the video (Youtube) and slides are linked from the conference presentation, and have the slides up on my personal landing page as well. Thanks to James for helping with a final practice run, its always good to have a typical candidate audience perspective beforehand.

I also did a shorter talk at the Open Hardware mini-conference, on hardening embedded Linux, using OpenWRT on devices like the carambola2 as an example. The video of the mini-conferences is a bit less polished due to resourcing, here I am on about 2/3 the way through. I was somewhat more flustered in my delivery due to late changes to some slides (see earlier blog article) and a problem with my laptop deciding to have thermal issues an hour before the talk. I managed to resolve these (thanks AndyK for your help!) but it put me off my mojo a bit unfortunately. The live demo I was quite happy with, it worked without issue, so perhaps the demo gods were appeased by my earlier mishaps… The final slides are here.

No responses yet

Linux.conf.au – so much all the things!

Jan 21 2015

Well, I’ve said it before and not followed through, but I am intending to blog about various stuff from last weeks LCA over the next month or two.

One things about LCA of course is how much you learn. Especially when you stand up in front of a room to share something and discover errors in your own understanding! In my own case, I had a talk at the Open Hardware miniconf about some security things related to embedded devices. Literally an hour before I had a ping on twitter alterting me to a factual error in my blog, which was also loudly proclaimed in the talk I was about to deliver. Luckily it was only one slide, and the misunderstanding did not impact the rest of the talk (or for that matter, most of the offending blog article.) So I have updated the original blog article with a correction.

No responses yet

Experiments with hardening OpenWRT: applying the grsecurity patches

Dec 14 2014

A well known set of security enhancements to the Linux kernel is the grsecurity patch.  The grsecurity patch is a (large) patch that applies cleanly against selected supported stock Linux kernel versions. It brings with it PAX, which protects against various well known memory exploits, plus  a number of other hardening features including logging time and mount changes. In particular it enables features such as Non-executable stack (NX) on platforms that do not provide NX in hardware, such as MIPS devices and older x86.
UPDATE Unfortunately, NX protection for MIPS 32-bit devices is not in fact supported in software. This would be very useful. Whilst I was teaching myself I managed to mix things up, so be aware when reading the rest of this blog entry. Otherwise, the usefulness of grsecurity and the mechanism for patching into OpenWRT is still valid.

Note also, a more detailed procedure you can use to rebase the patches is at https://github.com/pastcompute/openwrt-cc-ar71xx-hardened/wiki .

OpenWRT hardening

OpenWRT is a widely adopted embedded / router Linux distribution. It would benefit greatly from including grsecurity, in particular given most MIPS platforms do not support NX protection in hardware. However for a long time the differences between the OpenWRT kernel and the kernel revisions that grsecurity is supported on have been significant and would likely have taken an extreme effort to get working, let alone get working securely.

This is a shame, because there is malware targeted at consumer embedded routers, and it must only be a matter of time before OpenWRT is targeted.  OpenWRT is widely regarded as relatively secure compared to many consumer devices, at least if configured properly,  but eventually some bug will allow a remote binary to be dropped. It would be helpful if the system can be hardened and stay one step ahead of things.

The OpenWRT development trunk (destined to become the next release, ‘Chaos Calmer’ in due course) has recently migrated most devices to the 3.14 kernel tree.  Serendipidously this aligns with the long term supported grsecurity revision 3.14.  When I noticed this I figured I’d take a look at whether it was feasible to deploy grsecurity with OpenWRT.

Applying grsecurity – patch

In late November I pulled the latest OpenWRT sources and the kernel version was 3.14.25, which I noticed matched the current grsecurity stable branch 3.14.25

The grsecurity patch applies cleanly against a stock kernel, and OpenWRT starts with a stock kernel and then applies a series of patches designed to extend hardware support to many obscure embedded things not present in the mainline kernel, along with patches that reduce the memory footprint. Some of the general patches are pushed upstream but may not yet have been accepted, and some could be backports from later kernels.  Examples of generic patches  include a simplified crash report.

Anyway, I had two choices, and tried them both: apply grsecurity, then the OpenWRT patches; or start with the OpenWRT patched kernel.  In both cases there were a number of rejects, but there seemed to be less when I applied grsecurity last. I also decided this would be easier for me to support for myself going forward, a decision later validated successfully.

OpenWRT kernel patches are stored in two locations; generic patches applying against any platform, then platform specific patches.  My work is tested against the Carambola2, an embedded MIPS board supported by the ‘ar71xx’ platform in OpenWRT, so for my case, there were ar71xx patches.

To make life easy I wrote a script that would take a directory of OpenWRT kernel patches, apply to a git kernel repository and auto-commit. This allowed me to use gitg and git difftool to examine things efficiently.  It also worked well with using an external kernel tree to OpenWRT so I didnt have to worry yet about integrating patches into OpenWRT. This script is on github, it should be easily adaptable for other experiments.

(Note: to use an external tree, managed by git, use config options like the following:

There were four primary rejects that required fixing.  This involved inspecting each case and working out what OpenWRT had changed in the way. Generally, this was caused because one or the other had modified the end of the same structure or macro, but luckily it turned out nothing significant and I was able to easily reconcile things. The hardest was because OpenWRT modifies vmstat.c for MIPS and the same code was modified by grsecurity to add extra memory protections.  At this point I attempted to build the system, and discovered three other minor cases that broke the build. These mispatches essentially were due to movements in one or two lines, or new code using internal kernel API modified by grsecurity, and were also easily repaired.  The most difficult mispatch to understand was where OpenWRT rewrites the kernel module loader code, apparently to make better use of MIPS memory structures and it took me a little while to understand how to try and fix things.

The end result is on github at https://github.com/pastcompute/openwrt-cc-linux-3.14.x-grsecurity

Applying grsecurity – OpenWRT quirks

One strange bug that had to be worked around was some new dependency in the kernel build process, where extra tools that grsecurity adds were not being built in the correct order with other kernel prerequisites.

In the end I had to patch how OpenWRT builds the kernel to perform an extra ‘make olddefconfig‘ to sort things out.

I also had to run ‘make kernel_menuconfig‘ and turn on grsecurity.

As the system built, I eventually hit another problem area: building packages. This was a bit of an ‘OH-NO’ moment as I thought it had the potential to become a big rabbit hole. Luckily as it turned out, only one package was affected in the end: compat-wireless.  This package builds some extra user space tools and wifi drivers, and used a macro, ACCESS_ONCE, that was changed by grsecurity to be more secure; and required use of a new macro to make everything work again, ACCESS_ONE_RW. There were rather a number of calls to this macro, but luckily it turned out to be fixable using sed!

Booting OpenWRT with grsecurity – modules not loading

I was able to then complete an INITRAMFS image that I TFTP’d into my carambola2 via uboot.

Amazingly the system booted and provided me with a prompt.

I then discovered that no kernel modules were loading. A bit of digging and it turns out that a grsecurity option, CONFIG_GRKERNSEC_RANDSTRUCT  will auto-enable CONFIG_MODVERSIONS. One thing I learned at this point is that OpenWRT does not support CONFIG_MODVERSIONS=y, due to the way it packages modules with its packaging system. So an iteration later with the setting disabled, and everything appeared to be “working”

Testing OpenWRT with grsecurity

Of course, all this work is moot if we cant prove it works.

Easy to check is auditing. For example, we now had these messages:

However, the acid test would be enforcement of the NX flag. Here I used the code from http://wiki.gentoo.org/wiki/Hardened/PaX_Quickstart to test incorrect memory protections. Result:


Revisiting Checksec, and tweaking PAX

In an earlier blog I wrote about experimenting with checksec.  Here I used it to double-check that the binaries were built with NX protection. MOst were, due to a patch I previously submitted to OpenWRT for MIPS. However, openssl was missing NX. It turns out that OpenSSL amongst everything else it has been discussed for this year, uses assembler in parts of the encryption code! I was able to fix this by adding the relevant linker ‘.note.GNU-stack‘ directive.

The PAX component can be tweaked using the paxctl command, so I had to build that with the OpenWRT toolchain to try it out. I discovered that it doesnt work for files on the JFFS2 partition, only in the ramdisk. Further to enable soft mode, you need to add a kernel boot command line argument. To do this for OpenWRT, edit a file called target/linux/$KERNEL_PLATFORM/generic/config-default where in my case, $KERNEL_PLATFORM is ar71xx

Moving Targets

Right in the middle of all this, OpenWRT bumped the kernel to 3.14.26. So I had to exercise a workflow in keeping the patch current.  As it happened the grsecuroty patch was also updated to 3.14.26 so I presume this made life easier.

After downloading the stock kernel and pulling the latest OpenWRT, I again re-created the patch series, then applied grsecurity 3.14.26.  The same four rejects were present again, so fingers crossed I cherry-picked all my work from 3.14.25 onto 3.14.26. As luck would have it this was one smooth rebase!

Recap of OpenWRT grsecurity caveats

  • CONFIG_GRKERNSEC_RANDSTRUCT is not compatible with the OpenWRT build system; using it will prevent modules loading
  • Some packages may need to be modified to support NX – generally, if these use assembly language and don’t use the proper linker directive.
  • For some reason paxctl only seems to work on files in /tmp not in the JFFS overlay. This is probably only a problem when debugging
  • Your experience with the debugger gdb will probably be sub-optimal unless you put the debug target on /tmp and use paxctl to mark it with exceptions


After concluding the above, I converted the change set from my local Linux working copy into a set of additional patches on OpenWRT and rebuilt everything to double check.

The branch ‘ar71xx-3.14.26-grsecurity’ in https://github.com/pastcompute/openwrt-cc-ar71xx-hardened has all the work, along with some extra minor fixes I made to some other packages related to checksec scan results.

THIS MAY EXPLODE YOUR COMPUTER AND GET YOU POWNED! This has been working for me on one device with minimal testing and is just a proof of concept.

No responses yet

« Newer posts Older posts »