Fixing sluggish write performance of USB flash (thumb) drives

Nov 13 2012 Published by under linux, tech

    This has been noted in various places around the web but in practice what I did seems to be a combination of various writings so I have documented my own experiences here.

    Background

    I recently acquired a (yet another) USB flash drive, this a 16 GB “Dolphin” brand. The actual device reports as “048d:1165 Integrated Technology Express, Inc.” when interrogated using lsusb. I am using it to transfer transcoded Kaffeine PVR recordings from my PC to the set top box in the lounge for more comfortable watching.

    On first use, however, it took what seemed like forever to transfer a 250MB AVI file, over USB2, and looking at the GKrellM chart the write data rate appeared to be a very poor 350 kB/sec. So it seemed yet again, I needed to optimise a USB disk before it was adequate for use.

    In theory, to simplify things to one sentence, flash disk (and in particular, modern SSD) should be faster than spinning disks, as access is a true physical random access operation, without having to wait for the heads to be in the right spot. However this is invalidated due the blocky nature of flash disk writes. The actual reason for the poor write speed is that the default partition starts at the 63rd sector (byte 32256) on the disk, and USB flash drives, SD cards, etc. are designed to write data in chunks of say 128kB at a time. Even if you only write one sector, the entire 128kB (or 256 sectors) must be (re-read first and) written. So when a partition is not aligned on a 128kB boundary, more writes than otherwise necessary are required, slowing performance. USB flash drives generally employ FAT32 so they are usable on the widest variety of devices (including set top boxes) and the general experience of FAT32 is that write performance is severely affected if the partition alignment does not match the flash write size, for both the partition and the FAT master table itself.

    Procedure

    The procedure I follow for doing fixing misaligned flash drives is:

    1. Find a Linux computer, or reboot using a live Linux distribution such as SysRescueCD
    2. Destroy the existing partition.
    3. Recreate a single partition, ensuring it starts at the 256th sector (byte 131072, or 128kB)
    4. Format the partition to FAT32. with the following non default options:
      • override the default sectors per cluster to ensure clusters are aligned. This comes at some expense of apparent usable space, but the performance gain for writing large files such as video files is more than worth it.
      • Adjust the “reserved” sectors so that the FAT table itself is aligned to 128kB.

    Detailed Steps

    The following command sequence will accomplish this under Linux. This assumes your drive is at /dev/sdd, this will vary depending on what other disks you have.

    1. Run GNU fdisk with units in sector mode not cylinder mode. Then print the existing partition table (enter p when prompted. Below you can see the start sector of the existing partition is at sector 63. Note this is also a primary partition. This is typical of USB flash disks you might purchase at the local supermarket…

    2. Delete the partition:
    3. Recreate the partition, aligned at sector 256 (131072 bytes), and set the type back to FAT32 LBA (in this case matching what previously existed) (type ‘c’, or 0x0c, i.e. FAT32 LBA). Use of FAT32 LBA allows use to start the filesystem on an arbitrary sector bearing no relationship to legacy cylinders, etc. The final sector depends on the disk size.
    4. Save changes:
    5. Format the partition, setting the number of reserved sectors so that the FAT table remains aligned at a 128kB boundary. Assuming sectors per cluster, s=128 (65536 bytes), and our partition length of 31567469 sectors, we want the first fat to start at the 256th sector within the partition (which is OK as the partition itself is aligned.) For some sizes of flash disk, this can be an iterative process, but generally setting the number of reserved sectors to 256 will achieve what we want.

    6. This is the most important step – verify that the chosen number of reserved sectors has resulted in an aligned FAT table and aligned data area.


      The important figure here, is the data area sector – it must be an integer multiple of 256, and 256 x 17 == 4362 in this example.
    7. Test the result. I copied a 256 MB file onto the drive, and GKrellM is now reporting ~2.5MB/sec. More importantly, it finished in approx. one eighth of the time compared to before reformatting.

    The improved write performance should be just as noticeable from Windows.

No responses yet

Patching and Building a custom Linux Kernel in Debian

Apr 10 2012 Published by under linux

These posts cover a topic which seems to be documented to varying degrees across the net, but nothing quite exactly matched what I wanted to do. In the end this is a result of multiple sources of information and inspiration (and perspiration…)

For some time I had been getting a Kernel fault report popup with irritating regularity. In the end I isolated it to something going wrong with my external Firewire drive after my computer was resuming from suspend (specifically Suspend to RAM.)
In the end chasing this down required working through the following tasks:

  1. Disabling the proprietary NVidia driver and activating ‘nv’ ( I was unable to successfully configure nouveau to work with my particular dual head configuration), so that my kernel was no longer ‘TAINTED’, which would have led me into a brick wall if I had been required to report a kernel bug.
  2. Consistently replicating the fault, which included learning about a bunch of stuff in the Linux /sys filesystem.
  3. Finally getting a 3-series kernel to work on Debian Squeeze – it turns out by now 3.2 has been packaged into Debian backports, which gets me past an earlier roadblock with kernel upgraded. Upgrading to the latest kernel would eliminate if the problem had been resolved (which is was not at least of 3.2.9)
  4. Rebuilding the kernel from source – (something I have done this many times before, but it doesn’t hurt to recap) and applying the patches needed
  5. Re-enabling NVidia – which involved verifying my DKMS setup was still working.

I haven’t blogged recently due to various family mini-crises to do with pets, sickness and other issues, as well as extra busyness at work.

As it is getting late this post will conclude with the command line used to build and install my kernel, and I will expand on this in the next post.

Things to note:

  • The above will build a kernel using the same configuration as an installed Debian backports 3.2 kernel, assuming the backports kernel an source packages have been installed. There are no changes or patches yet
  • Your user must be in the ‘src’ group for the make-kpkg command to work as-is.
  • The 3.2 kernel in backports (as of March 2012) was in fact version 3.2.9 although this is not indicated in the Debian version for some reason.

No responses yet

Remastering Youtube videos into AVI for your Set Top Box

Jan 22 2012 Published by under howto, tech

I wasn’t going to post here for a week or so as I need to clear my head and catch up with real world stuff, but I did this and thought it would be helpful to share.

As of now most of the linux.conf.au sessions have been uploaded onto YouTube. All the LCA2012 videos can be found at www.youtube.com/user/linuxconfau2012/videos. There is speculation (most likely) they will be mirrored at some point but I don’t want to wait so I have started watching them from YouTube.

On Linux you can download YouTube videos for later viewing using the clive program. This is packaged in Debian, YMMV for other distributions. Clive works like wget in that you can simply copy and paste the YouTube URL onto the command line and grab the FLV file. The file can be played on your PC or moved onto a smart phone or media player device.

I tend to use my PC as a PVR and play recordings on my set top box by USB key. Unfortunately the FLV files cannot be played directly on my STB, which is a cheap Dick Smith model, so need to be converted first.

Procedure

  1. If required, install clive and mencoder. On Debian, this is done using the following command:
  2. Navigate to the YouTube video page, and copy the URL. Then in a terminal, enter the following command, pasting the URL where indicated. Note that the single quotes are important as a couple of times I found that the URLs have characters that have meaning to bash:
  3. After downloading completes, remaster the file into AVI format.

Copy the resulting file onto your USB stick and enjoy in your loungeroom :-) This is good for showing them to digitally challenged friends and family as well…

If the video looks stretched, see below.

You may omit the backslash above and type the commands with a backslash all on one line.

I will trying to show to quite a few people Karen Sandlers keynote video, which to me is about ‘the Freedom to Stay Alive’, and shoving a USB into someones STB or Windows laptop can often be more effective than even emailing a YouTube link.

PS. kudos to the Crayon syntax highlighter plugin for WordPress.

TL; DR

You could probably go further, and write a scraper to do all the videos and encoding in one fell swoop but I don’t have time so I have been cherry picking what I want to see soonest or show others, first.

Caveat

My system is not a pure Debian Squeeze installation, I have quite a number of repositories in my /etc/apt/sources.list. My version of mencoder actually comes from www.debian-multimedia.org and it is possible that the pure Debian version may not perform identically… my system has also been configured for some time so there may be other dependencies not listed above, again, YMMV.

The threads=3 argument is for my quad core Phenom, so you may need to reduce this for a dual core processor.

I saved the command into a script:

Then you can run it in a loop over all videos:

The conference video is recorded at 320×240 which is a 4:3 aspect ratio. This means by default it plays fine on older televisions, but will often be stretched by default on modern televisions with a 16:9 aspect ratio. This can be centered usually with a button press on the remote, but sometimes even this is beyond, umm, less experienced friends and family (like elderly relatives) so you can instead ‘pillar-box’ the video into 16:9 so that it doesn’t need reducing – although this will cause it to look compressed if played on a 4:3 television:

No responses yet

How to Enable / Disable remote X connections in Debian Squeeze

Jan 19 2012 Published by under howto, linux

Background

Using my AspireOne at LCA2012 I realised I had a hole that really needed to be tightened before using the Most Excellent LCA2012 wireless network instead of a 3G dongle.

One of these is that I was gaining an ipv6 address on the LCA network. (This was not the hole.) My 3G only has ipv4 and at I only run ipv4 at home.

After connecting I had both ipv6 and ipv4 addresses, but importantly upon running netstat -antp realised I had kdm [1] listening on 6000 wide open – I had it firewalled out on ipv4 but had never setup ip6tables (oops)

To sort things quickly I subsequently disabled ipv6 [2] but I first killed off my local X/server from accepting connections anyway (you cant be too cautious, in a large crowd of very skilled people, potentially prank-minded, right? following good advice – ‘take precautions’ …)

me@xxxxxxx:~$ sudo netstat -lntp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:6000            0.0.0.0:*               LISTEN      13494/X         
tcp6       0      0 :::6000                 :::*                    LISTEN      13494/X

Procedure

To disable network connections on port 6000 using kdm:

  1. Edit /etc/kde4/kdm/kdmrc
  2. Look for a section [X-:*-Core]
  3. If missing or commented out, add the line ServerArgsLocal=-nolisten tcp, if it is already there, instead append -nolisten tcp to the line starting with ServerArgsLocal
  4. Either reboot, or kill X and restart kdm

Result:

me@xxxxxxx:~$ sudo netstat -lntp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

To _enable_ connections, simply ensure the -nolisten tcp arguments are not present.

FWIW I have had to do this on gdm in the past as well.
Instructions for this are actually provided in the Debian Reference Manual, Chapter 7 (section 7.4.2, tips)

YMMV if you have more than a basic configuration or are running some other variant or distribution.

Footnotes
[1] I only use kdm for login, for performance I run openbox
[2] Yes, I know, I should use ipv6, I even agree with all the reasons listed by Julien Goodwin at his excellent SysAdmin miniconf talk on why I should use ipv6, but that will need to wait until I get home (and depends on my ISP.)
(And facepalm to self for mixing up selinux with ipv6 before coffee this morning)

No responses yet

Debian Squeeze and Linux Kernel 3

Dec 27 2011 Published by under linux

A couple of months ago I splashed out and upgraded my motherboard to an ASUS Sabretooth 990FX, in theory to complement my quad core Athlon II 640 with faster RAM and SATA 3, more PCI-E slots etc. (but the camoflauge colour scheme and mil-spec capacitors didn’t hurt either… :-) )

However it wasn’t all smooth sailing, Continue Reading »

No responses yet

Older posts »