Sign up for our newsletter and get the latest HPC news and analysis.

Activity in Linux file systems, BTRFS and ext4

This may not be news to the file system aware among you, but I’m part of the blissfully ignorant crowd that complains about the old file system until a shiny new one shows up — seemingly out of nowhere — to fix old problems and introduce new ones. Many of the performance problems experienced by users I work with every day boil down to file systems and their interactions with data movement patterns, so this is an area that is increasingly relevant to both tactical and strategic thinking in HPC centers.

Drew Robb writes over at EnterpriseStorageForum.com about two significant developments in the Linux file system world. One, ext4, already being used in prime time after being stabilized over the past year while the other, BTRFS, is still getting off the blocks

Sun (NASDAQ: JAVA) open sourced ZFS and acquired Lustre. Quantum (NYSE: QTM), Panasas, IBM (NYSE: HPQ), HP (NYSE: HPQ), NetApp (NASDAQ: NTAP) and Red Hat (NYSE: RHT) have all boosted their file system development efforts. And NFS is getting a major overhaul in the form of pNFS. Not surprisingly, all this development effort has stimulated work in open source communities to come up with higher performance file systems for Linux.

…Although it was developed in 2006, ext4 has only really been stabilized over the last year and is only now beginning to see any volume of deployment. …Ext4 is very much an evolution of the popular ext3 file system. Ext3 is limited to 16TB of storage, while ext4 has been designed to handle even larger file systems. One data integrity feature that has been added to ext4 is support for checksums on its internal journal transactions, which gives it some additional robustness in the face of storage errors.

But ext4 isn’t so significant a development as BTRFS, at least in terms of potential. BTRFS may eventually pose more of a threat to ZFS and others, though it has yet to be finalized. It brings online defragmentation support, a mode just for solid state drives (SSDs), logging for copy-on-write, and various compression, mirroring, stripping and snapshot bells and whistles.

I was interested to learn that BTRFS was originally developed by Oracle, and that both ext4 and BTRFS are in Fedora 11. I wonder what happens next, now that Oracle is acquiring Sun and Lustre?

Comments

  1. I’ve been running btrfs on my laptops SSD for, umm, 9 months or so now and it’s still not broken! :-) Agreed it’s got a while to go, but it’s trucking on at the moment with Red Hat, Intel, Oracle and others all contributing code.

    As for ext4, that article seems to completely miss the fact that it now defaults to being an extent based filesystem (a la XFS) rather than a block based one!

    They also completely missed NILFS2 (www.nilfs.org) which arrived in the mainline kernel around the same time as btrfs which features continuous checkpointing (and GC of those after a defined time) with the capability of converting a transient checkpoint into a permanent snapshot (and, if you no longer need it, back again).

  2. Chris – great info, thanks. I have 0 filesystem chops, so the added detail is much appreciated.

  3. Not a problem John, FWIW I highly recommend reading LWN (http://lwn.net/) for up to date info on what’s going on, their weekly kernel page is a great resource for developments in the Linux kernel.

    I also wrote an article for LinuxWorld almost 2 years ago about those (and other) filesystems – the benchmarks will be way out of date now but the background info might be interesting for you – http://www.csamuel.org/articles/emerging-filesystems-200709/

Trackbacks

  1. […] See more here:  Activity in Linux file systems, BTRFS and ext4 | insideHPC.com […]

  2. […] Activity in Linux file systems, BTRFS and ext4 | insideHPC.com By John West insideHPC: HPC news for supercomputing professionals. insideHPC – http://insidehpc.com/ […]

  3. […] Continued here:  Activity in Linux file systems, BTRFS and ext4 | insideHPC.com […]

Resource Links: