This may not be news to the file system aware among you, but I’m part of the blissfully ignorant crowd that complains about the old file system until a shiny new one shows up — seemingly out of nowhere — to fix old problems and introduce new ones. Many of the performance problems experienced by users I work with every day boil down to file systems and their interactions with data movement patterns, so this is an area that is increasingly relevant to both tactical and strategic thinking in HPC centers.
Drew Robb writes over at EnterpriseStorageForum.com about two significant developments in the Linux file system world. One, ext4, already being used in prime time after being stabilized over the past year while the other, BTRFS, is still getting off the blocks
Sun (NASDAQ: JAVA) open sourced ZFS and acquired Lustre. Quantum (NYSE: QTM), Panasas, IBM (NYSE: HPQ), HP (NYSE: HPQ), NetApp (NASDAQ: NTAP) and Red Hat (NYSE: RHT) have all boosted their file system development efforts. And NFS is getting a major overhaul in the form of pNFS. Not surprisingly, all this development effort has stimulated work in open source communities to come up with higher performance file systems for Linux.
…Although it was developed in 2006, ext4 has only really been stabilized over the last year and is only now beginning to see any volume of deployment. …Ext4 is very much an evolution of the popular ext3 file system. Ext3 is limited to 16TB of storage, while ext4 has been designed to handle even larger file systems. One data integrity feature that has been added to ext4 is support for checksums on its internal journal transactions, which gives it some additional robustness in the face of storage errors.
But ext4 isn’t so significant a development as BTRFS, at least in terms of potential. BTRFS may eventually pose more of a threat to ZFS and others, though it has yet to be finalized. It brings online defragmentation support, a mode just for solid state drives (SSDs), logging for copy-on-write, and various compression, mirroring, stripping and snapshot bells and whistles.
I was interested to learn that BTRFS was originally developed by Oracle, and that both ext4 and BTRFS are in Fedora 11. I wonder what happens next, now that Oracle is acquiring Sun and Lustre?