ZFS,XFS,and EXT4 filesystems compared

This is my attempt to cut through the hype and uncertainty to find a storage subsystem that works. I compared XFS and EXT4 under Linux with ZFS under OpenSolaris. The machine is a 2200MHz Athlon 64 with 1GB of memory and 8 400GB disks attached to an Areca ARC-1220 SATA controller. Linux is 2.6.22.5 and OpenSolaris is 5.11 snv_70 "Nevada".

Aside from the different kernels and filesystems, I tested internal and external journal devices and software and hardware RAIDs. Software RAIDs are "raid-10 near2" with 6 disks on Linux. On Solaris the zpool is created with three mirrors of two disks each. Hardware RAIDs use the Areca's RAID-10 for both Linux and Solaris. Drive caches are disabled throughout, but the battery-backed cache on the controller is enabled when using hardware RAID. If an external journal is used, it is on a single disk with write caching disabled. The system is installed on the remaining disk, which is not involved in the testing.

XFS and ZFS filesystems were created with default parameters. EXT4 filesystems are created with -b 4096, to force 4096 block size when using the external journal. All filesystems were mounted with atime disabled. EXT4 is mounted with extents.

The benchmark uses Bonnie++ 1.03a, randomio 1.3, FFSB 5.2.1, postmark 1.51, dd, tar, and cpio with a tarball of Linux 2.6.22.5. FFSB and randomio were patched to build on Solaris, and to use the Solaris equivalent of Linux's O_DIRECT. The test proceeded as follows:

1. bonnie++
2. copy linux kernel tarball to filesystem under test
3. untar it
4. dd if=/dev/zero of=bigfile bs=1048576 count=4096
5. randomio bigfile 10 .25 .01 2048 60 1
6. find linux-2.6.22.5 | cpio -pdm linux
7. postmark 1, 10, and 100
8. ffsb read, write, mixed, and huge
9. tar -cf linux.tar linux

The results are below. EXT4 is fast for metadata operations, tar, untar, cpio, and postmark. EXT4 is much faster than the others under FFSB. EXT4 with hardware RAID and external journal device is ludicrously fast. EXT4 seems to have a bad interaction with software RAID, probably because mkfs fails to query the RAID layout when setting the filesystem parameters.

ZFS has excellent performance on metadata tests. ZFS has very bad sequential transfer with hardware RAID and appalling sequential transfer with software RAID. ZFS can copy the linux kernel source code in only 3 seconds! ZFS has equal latency for read and write requests under mixed loads, which is good.

XFS has good sequential transfer under Bonnie++. Oddly XFS has better sequential reads when using an external journal, which makes little sense. Is noatime broken on XFS? XFS is very slow on all the metadata tests. XFS takes the RAID layout into consideration and it performs well on randomio with hardware or software RAID.

The Areca controller seems to have some problem with large requests. The "huge" FFSB test does reads and writes in blocks of 1MiB. The hardware RAID is only half as fast as the software RAID in this test. To check that there's nothing weird about the Areca driver for Solaris, I spot-checked sequential transfer using dd, and rates were in excess of 150MB/s.

xfs/internal/sw	xfs/external/sw	xfs/internal/hw	xfs/external/hw	ext4/internal/sw	ext4/external/sw	ext4/internal/hw	ext4/external/hw	zfs/internal/sw	zfs/external/sw	zfs/internal/hw	zfs/external/hw
bonnie
read MB/s	129	141	202	205	121	120	193	195	53	55	168	166
write MB/s	156	155	187	185	162	176	160	156	95	77	80	80
mixed MB/s	45	46	70	69	53	54	73	71	37	36	54	50
sequential create /s	3874	1887	17307	5238	>99999	>99999	15240	>99999	22975	>99999	>99999	>99999
sequential delete /s	1958	2025	9946	2466	>99999	>99999	>99999	>99999	>99999	>99999	>99999	>99999
random create /s	3909	2208	12599	2754	>99999	>99999	15058	>99999	>99999	22269	>99999	22067
random delete /s	3366	1993	6963	1836	>99999	>99999	>99999	>99999	>99999	>99999	15542	>99999
randomio
create 4GB file MB/s	125	126	110	102	144	143	123	141	76	75	76	65
random io/s	408	381	550	496	192	190	605	609	443	451	304	318
random read latency ms	11	10	22	24	49	49	20	20	23	22	33	31
random write latency ms	66	75	7	7	61	62	6	6	23	24	32	32
random read latency std dev ms	11	8	50	52	38	38	54	47	32	27	63	38
random write latency std dev ms	54	60	53	53	40	40	56	46	43	61	48	43
ffsb
multithreaded read io/s	534	542	488	488	1079	1058	549	1018	496	511	300	319
multithreaded write io/s	70	98	118	168	66	108	173	175	52	33	145	34
multithreaded mixed io/s	241	282	294	408	316	301	447	601	290	300	282	244
multithreaded mixed huge io/s	4375	4389	3563	3480	7077	7928	3862	4529	6323	5727	2766	3119
kernel
untar kernel s	69	42	11	10	4	5	11	4	15	18	21	18
copy kernel s	194	197	213	212	27	33	70	26	4	3	38	4
tar kernel s	161	162	213	215	8	11	27	7	30	26	31	32
postmark 1
read KB/s	421	536	1670	781	5340	6680	8900	8900	26710	13360	13360	8900
write KB/s	1310	1660	5300	2420	16970	21210	28280	28280	84830	42410	42410	28280
postmark 10
read KB/s	515	817	2260	1180	3880	2710	5430	9050	9050	13570	9050	13570
write KB/s	1580	2500	7090	3700	12150	8510	17020	28360	28360	42540	28360	42540
postmark 100
read KB/s	591	1010	3020	1940	1130	868	13570	9050	6780	9050	9050	9050
write KB/s	1810	3150	9450	6080	3550	2660	42540	28360	21270	28360	28360	28360