Solaris - 2

Whenever I build openssl from source (like today for openssl-1.0.0c) I am annoyed by the problem described in section E.3 of this document.

The recommended workaround is to build custom versions of the values-X[act].o objects, so that gcc links every program against these instead of their original versions in /usr/lib. They contain an (admittedly nice) hack to jump over the zeroes (instead of NOPs) inserted by the SUN linker. I didn't like that. A broad workaround for an isolated problem... In fact, there are voices saying that this behaviour of Solaris ld is a feature, not a bug.

I prefer to patch openssl, like this:

diff -ur openssl-1.0.0c~/crypto/perlasm/x86gas.pl openssl-1.0.0c/crypto/perlasm/x86gas.pl
--- openssl-1.0.0c~/crypto/perlasm/x86gas.pl 2008-12-17 20:56:47.000000000 +0100
+++ openssl-1.0.0c/crypto/perlasm/x86gas.pl 2011-01-31 17:22:20.551210508 +0100
@@ -211,7 +211,6 @@
.section .init
call $f
jmp .Linitalign
-.align $align
.Linitalign:
___
}

I.e., simply remove the alignment at the end of the custom .init section in the assembler files (which are generated via a preprocessing script written in perl).

For a long time I had trouble building and running newer versions of ffmpeg on Solaris. It would throw errors at me like:

[NULL @ 8be5140] [Eval @ 8046b9c] Invalid chars 'x01000004' at the end of expression '0x01000004'
[NULL @ 8be5140] Unable to parse option value "0x01000004"

The processing of the command line parameters seemed to be botched. The following patch fixes the problem for me:

--- libavutil/opt.c~ 2010-09-29 23:42:03.000000000 +0200
+++ libavutil/opt.c 2010-10-15 17:42:25.051174575 +0200
@@ -240,7 +240,7 @@
if (o_out) *o_out= o;

switch (o->type) {
- case FF_OPT_TYPE_FLAGS: snprintf(buf, buf_len, "0x%08X",*(int *)dst);break;
+ case FF_OPT_TYPE_FLAGS: snprintf(buf, buf_len, "%u", *(unsigned*)dst);break;
case FF_OPT_TYPE_INT: snprintf(buf, buf_len, "%d" , *(int *)dst);break;
case FF_OPT_TYPE_INT64: snprintf(buf, buf_len, "%"PRId64, *(int64_t*)dst);break;
case FF_OPT_TYPE_FLOAT: snprintf(buf, buf_len, "%f" , *(float *)dst);break;


A few months ago I published a modified zpool program to create pools with a higher ASHIFT value, suitable for disks with 4k sectors. I expected the program compiled on s10u8 to be upward compatible. Apparently, it isn't.

To compile "zpool" on s10u9, I used the sources from OpenSolaris b138. The instructions how to compile are mostly unchanged from s10u8. The only required additional step is to compile and link against /usr/src/cmd/stat/common/timestamp.c. I am omitting further details.

For Solaris 11 Express, I used b147 as the basis. I was unable to find a complete on-src.tar.bz2 for this release, but had to pull the sources from hg.openindiana.org via hg (mercurial). I then compiled with:

cd /tmp
mkdir -p usr/src
cd ~ftp/pub/OpenSolaris/hg.openindiana.org/onnv-gate/usr/src
find \
cmd/stat/common \
common/zfs \
cmd/zpool \
lib/libuutil/common \
lib/libdiskmgt/common \
| cpio -pmdv /tmp/usr/src

cd /tmp/usr/src/cmd/stat/common
gcc -O2 -DTEXT_DOMAIN='"en_US"' -c timestamp.c
cd /tmp/usr/src/cmd/zpool
ln -s /usr/lib/libuutil.so.1 libuutil.so
gcc -O2 \
-Dzpool_rewind_policy_t=zpool_load_policy_t \
-DZPOOL_REWIND_REQUEST=ZPOOL_LOAD_REWIND \
-DZPOOL_REWIND_POLICY=ZPOOL_LOAD_POLICY \
-DZPOOL_REWIND_REQUEST_TXG=ZPOOL_LOAD_REWIND_TXG \
-DZPOOL_NO_REWIND=ZPOOL_NORMAL_LOAD \
-DTEXT_DOMAIN='"en_US"' \
-I/tmp/usr/src/cmd/stat/common \
-I/tmp/usr/src/common/zfs \
-I/tmp/usr/src/lib/libuutil/common \
-I/tmp/usr/src/lib/libdiskmgt/common \
-c *.c

gcc -o zpool *.o ../stat/common/timestamp.o \
-L. \
-lzfs \
-lnvpair \
-ldevid \
-lefi \
-ldiskmgt \
-luutil \
-lumem \
-L/lib -lcryptoutil

The modification to zpool_vdev.c is always the same (at line 474/475):

--- zpool_vdev.c.orig 2011-01-11 16:01:14.906720955 +0100
+++ zpool_vdev.c 2011-01-11 16:39:00.334827548 +0100
@@ -471,6 +471,7 @@
verify(nvlist_add_string(vdev, ZPOOL_CONFIG_PATH, path) == 0);
verify(nvlist_add_string(vdev, ZPOOL_CONFIG_TYPE, type) == 0);
verify(nvlist_add_uint64(vdev, ZPOOL_CONFIG_IS_LOG, is_log) == 0);
+ verify(nvlist_add_uint64(vdev, ZPOOL_CONFIG_ASHIFT, 12) == 0);
if (strcmp(type, VDEV_TYPE_DISK) == 0)
verify(nvlist_add_uint64(vdev, ZPOOL_CONFIG_WHOLE_DISK,
(uint64_t)wholedisk) == 0);

For your convenience, I offer pre-compiled versions of the modified zpool program for s10u8, s10u9 and Solaris 11 Express.

When building "unrar" from Source do not forget to set "-DALLOW_NOT_ALIGNED_INT". This provides a huge performance benefit for encrypted files.

I use:

make -f makefile.unix CXX=g++ CXXFLAGS="-O2 -march=pentium3" DEFINES="-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DLITTLE_ENDIAN -DALLOW_NOT_ALIGNED_INT" STRIP=/bin/true DESTDIR=/usr/local

Solaris 10 10/09 09/10 (a.k.a. s10u9) has arrived, and so have my two WD20EARS disks. Unfortunately, the situation regarding the ashift value is pretty much unchanged from a users point of view.

The latest Solaris release does seem to contain the changes described in PSARC 2008/769, however it still creates zpools on a WD20EARS drive (with 4K physical sectors) with an ashift value of 9.

I am not sure what the reason for this is. Most likely the WD drives do not tell the truth even about their physical sector size. What a pity. So, my workaround (the modified zpool program) is still necessary. It works on s10u9 without changes.

There is one more issue with these disks. Allegedly they unload their heads after an idle period of only 8 seconds. Many users have reported very high head load/unload counts as reported in S.M.A.R.T. data. As a precaution, I have put

set zfs:zfs_txg_timeout = 5

into /etc/system. The default in s10u9 is still 30 seconds, although the latest opensolaris code (as of mid-August, when Oracle stopped their putbacks) also sets it to 5 seconds. This should keep the disk busy in case of sporadic updates

For your convenience, you can find the zpool binary mentioned below to create pools with an "ashift" value of 12 here.

Some days ago, I was just about to purchase two of those WD20EARS disks, when I stumbled over reports about their bad performance together with ZFS. I had a look at the I/O pattern produced writing large chunks of zeroes to one of my current disks, using "iosnoop" from the excellent DTrace Toolkit.

(...)
102 64 0 0 W 1103045891 131072 sched
102 64 0 0 W 1103046147 131072 sched
102 64 0 0 W 1103046403 131072 sched
102 64 0 0 W 1103046659 131072 sched
102 64 0 0 W 1103046915 131072 sched
102 64 0 0 W 1103047171 131072 sched
102 64 0 0 W 1103047427 131072 sched
(...)

We see that Solaris nicely chunks the data up into 128K blocks, but the I/O's are to uneven block numbers, so indeed the EARS drives will have a problem with this (lots of read-modify-write cycles needed). What's worse, we have also writes smaller than 4096 bytes, probably metadata:

(...)
102 64 0 0 W 268531009 1536 sched
102 64 0 0 W 1226324923 1536 sched
102 64 0 0 W 1226324926 1024 sched
102 64 0 0 W 268531012 1024 sched
102 64 0 0 W 847259994 1024 sched
102 64 0 0 W 1226324928 1024 sched
(...)

Fortunately, ZFS (even as delivered in stock Solaris 10) has a way to enforce a proper alignment of sectors. It is the "ashift" parameter which is determined at pool creation time and stored as part of the pool configuration. The "zdb -C " command show this:

(...)
$ zdb -C be02|tail -6 | sed -e"s/^ *//"
metaslab_array=23
metaslab_shift=32
ashift=9
asize=750134231040
is_log=0
DTL=128
(...)

The "ashift=9" means that blocks will have an alignment of 2**9 = 512 bytes. What we would want is an alignment of 2**12 = 4096 bytes.

So, how to increase this "ashift" value? As far as I know, there is no way to increase it without recreating the pool. Because if it was possible, existing blocks would need to be re-aligned and understandably this would be quite hard to do on a live zpool!

Fair enough, so, how to specify this "ashift" value at pool creation time? Well, in stock Solaris 10 there is no way to do that. The "ashift" value is not among the configuration parameters supplied by zpool to the ZFS_IOC_POOL_CREATE ioctl. It is currently left to the kernel to derive a proper value (from the hardware) for "ashift". In normal circumstances, this is probably the correct way to do it, in our case however, we want control over this value.

So we need to compile our own version of zpool, where we add this configuration parameter. I found out that b116 is the OpenSolaris version whose code is likely to be compatible with Solaris 10 10/09 (s10u8). So I grabbed on-src.tar.bz2 and extracted ./usr/src/cmd/zpool.

Without a proper build enviroment in place, i tried to compile it on-the-fly, and discovered that I need a few more source components. The following shows the steps needed to recompile "zpool":

$ cd /tmp
$ tar -b120 -xvf on-src.tar.bz2 \
./usr/src/cmd/stat/common \
./usr/src/common/zfs \
./usr/src/cmd/zpool \
./usr/src/lib/libuutil/common/ \
./usr/src/lib/libdiskmgt/common/
$ cd ./usr/src/cmd/zpool
$ ln -s /usr/lib/libuutil.so.1 libuutil.so
$ gcc -O2 -DTEXT_DOMAIN='"en_US"' \
-I/tmp/usr/src/cmd/stat/common \
-I/tmp/usr/src/common/zfs \
-I/tmp/usr/src/lib/libuutil/common \
-I/tmp/usr/src/lib/libdiskmgt/common \
-c *.c
$ gcc -o zpool *.o -L. -lzfs -lnvpair -ldevid -lefi -ldiskmgt -luutil -lumem

I tried whether this "zpool" was functional, and it indeed worked as the original one.

We now need to extend zpool_vdev.c to append "ashift" to the list of pool properties "zpool" passes on the ZFS ioctl. The relevant function seems to be "make_leaf_vdev". A first attempt, where I modifed "construct_spec" was unsuccessful.

--- zpool_vdev.c.orig 2009-06-01 06:33:27.000000000 +0200
+++ zpool_vdev.c 2010-08-07 20:07:53.010531000 +0200
@@ -471,6 +471,7 @@
verify(nvlist_add_string(vdev, ZPOOL_CONFIG_PATH, path) == 0);
verify(nvlist_add_string(vdev, ZPOOL_CONFIG_TYPE, type) == 0);
verify(nvlist_add_uint64(vdev, ZPOOL_CONFIG_IS_LOG, is_log) == 0);
+ verify(nvlist_add_uint64(vdev, ZPOOL_CONFIG_ASHIFT, 12) == 0);
if (strcmp(type, VDEV_TYPE_DISK) == 0)
verify(nvlist_add_uint64(vdev, ZPOOL_CONFIG_WHOLE_DISK,
(uint64_t)wholedisk) == 0);

Please forgive me that I actually hardcoded a value of "12", but this is currently only meant as a proof-of-concept anyway.

Recompile, and we now have a "zpool" command which creates zpools with an "ashift" value of 12. Let's try it out:

$ ./zpool create be02 c0d0s7
$ zdb -C be02|tail -6 | sed -e"s/^ *//"
whole_disk=0
metaslab_array=23
metaslab_shift=29
ashift=12
asize=59430928384
is_log=0

And let's have a look at the I/O pattern on this zpool:

(...)
102 7 0 0 W 163904 131072 sched
102 7 0 0 W 164160 131072 sched
102 7 0 0 W 164416 131072 sched
102 7 0 0 W 164672 131072 sched
102 7 0 0 W 164928 131072 sched
102 7 0 0 W 165184 131072 sched
102 7 0 0 W 680912 77824 sched
102 7 0 0 R 681216 65536 sched
102 7 0 0 W 681536 4096 sched
102 7 0 0 W 22032328 4096 sched
102 7 0 0 W 681544 4096 sched
102 7 0 0 W 22032336 4096 sched
102 7 0 0 W 681552 4096 sched
102 7 0 0 W 22032344 4096 sched
102 7 0 0 W 681560 4096 sched
102 7 0 0 W 22032352 4096 sched
102 7 0 0 W 44049928 4096 sched
102 7 0 0 W 681568 4096 sched
102 7 0 0 W 681576 8192 sched
(...)


All I/O's seem to be aligned to 4096 bytes, and also seem to have a minimum size of 4096 bytes.

I think I can now go ahead and buy those WD20EARS drives...

As disk capacity keeps growing to 2TB per disk and beyond, the traditional sector size of 512 bytes becomes more and more inefficient. Some vendors have already introduced disks with an internal sector size of 4096 bytes, e.g. Western Digital with their WDxxEARS series.

Some operating systems (e.g. Microsoft Windows) are not able to support sector sizes of more than 512 bytes, so the disk uses 4096 bytes only internally, and behaves to the outside world like a traditional disk.

For Solaris 10 and ZFS this poses a major problem, as ZFS assumes the sector size the drive reports to it is also its physical sector size. If the drive reports a sector size of 512 bytes, it will allocate data on 512 byte sector boundaries and it will read and write data in chunks of at least 512 bytes. Of course, ZFS will also write data in much bigger chunks, but only to optimize the overall I/O pattern. As a consequence, the newer 4K-sector-disks will exhibit terrible performance problems (the disk has to carry out read-modify-write cycles).

Oracle has addressed this problem in PSARC 2008/769. This seems to be a fairly generic and flexible solution, however it will only be available in the next release of Solaris. It might be already integrated into OpenSolaris, but a migration to OpenSolaris is currently not an option for me, even though I'm only using it at home.

So, is there a solution to the problem in Solaris 10? Yes, there is.

Stay tuned.

Situation: Home network, partly Ethernet, partly WLAN. The probability that someone sniffes your network traffic or does a brute-force-attack on your WPA2 encryption is rather low. Still, you may want to have your telnet and r*-Services disabled.

Imagine one or more of your computers are real legacy hardware (Pentium 3) and are running Solaris 10 x86. Using SSH out of the box is very slow for big file transfers (scp, sftp), as the payload gets encrypted with aes-128-ctr by default. Applying "-c arcfour" helps a bit, but only so much.

Using "-c none" does not work, as "none" is not configured as a valid SSH2 cipher in cipher.c:

(...)
} ciphers[] = {
{ "none", SSH_CIPHER_NONE, 8, 0, 0, EVP_enc_null },
{ "des", SSH_CIPHER_DES, 8, 8, 0, EVP_des_cbc },
{ "3des", SSH_CIPHER_3DES, 8, 16, 0, evp_ssh1_3des },
{ "blowfish", SSH_CIPHER_BLOWFISH, 8, 32, 0, evp_ssh1_bf },
{ "3des-cbc", SSH_CIPHER_SSH2, 8, 24, 0, EVP_des_ede3_cbc },
{ "blowfish-cbc", SSH_CIPHER_SSH2, 8, 16, 0, EVP_bf_cbc },
(...)

Now, recompiling ssh from OpenSolaris Source is quite a bit of work and how would you install that on top of your Solaris 10? The same problem applies to OpenSSH, as Sun added a few features and you may want to be 100% compatible to Sun's SSH.

What do you do? Well, you patch the binaries:

$ pwd
/usr/bin
$ ls -l ssh ssh.orig
-r-xr-xr-x 1 root bin 239432 Feb 17 16:50 ssh
-r-xr-xr-x 1 root bin 239432 Aug 9 2009 ssh.orig
$ cmp -lc ssh.orig ssh
231141 0 ^@ 375 ý
231142 0 ^@ 377 ÿ
231143 0 ^@ 377 ÿ
231144 0 ^@ 377 ÿ
$ cd /usr/lib/ssh
$ ls -l sshd sshd.orig
-r-xr-xr-x 1 root bin 328136 Feb 17 15:52 sshd
-r-xr-xr-x 1 root bin 328136 Aug 9 2009 sshd.orig
$ cmp -lc sshd.orig sshd
318869 0 ^@ 375 ý
318870 0 ^@ 377 ÿ
318871 0 ^@ 377 ÿ
318872 0 ^@ 377 ÿ

Now, Solaris will happily accept "none" as a SSH2 cipher.