Author: Willem Jan Withagen (wjw@withagen.nl), date: 24 February, 2004
Driven by remarks on the FreeBSD mailing lists and personal unhappiness about the (then) current nfs_mount(1) defaults, I started some tests to see where these claims could be substantiated.
Personally I found in my environment that using UDP (instead of TCP) and a 16 Kbytes read/write size gave better performance.
From the man page:
Bonnie tests the speed of file I/O from standard C library calls. It reads and writes 8KB blocks to find the maximum sustained data rate (usually limited by the drive or controller) and additionally rewrites the file (better simulating normal operating conditions and quite dependent on drive and OS optimization). The per character read and write tests are generally limited by CPU speed only on current generation hardware. It takes some 35 SPECint92 to read or write a file at a rate of 1MB/s using getc() and putc(). The seek test results depend on the buffer cache size, since the fraction of disk blocks that fits into the buffer cache will be found without any disk operation and will contribute zero seek time samples. (See BUGS below.)
In general Bonnie is considered a (very) poor test for disk IO-performance, but in this case it is used as a simple first approach to see what is all going on, and to get a first indication of what to expect. So I created a simple setup and ran the first initial tests.
ML-370 functions as a NFS-Server with a regular/default NFS setup. It is connected through a plain 100Mbit switch with a client: Toy. This client is run with several memory sizes as NFS-client.
ML-370 has a fully empty disk (/mnt4) which is formatted with the normal FreeBSD setup which is mounted/unmounted in every Bonnie run. The disk is rm-ed with all leftovers before a new test is started, it is not newly formatted. This partition is mounted by a single client (Toy) Which runs sequences of Bonnies, in varying size: 1,2, 4, 8, 16, 32, 64, 128, 194, 256, 384, 512 Mbytes, on the mounted disk (/mnt4) with various size parameters of the mount command. Used is a full range: 1, 2, 4, 8, 16, 32, 64 Kbytes. Every mountsize:filesize combination is tested several times (3-6 times) to get some averaging on the results. Per combination the protocol is also iterate over UDP and TCP.
Running these tests there are very little errors, close to 0, on either of the ethernet interfaces. As well as in the NFS services the number of errors is next to zero ( < 0.001% )
Bonnie is run for 2 different system configs on the test disk on the ML-370 server:
# Debugging for use in -current options DDB
#Enable the kernel debugger
options INVARIANTS #Enable calls of extra sanity checking
options INVARIANT_SUPPORT #Extra sanity checks of internal structures,
#required by INVARIANTS
options WITNESS #Enable checks to detect deadlocks and cycles
options WITNESS_SKIPSPIN #Don't run witness on spinlocks for speed
Legend:
Robert Watson writes: "We turn them off in releases,
and once 5.x becomes 5-stable, we'll turn it off by default also. However,
they're invaluable tools when debugging the development system, so we have
them on in the development branch by default. I would encourage people to
generally run with them turned on unless performance of a system requires
them to be off, as it really helps the debugging process, as well as
helping to identify locking problems as the system evolves."
There measurements are based on 12 runs for each sample point.
In essence is this the most important result, since block-reading is what is done most on general systems.
The NFS-server's kernel (ML370MIN) does not have the WITNESS and INVARIANT
debugging features included.
The client (TOY) is an old(er) PII system, running 4.6-RELEASE.
| systems | type | CPU | Memory | OS | |
| ML-370 | Compaq ML-370 | PIII 667 Mhz | 256Mb | FBSD 5.2.* | NFS server |
| Toy | Clone | PII 233 Mhz | 64,96,128 Mb | FBSD 4.6-RELEASE | Client |
On both the server and clients all processes which are not required for the tests are terminated, especially cron(1), syslog(1) and sendmail(1).
