Installing FreeBSD 9.2 with root on ZFS

Background: I’ve been using Debian for the last years on my server. Debian was always easy to upgrade, but there is no option to have ZFS running natively on the server with still out-of-the-box upgrades.

Furthermore my services were running in vmware server, a product that was easy to run at the time, but getting IPv6 support correctly working was not my idea of fun.

So I looked into FreeBSD and started to get the hang on it’s jails system. Esp. with ZFS. I showed it a friend of mine how easy it is and showed him the installation on my desktop Mac inside virtual machine.

I got stuck with selecting ZFS as root filesystem. There was no option for that ! I wondered and found out later, that the option to select ZFS as root was something my hosting service (hetzner) added to their FreeBSD setup routine.

The documentation I found (on how to install root on zfs) on the interwebs was inadequate. Some installation instructions were using commands not even needed any more , so that I created my own manual:

First: Download freebsd disk1 image, boot from that image.

Select 2 for single user boot.

Setup the partition table:

gpart create -s gpt da0                                     # creates gpt partition table on device da0
gpart add -s 64k -t freebsd-boot da0                        # creates partition for bootloader size 64 kBytes
gpart add -t freebsd-zfs da0                                # create zfs partition for the rest of the drive
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0   # write zfsbootloader to boot partition 1, adding virtual master boot loader for systems that can't boot from gpt

Setup ZFS. I’m creating a zroot/9.2 for the root directory, so we can have the next FreeBSD version 10.0 as separate zfs dataset later, when that is released.

zpool create -o altroot=/mnt -O mountpoint=none zroot /dev/da0p2    # create zfs pool named zroot on partition 2
zfs create -o mountpoint=/ zroot/9.2                                # create zfs dataset root in pool 
zfs create zroot/9.2/var
zfs create zroot/9.2/tmp
zpool set bootfs=zroot/9.2 zroot

After setting up ZFS, just copy over the system needed files:

cd /mnt                                                 
for i in base kernel lib32              # untar kernel base system and libraries
do
tar -zxpf /usr/freebsd-dist/$i.txz
done

Setting up our new machine so that it loads the zfs kernel module

echo 'zfs_load="YES"' >/mnt/boot/loader.conf    # this tells the loader to load the zfs kernel module.
echo 'zfs_enable="YES"' >/mnt/etc/rc.conf       # this will start automounting zfs filesystems other than root

From here on you can setup your network configuration or root password, but the core os should run:

reboot
Mail.app on Mavericks with GMail

I’ve had problems with Mail.app behaving weird on Mavericks. As you know from an earlier post, I’m using GMail accounts and have set them up in the past so that GMail’s labels - which are different to imap folders - work.

To get Mail.app working with Gmail, you have to first take all accounts offline (via the menu). Then remove the [Gmail] imap path prefix in your settings:

On iOS 7 you need to remove the imap path prefix as well.

After that enable the All Mail in Gmails webfrontend:

Remove all [Gmail] prefixes from labels within Gmails webfrontend. Restart Mail.app and taking all accounts online again and you are back in business.

Additional note: I’ve had problems with search in Mail on Mavericks. I could find messages in system spotlight, but not within Mail.app. I’ve reset spotlight on the whole disk and now everything works like a charm.

Time to say goodbye to ZEVO-ZFS and GreenBytes

Up until now I was using ZEVO on my main machine and backup machine. I even wrote a complete TimeMachine like backup system for ZFS just because I needed it ( http://github.com/jollyjinx ). I heavily rely on my data being correct, the main reason to go with ZFS and ZEVO.

But as a developer I need to work at least on the current version of the OS.

Communication from GreenBytes has been almost non existent:

  • Last update to ZEVO: More than a year ago.
  • Last board announcement regarding new versions: More than a year ago.
  • Beta versions available for Mavericks: NONE
  • Support for Mavericks announced on twitter a few weeks ago. I have not seen even a beta version given that Mavericks beta versions were available since June, not to mention that is out already.

This is clearly a way on loosing potential customers (I purchased ZEVO). I’m done with GreenBytes and will not recommend any of its products in the future.

ZEVO had the potential - I hope that OpenZFS succeeds as HFS+ is just intolerable in regards to data integrity.

P.S.: I’ll switch to 10.9, I will keep my backups on a freebsd machine (on ZFS) to be sure that data does not corrupt after backing up, but on my main machine I will switch to HFS+ again.

A way to ensure consistent data for me is to have my important data being checked by git.

Update: It looks as if the CEO of GreenBytes has resigned and a couple of people left the company: The Register: Greenbytes chairman and CEO resigns

Setting up Mail.app to work with GMail

Update: This post relates to iOS 6 and Mac OSX 10.8 for iOS7 and OX 10.9 look at my newer post.

Maybe I’m not the only one who wants to have a correct Mail.app setup for GMail.

If you setup a GMail account in Mail.app everything works, but not optimal. You get an akward grey [Gmail] folder (differs in some countries) with some subfolders in Mail.app. When you delete a message it will be kept in a All Mail folder and that folder contains every mail in all other folders you have. So Mail.app stores all your Mail at least twice internally.

Reason for that is, that GMail internally uses labels instead of folders and that setup does not play well with the normal Mac Setup.

There is a way to setup Mail.app to play with GMail, and here is how:

Log in to GMail via web and go under settings to “Forwarding and POP/IMAP”. Change the default _ Auto-Expunge on_ to off and the When a message is marked as deleted and expunged from the last visible IMAP folder: to Move the message to the Trash and save that.

The important thing here is a litlle remark last visible IMAP folder as we will check off the default IMAP visiblity for All Mail in the setttings for Labels next.

Label settings in GMail

Forwarding settings in GMail

Before you continue, make sure that all your own labels in GMail have a [Gmail]/ prefix.

In Mail.app change the IMAP Prefix to [GMail]: Setting Gmail prefix in Mail.app

Disable Junk filtering in Mail.app (Google does that) Disableing Spam filter in Mail.app

And while you are at it, change in the outgoing mail server settings the port number from default to 587. This will use authorized sending of mail and will keep you from getting headaches when you are in a hotel and Mail can’t send mail as port 25 is only half blocked.

Setting port 587 in Mail.app

Now Mail.app should work fine with GMail. You can Archive Mail, delete mail and won’t have cluttered your disk with every mail twice.

Remark: I used US-English settings on GMail and on the Mac. If you have different language settings it might be that you have tell Mail.app via the Mail->Mailbox->Use this mailbox for menu the correct location for Trash and Sent Mail.

For some [Gmail] is in fact [Google Mail].

Remark2: If you are using a iOS device, make sure to uncheck the ‘Archive Messages’. By unchecking you will get a correctly labeled ‘Delete Message’ button when swiping over a message.

Otherwise the same button would be labled ‘Archive Message’ and in fact move the message to Trash.

Uncheck Archive Messages on iOS

Encrypting Dropbox et alii

I stumbled upon encFS (http://www.arg0.net/encfs), which encrypts your data on top of a current filesystem, but it uses the folder structure of the underlying filesystem.

You can use encFS for Dropbox or other cloud storage as it does a one-to-one mapping between files and encrypted files. It has even a setting to let you not encrypt the filenames but only the data contained.

This is great news as you can browse your Dropbox website for removed files and you can restore them, even though they are encrypted.

Installing or building encfs on Mac OS X is a pain, which why I used Homebrew (http://mxcl.github.com/homebrew/) to install it.

Before I started using encFS, I did a few tests and things like rsynced to encFS. I got failures with that. The homebrew version of encFS does change the date/time of the destination when changing the date/time on links, not the link itself.

I looked at the code and it became clear how to fix that. While at it, I’m now using macosx.fuse (http://osxfuse.github.com) as Fuse, which is newer than the fuse version brew is using (fuse4x) .

Compiling encFS is a pain, so I’ve created a script to install it. Have a try at https://github.com/jollyjinx/encfs.macosx .

Fusion Drive last words

The 10 minutes I’ve seen in previous tests, seem to be depending on the need what has to be moved around, not something that is hardcoded. I have not seen it to exceed the 10 minutes mark though.

As FUD is not used in portable hardware that Apple sells, there is no need to worry about battery live from FUDs perspective.

FUD will work with just two HDDs

Just out of curiosity I created a Fusion drive (FUD) with two HDDs (same size, one USB, one FW connected). Guess what - FUD is shovelling the data from the second disk to the first one as soon as one accesses the files on disk2 more often.

Interesting enough it always chose the faster drive (USB in my case) to be the caching drive, independent on the order of the drives I used on the command line or the order they were connected.

Furthermore I created a FUD drive with one SSD and two HDDs. It first stored data to the SSD, then to HDD1, then to HDD2. So HDDs were not striped and even though HDD2 was faster it used the HDD1 I’ve given it on the commandline to store things after the SSD.

It did not move data around devices as well so did just exhibit normal concatenated disk characteristics, nothing FUD like.

Operating System file caches will affect FUD

Another item I found out: Blocks that are in the filesystem cache (main memory) won’t affect the caching strategy on FUD. So when you read a file that’s on the SSD and it gets later swapped out to the HDD by FUD. Accessing the file again and again won’t bring it to SSD as long it’s still in the filesystem memory cache.

Last notes

I toyed around with FUD only cause Apple did not release any technical information on how it works. Playing around with it I got a few kernel panics so be warned that using the diskutil core storage commands might not be safe.

I would not suggest using such a FUD drive in a real world scenario. But that also comes from my bad experience I had with HFS+ and TimeMachine in the past.

Fusion Drive - loose ends

As hinted in my last post, I’ve tried using ZFS on a Fusion Disk (FUD).

You can use the disk that core storage creates and use ZFS on it. You can’t set the volume format to ZFS , but you can use the device that core storage creates:

TIN>diskutil unmount blub                                                                                                                                                        
Volume blub on disk8 unmounted
TIN>diskutil eraseVolume 'ZFS Pool' '%noformat%' /dev/disk8
The specified file system is not supported by Core Storage for use on a Logical Volume
TIN>zpool create foozfs /dev/disk8
TIN>zfs list
NAME                                                        USED   AVAIL   REFER  MOUNTPOINT
foozfs                                                     776Ki   457Gi   464Ki  /Volumes/foozfs

10 Minutes ?

So I created data on it again. This time I created random data as ZFS automatically creates empty holes in files where there are zeros (sparse files). After writing the first 120 GByte to the SSD, data was written to the HDD. When I stopped the process, it immediately did copy data between HDD and SSD for about 600 seconds / 10 minutes.

Accessing the first megabyte of the files in directories 20 to 22 for a few rounds - like in my last post - I stopped that process. Core storage kept on coping data back and forth for about 10 minutes.

I wonder if the 10 minutes is somewhere hardcoded or it’s just a coincidence.

And low and behold - Yes, after those 10 minutes I started reading the first megabyte again and iostat looked like this:

          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
  124.10 1738 210.58     0.00   0  0.00  15  6 78  1.15 0.98 0.99
  124.46 1731 210.41     0.00   0  0.00  16  6 78  1.15 0.98 0.99
  122.01 2006 239.05     0.00   0  0.00  17  8 75  1.21 1.00 0.99
  104.96 2119 217.15     0.00   0  0.00  16  7 77  1.21 1.00 0.99
  101.27 2105 208.22     0.00   0  0.00  17  8 75  1.21 1.00 0.99

So moving last accessed data to the SSD is filesystem agnostic.

Addition 1: Even though that after rebooting ZFS recognizes the zpool on the FUD disk automatically it’s not feasible to use ZFS on FUD as a single ‘zfs scrub’ might drestroy the access pattern for which data should reside on the SSD and which are going to the HDD.

More on BYO Fusion drive

I wanted to know how Fusion Drive (FUD) does decide on what to put on the SSD and disabled atime. This way FUD could no longer use that for sending accessed data to the SSD. FUD works even with the mount option ‘noatime’ set.

I then wondered if it might be in fact be block oriented (which would make most senses as core storage would have not need to know the inner workings of HFS+).

Is Fusion drive block oriented ?

So I created another 30 GBytes on the FUD which then got stored on the HDD first. I then started reading just the first Megabyte of the 100MBytes large files in a loop:

TIN> while `true`
do
for d in {30..33}                                                                                                                                                                                                                                                                                                                                      
for i in $d/*; do echo $i;
dd if=$i of=/dev/zero bs=1m count=1
done
done

Which resulted in this:

    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
   65.10  51  3.24    98.04 143 13.67  82  5 13  7.47 8.27 8.31
   62.62  61  3.72    97.63 157 14.94  81  6 13  7.47 8.27 8.31
   59.12  41  2.36   102.34 161 16.07  83  5 12  7.47 8.27 8.31
   69.71  75  5.10    93.75 142 12.99  83  5 12  7.47 8.27 8.31

So FUD was reading in the first MB from the HDD but had activity on the SSD as well. I presume it to directly store it on the SSD as well as copying things from the SSD to the HDD.

I gave FUD some time as I could clearly see some IO going on after I stopped the readout. Unmounted and remounted the drive to clear it’s caches and read again:

          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    0.00   0  0.00     0.00   0  0.00   1  1 98  1.42 3.22 5.70
   97.15 775 73.52     0.00   0  0.00   6  6 88  1.42 3.22 5.70
   91.00 1420 126.17     0.00   0  0.00   8 10 82  1.42 3.22 5.70
   88.98 1425 123.80     0.00   0  0.00   6 10 85  1.39 3.18 5.67
    0.00   0  0.00     0.00   0  0.00   9 13 78  1.39 3.18 5.67

Wow. So it read the first megabyte of 300 files just from the SSD. To verify I read the whole files again:

          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
   12.75 101  1.26   128.00 200 24.96  87  7  7  7.70 7.85 8.17
   12.80 100  1.25   128.00 187 23.43  87  7  6  7.70 7.85 8.17
   12.71 102  1.26   128.00 196 24.45  85  5 10  7.70 7.85 8.17
   12.80 100  1.25   128.00 200 24.98  85  8  7  7.88 7.88 8.18
   12.92 118  1.49   122.01 211 25.12  84  7 10  7.88 7.88 8.18
   13.22  92  1.19   128.00 196 24.48  86  5  9  7.88 7.88 8.18

So the first Megabyte was read from the SSD, the rest from the HDD. Which proves for me that FUD is working block oriented. Which is good.

So I wonder if it’s possible to use a CoreStorage FUD with ZFS ;-) As stated before, I would never trust Data onto HFS+, it’s just too risky.

Fusion drive on older Macs? YES!

Since Apple has announced the Fusion drive, people have wondered if it’s possible to use the fusion drive on older machines with a SSD and HDD.

Fusion drive uses a core storage VolumeGroup as can be seen in this Apple support document

Well, let’s try create our own Fusion drive: Attached are a 120GB SSD (disk1) and a 750GB HDD (disk7) to my Mac. I attached the SSD via SATA to be sure that the system could figure out that it’s a SSD via SMART. The HDD is attached via USB. USB I chose to clearly see a difference in speed.

Part One - setting up a Fusion drive

Now enter the depths of diskutil by using Terminal:

TIN>diskutil cs create bla disk1 disk7
Started CoreStorage operation
$<3>Unmounting disk1
$<3>Repartitioning disk1
$<3>Unmounting disk
$<3>Creating the partition map
$<3>Rediscovering disk1
$<3>Adding disk1s2 to Logical Volume Group
$<3>Unmounting disk7
$<3>Repartitioning disk7
$<3>Unmounting disk
$<3>Creating the partition map
$<3>Rediscovering disk7
$<3>Adding disk7s2 to Logical Volume Group
$<3>Creating Core Storage Logical Volume Group
$<3>Switching disk1s2 to Core Storage
$<3>Switching disk7s2 to Core Storage
$<3>Waiting for Logical Volume Group to appear
$<3>Discovered new Logical Volume Group "DE85044F-EADA-4F26-93B7-8CD0ADF006EC"
$<3>Core Storage LVG UUID: DE85044F-EADA-4F26-93B7-8CD0ADF006EC
$<3>Finished CoreStorage operation


TIN>diskutil cs list
CoreStorage logical volume groups (1 found)
|
+-- Logical Volume Group DE85044F-EADA-4F26-93B7-8CD0ADF006EC
    =========================================================
    Name:         bla
    Size:         869502550016 B (869.5 GB)
    Free Space:   865776689152 B (865.8 GB)
    |
    +-< Physical Volume 682DCC34-74A4-4290-80AE-EB127BA24746
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk1s2
    |   Status:   Online
    |   Size:     119690149888 B (119.7 GB)
    |
    +-< Physical Volume 5FA828A9-3EDD-4CBC-8C93-27C0E07C2E8A
        ----------------------------------------------------
        Index:    1
        Disk:     disk7s2
        Status:   Online
        Size:     749812400128 B (749.8 GB)


TIN>diskutil coreStorage createVolume DE85044F-EADA-4F26-93B7-8CD0ADF006EC jhfs+ blub 500g                                                                                           (13:12:32)
Started CoreStorage operation
$<3>Waiting for Logical Volume to appear
$<3>Formatting file system for Logical Volume
$<3>Initialized /dev/rdisk8 as a 466 GB HFS Plus volume with a 40960k journal
$<3>Mounting disk
$<3>Core Storage LV UUID: 6CBDCB9F-A08E-4E79-BDA7-69853BA24C1E
$<3>Core Storage disk: disk8
$<3>Finished CoreStorage operation

Now in DiskUtility the individual disks no longer show, but the Logical Volume (LV) shows as one disk. Part One is finished, we’ve created a single Volume consisting of a SSD and a HDD.

Part Two - writing data to the fusion drive

Before we start testing let’s turn off spotlight on the volume:

TIN>sudo mdutil -i off /Volumes/blub
/Volumes/blub:
    Indexing disabled.

Let’s see how this Volume performs when writing data onto it:

TIN> cd /Volumes/bla
TIN> for d in {00..13}
do
mkdir $d                   
for f in {000..100}
mkfile 100m $d/$f
done

This will create directories 00 till 13 with 100 files each 100 Mbytes in size. So 14 times 10 GB a total of 140 GBytes. While this job runs, let’s see what happens on the disks:

TIN> iostat disk1 disk7 1
          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
  127.89 1138 142.14     0.00   0  0.00   2 13 84  2.36 1.75 1.23
  127.90 1180 147.39     0.00   0  0.00   2 13 84  2.36 1.75 1.23
  127.89 1147 143.21     0.00   0  0.00   2 13 84  2.36 1.75 1.23
  127.89 1178 147.15     0.00   0  0.00   4 14 83  2.49 1.79 1.25
  127.89 1150 143.66     0.00   0  0.00   5 15 81  2.49 1.79 1.25
  126.79 1192 147.59     0.00   0  0.00   3 15 82  2.49 1.79 1.25
  127.89 1096 136.91     0.00   0  0.00   4 17 79  2.49 1.79 1.25
  127.89 1176 146.85     0.00   0  0.00   3 15 82  2.49 1.79 1.25
  127.90 1183 147.77     0.00   0  0.00   2 13 84  2.53 1.81 1.26
  127.89 1144 142.90     0.00   0  0.00   2 13 84  2.53 1.81 1.26

So it’s clearly using disk1 ( the SSD ) which is expected. Later on directory 11 gets created ( a total of 120 GB had been written to the LV by then) it changes to look like this:

  127.89 1135 141.77     0.00   0  0.00   2 12 86  1.83 1.75 1.27
  127.89 1161 145.02     0.00   0  0.00   3 13 83  1.83 1.75 1.27
  127.90 1190 148.64     0.00   0  0.00   2 13 85  1.83 1.75 1.27
  127.79 1152 143.77     0.00   0  0.00   2 13 85  1.83 1.75 1.27
  127.90 1183 147.77     0.00   0  0.00   2 13 85  1.83 1.75 1.27
  127.90 1203 150.26     0.00   0  0.00   2 13 85  1.77 1.74 1.27
  121.93 1158 137.85     0.00   0  0.00   2 13 85  1.77 1.74 1.27
  127.89 1132 141.40     0.00   0  0.00   2 13 85  1.77 1.74 1.27
  127.89 1154 144.14     0.00   0  0.00   2 13 86  1.77 1.74 1.27
  127.90 1184 147.83     0.00   0  0.00   2 13 85  1.77 1.74 1.27
  127.78 1144 142.77     0.00   0  0.00   2 13 85  1.71 1.72 1.27
          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
  127.20 260 32.25     0.00   0  0.00   1  4 96  1.71 1.72 1.27
   28.00   6  0.16     0.00   0  0.00   1  0 99  1.71 1.72 1.27
    0.00   0  0.00     0.00   0  0.00   0  0 99  1.71 1.72 1.27
    0.00   0  0.00     0.00   0  0.00   0  0 99  1.71 1.72 1.27
    0.00   0  0.00     0.00   0  0.00   0  1 99  1.57 1.70 1.26
    0.00   0  0.00     0.00   0  0.00   2  1 96  1.57 1.70 1.26
    0.00   0  0.00     0.00   0  0.00   1  1 99  1.57 1.70 1.26
    0.00   0  0.00     0.00   0  0.00   0  0 99  1.57 1.70 1.26
  103.40 863 87.11   100.00   9  0.88   1  8 91  1.57 1.70 1.26
   92.77 490 44.41   127.76 165 20.55   1  6 93  1.68 1.72 1.27
   13.22  92  1.19   128.00 216 26.96   2  4 94  1.68 1.72 1.27
   12.62 104  1.28   128.00 227 28.33   1  3 96  1.68 1.72 1.27
   12.53 105  1.28   127.44 229 28.46   0  3 97  1.68 1.72 1.27
   13.83 103  1.39   128.00 228 28.45   1  3 96  1.68 1.72 1.27
   12.71 102  1.26   128.00 228 28.46   4  4 91  1.71 1.72 1.27

So it’s clearly using the HDD now. So to tell the System that we now want to have the data on the SSD we have to just access the data stored in directories 11-13:

TIN>while `true`
do
for d in {11..13}
for i in $d/*; do echo $i;
dd if=$i of=/dev/zero bs=1m
done
done

The iostat output on a different Terminal shows in the beginning:

          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    0.00   0  0.00     0.00   0  0.00  14  1 85  0.80 0.79 0.86
    0.00   0  0.00     0.00   0  0.00  13  1 86  0.80 0.79 0.86
    0.00   0  0.00     0.00   0  0.00  14  1 85  0.82 0.80 0.86
    0.00   0  0.00     0.00   0  0.00  13  1 87  0.82 0.80 0.86
    0.00   0  0.00     0.00   0  0.00  13  1 86  0.82 0.80 0.86

which is expected as everything is in the warm caches of the OS. Let’s wait a while and we see that it begins to actually read from the disks:

          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    0.00   0  0.00     0.00   0  0.00  13  1 87  1.09 0.86 0.88
    0.00   0  0.00     0.00   0  0.00  15  6 79  1.09 0.86 0.88
  128.00 1386 173.22     0.00   0  0.00  16  8 77  1.08 0.86 0.88
  127.94 654 81.70   127.48 170 21.13  13  3 83  1.08 0.86 0.88
    0.00   0  0.00   128.00 279 34.83  13  2 85  1.08 0.86 0.88
    0.00   0  0.00   128.00 280 34.95  13  2 85  1.08 0.86 0.88
    0.00   0  0.00   127.53 271 33.71  13  2 85  1.08 0.86 0.88
    0.00   0  0.00   128.00 280 34.94  14  2 84  1.16 0.88 0.88

and later in the process it reads data from the first disk as well ( presumably directory information ):

          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
   15.06  68  1.00   128.00 209 26.08   1  1 98  1.02 0.79 0.73
   16.22  73  1.15   128.00 221 27.58   1  1 98  1.02 0.79 0.73
   13.71  84  1.12   128.00 222 27.71   0  1 98  1.02 0.79 0.73
   16.70  69  1.12   128.00 215 26.86   2  1 96  1.02 0.79 0.73
   13.71  84  1.12   128.00 201 25.09   1  2 98  1.02 0.79 0.73
   14.67  72  1.03   128.00 211 26.34   1  1 98  0.93 0.78 0.72
   14.52  92  1.30   128.00 211 26.33   1  1 98  0.93 0.78 0.72
   16.70  69  1.12   128.00 215 26.83   1  1 98  0.93 0.78 0.72
   13.71  84  1.12   128.00 209 26.11   1  1 98  0.93 0.78 0.72

I left this running for quite some time to show the system that I really only read the files in 11 to 13. Then I stopped the process.

Part Three - Fusion starts working

Interesting enough the disks did not stop working ! Immediately the system began shovelling data from the SSD to the HDD (line 7 in this output was when I stopped the dd reading).

          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
   17.23  91  1.53   128.00 213 26.58   0  1 99  0.69 0.73 0.71
   15.06  68  1.00   128.00 217 27.08   2  2 96  0.80 0.76 0.71
   16.37  86  1.37   128.00 227 28.33   1  1 98  0.80 0.76 0.71
   15.06  68  1.00   128.00 204 25.46   1  2 97  0.80 0.76 0.71
   15.06  68  1.00   128.00 216 26.96   1  1 98  0.80 0.76 0.71
   41.35 169  6.81   126.80 103 12.73   5  2 92  0.80 0.76 0.71
  128.00 197 24.59   128.00 198 24.72   1  1 99  0.73 0.74 0.71
  128.00 199 24.84   128.00 198 24.71   0  1 98  0.73 0.74 0.71
  128.00 196 24.46   128.00 197 24.59   1  1 98  0.73 0.74 0.71
  128.00 236 29.46   128.00 236 29.46   1  1 98  0.73 0.74 0.71
  108.38 263 27.80   128.00 220 27.46   0  1 98  0.73 0.74 0.71
  128.00 197 24.59   128.00 196 24.46   1  1 99  0.68 0.73 0.71
  128.00 198 24.74   128.00 198 24.74   0  1 99  0.68 0.73 0.71
  128.00 215 26.84   128.00 217 27.08   0  1 99  0.68 0.73 0.71
  126.51 242 29.85   128.00 238 29.71   1  1 98  0.68 0.73 0.71
  106.31 260 26.95   128.00 212 26.46   1  1 98  0.68 0.73 0.71

The size on tranfered to/from the drive 1 and 7 is nearly identical. After around 600 seconds or around 14 GB of copying it stopped:

          disk1           disk7       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
  128.00 193 24.09   128.00 193 24.09   2  2 96  0.49 0.75 0.72
  128.00 190 23.72   128.00 190 23.72  20 14 66  0.49 0.75 0.72
  128.00 213 26.58   128.00 214 26.71   1  1 97  0.49 0.75 0.72
  128.00 236 29.46   128.00 236 29.46   2  2 96  0.49 0.75 0.72
  102.17 251 25.02   127.37 197 24.48   3  2 94  0.53 0.76 0.72
  128.00 192 23.96   128.00 192 23.96   1  1 98  0.53 0.76 0.72
  128.00 195 24.34   128.00 196 24.46   0  2 98  0.53 0.76 0.72
  128.00 216 26.96   128.00 216 26.96   0  1 99  0.53 0.76 0.72
  128.00 229 28.58   128.00 229 28.58   1  1 98  0.53 0.76 0.72
  111.75 251 27.35   128.00 216 26.96   1  1 98  0.49 0.74 0.72
  128.00 200 24.96   128.00 201 25.09   1  1 98  0.49 0.74 0.72
   86.39 124 10.44   128.00  81 10.11   0  1 99  0.49 0.74 0.72
    0.00   0  0.00     0.00   0  0.00   4  2 95  0.49 0.74 0.72
    0.00   0  0.00     0.00   0  0.00   3  1 95  0.49 0.74 0.72
    0.00   0  0.00     0.00   0  0.00   1  0 99  0.45 0.73 0.71
    0.00   0  0.00     0.00   0  0.00   3  1 96  0.45 0.73 0.71
    0.00   0  0.00     0.00   0  0.00   1  1 99  0.45 0.73 0.71
    0.00   0  0.00     0.00   0  0.00   1  0 99  0.45 0.73 0.71
    0.00   0  0.00     0.00   0  0.00   1  0 99  0.45 0.73 0.71
    0.00   0  0.00     0.00   0  0.00   1  1 98  0.41 0.72 0.71

Now, keeping fingers crossed I’ve started the readout of directories 11 to 13 again. After no longer reading from the memory cache it looked like this:

      disk1           disk7       cpu     load average
KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
15.06  85  1.25   128.00 220 27.48   1  1 98  0.27 0.51 0.66
14.96  69  1.01   128.00 225 28.08   0  1 99  0.25 0.50 0.65
14.96  69  1.01   127.40 212 26.33   0  1 99  0.25 0.50 0.65
14.96  69  1.01   128.00 210 26.21   0  1 99  0.25 0.50 0.65
16.70  69  1.12   128.00 204 25.46   1  1 98  0.25 0.50 0.65
13.58  86  1.14   128.00 210 26.21   0  1 99  0.25 0.50 0.65

Bummer.

Then I keep reading files from direcories 11 to 13 and low and behold after about an hour of reading it looked like this:

      disk1           disk7       cpu     load average
KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
128.00 2105 263.18     0.00   0  0.00   2  5 93  0.75 0.67 0.68
128.00 2096 261.97     0.00   0  0.00   3  6 91  0.75 0.67 0.68
128.00 2097 262.07     0.00   0  0.00   4  6 90  0.69 0.66 0.68
128.00 2103 262.93     0.00   0  0.00   1  6 93  0.69 0.66 0.68
128.00 2096 262.05     0.00   0  0.00   1  5 94  0.69 0.66 0.68
128.00 2091 261.42     0.00   0  0.00   2  6 92  0.69 0.66 0.68
128.00 2112 263.95     0.00   0  0.00   2  6 92  0.69 0.66 0.68
128.00 2093 261.63     0.00   0  0.00   3  7 90  0.72 0.66 0.68

Success !

So after giving it some time it actually did transfer the data to disk 1.

Btw.: I will actually not use Fusion drive on a Mac as HFS+ is not really keeping my data safe (see my HFS+ fails miserably demo). Using two HFS+ disks concatenated just increases the risk of data failure. And TimeMachine as backup has failed me as well in the past.

I’m using ZFS to store my data ;-)

Addition 1: The SSD is a old Vertex2 so nothing Apple specific.

Addition 2: Mounting the drive with “-o noatime” does not change the caching behaviour, which is good.

Addition 3: I was using plain old 10.8.2, nothing special.

Addition 4: Before you see your drives in diskutil core storage you need to convert them with ‘diskutil cs convert’

One ring Net to bind them all

Why current newly developed social networks will fail

Currently I see a lot of people starting/developing new social networks. Primary reason has been the announcements from Twitter.

All the developers that start such a new social network are driven by the best motives, wanting to have a more open, twitter style, free social net.

I hate to see that great positive energy being wasted, but it seems that all of the developers do not seem to be interested in the real success of »their« social net. It looks to me as if in the end they will fail they at least can say: “well, I’ve tried hard”.

But that’s not enough.

What’s required for success

I do see the need of just one social network. It would make things way easier for users. But a few things are needed to make this network successful. And by successful I mean more than a billion users.

On day one of availability:

  • It has to cover at least what Facebook, Twitter, Instagram, Flickr, Tumblr, Jabber, Weibo, Path - to name a few - currently offer. It needs to be more attractive to »normal« users to make a switch.
  • Is has to be available on day one on the Web, Windows, Windows mobile, Android, iOS, Mac and in different languages and localizations.
  • When you move to this service it has to be able to incorporate all your connected people/friends/followers from the different existing services you use now and reconnect them automatically in a way that when they join they are automatically added to the different groups a user has.
  • Users have full control over their data, their followers/friends and the data a user shares to its different groups.
  • Good defaults for family, friends, acquaintances sharing.
  • Very easy to understand interface with options for advanced users.
  • Make it possible to move a user account inside this network from one server to another.
  • Must be free for users if they don’t’ want to pay. This makes it easy for providers to have an incentive to create a free server and use advertisements to cover the costs.

Development features:

  • It has to have an open interface so there can exist different implementations. an open initial implementation is best to get hundreds/thousands of servers world wide during the initial startup phase.
  • It needs to be distributed and you can setup your own server. ISPs would have an incentive to setup servers for their users like e-mail.
  • It should be easy to implement a minimal client so there will be enough developers who are willing to write clients for it. distributed, to spread high demand data quickly and not overload a single little server when somebody posts something interesting for the world.
  • Near realtime search for post and users (this might be the hardest given the distributed nature)
  • It needs headroom to scale up to 10 billion people within the next decade.
  • Nice to have:
    • Incorporate history (not just social network connections) from other services.
    • Realtime audio/video conferencing.
    • File sharing between users.
    • Trending topics.
    • Prevent spam through the protocol. Or easy for users to tag spammers for others.