zpool status

Got my LSI 9207 delivered and installed today!

Here is the approximate sequence of events:

  1. Updated backup
  2. Shutdown server, replaced Adaptec 6805 with LSI 9207
  3. Rebooted server into live media
  4. Discovered that I was able to see past the RAID metadata and import the existing ZFS pool
  5. Using my favorite guide on the planet, updated the GRUB bits to make sure it would boot properly
  6. Voila!  Rebooted and I was in my Ubuntu Xenial install just like before!

Curious to see if this will remedy the strange hangs my server has been experiencing lately.

The whole process took a handful of hours, but it mostly spent waiting on things (booting into live media, etc).  Very simple procedure.  We’ll see how things go from here!

Re: ZFS Compression

In preparation for the arrival of my HBA, I’m creating a backup of my server.   As things currently stand:

Filesystem      Size  Used Avail Use% Mounted on 
udev             79G     0   79G   0% /dev 
tmpfs            16G   86M   16G   1% /run 
rpool/root      6.9T  1.4T  5.5T  21% / 
tmpfs            79G   28K   79G   1% /dev/shm 
tmpfs           5.0M  8.0K  5.0M   1% /run/lock 
tmpfs            79G     0   79G   0% /sys/fs/cgroup 
cgmfs           100K     0  100K   0% /run/cgmanager/fs 
tmpfs            16G     0   16G   0% /run/user/1000

I’m using 1.4T.  Thats less than a formatted 2T hard drive.  That’ll definitely fit!

Except it doesn’t.  Left the rsync running overnight, got to work today, and the drive was full at approximately 1.8T.

Why?  Because apparently ZFS compression is doing it’s job..

That was a question I had regarding disk usage measurement with ZFS compression enabled.  du output is (surprise) how much space is used on the disk, not how much data you actually have.  In my case:

root@tnewman0:~# zfs get all rpool | grep compressratio                    
rpool  compressratio         1.17x                  - 
rpool  refcompressratio      1.00x                  -

1.17 x 1498796032 kilobytes is 1753591357 kilobytes, or 1.8T.  Tight fit.  Probably could have done a bit of slimming down and squeezed it in, but wheres the fun in that.

My solution:

root@tnewman0:~# zpool status 
  pool: backup 
 state: ONLINE 
  scan: none requested 
        NAME                            STATE     READ WRITE CKSUM 
        backup                          ONLINE       0     0     0 
          wwn-0x5000cca22de70c5e-part1  ONLINE       0     0     0 
errors: No known data errors 
  pool: rpool 
 state: ONLINE 
  scan: scrub repaired 0 in 2h15m with 0 errors on Mon Jan  9 22:03:44 2017 
        NAME                              STATE     READ WRITE CKSUM 
        rpool                             ONLINE       0     0     0 
          mirror-0                        ONLINE       0     0     0 
            scsi-22f6baa1200d00000-part1  ONLINE       0     0     0 
            scsi-22f4b9a2e00d00000-part1  ONLINE       0     0     0 
          mirror-1                        ONLINE       0     0     0 
            scsi-22f4be2f000d00000-part1  ONLINE       0     0     0 
            scsi-22f5b32bc00d00000-part1  ONLINE       0     0     0 
          mirror-2                        ONLINE       0     0     0 
            scsi-22f5b92a900d00000-part1  ONLINE       0     0     0 
            scsi-22f5bc2a900d00000-part1  ONLINE       0     0     0 
          mirror-3                        ONLINE       0     0     0 
            scsi-22f6b1ee800d00000-part1  ONLINE       0     0     0 
            scsi-22f6b5eb900d00000-part1  ONLINE       0     0     0 
          mirror-4                        ONLINE       0     0     0 
            scsi-22f7b0a1900d00000        ONLINE       0     0     0 
            scsi-22f7b4a0d00d00000        ONLINE       0     0     0 
          scsi-22f7bda1b00d00000          ONLINE       0     0     0 
          scsi-22f4b4ac400d00000-part1    AVAIL    
errors: No known data errors

Make a compression enabled pool on the external!

Aaaand now we wait for rsync to do its business..

UPDATE:  Interesting change in I/O wait time between filesystems.  When going from ZFS pool to EXT4, the average I/O wait percentage is ~13.14%.  When going from ZFS pool to ZFS pool, the I/O wait percentage is ~6.58%.

Openfire + Openfire Meetings

Didn’t take a screencap, but I now have XMPP and video chat functionality using Openfire with the Openfire Meetings plugin.

A few things I discovered..  During installation, you do need to create the database (MariaDB in my case) using:

mysql> CREATE DATABASE openfire CHARACTER SET utf8 COLLATE utf8_general_ci;

Simply ‘create database openfire;’ was not sufficient.  It was obvious that the application could connect to the database, but it couldn’t finish the rest of it’s business setting things up.

Also, read the stuff that it says during the setup wizard.  The last step has you create an administrative user.  The fields are ‘Email’, ‘Password’, and ‘Confirm Password’ (or something like that).  Then it takes you to the login screen for the first time.  The username is ‘admin’ and the password is whatever you supplied in that previous step.  I had trouble logging in because I thought that the username was my email.  It isn’t.  It’s ‘admin’.  Of course, you’re free to create other administrative users once you’re logged in.

The other bump I ran into had to do with SSL.  I’m a fan of free and letsencrypt works very well.  I put the key in like I was supposed to, put the full chain in for the cert (didn’t put anything in for ‘Password’ since the key was created without one), and whacked save.  But when I clicked on the cert I just added to view information about it (like you can do with the two existing self-signed certs) it threw up a bunch of Java errors.  systemctl restart openfire solved that problem.

Still need to do a bit more research into certs though.  When I installed the letsencrypt cert, there were three listed (letsencrypt + 2 self-signed).  I’m not sure exactly how to tell openfire which cert to use.  My low-tech solution was to (make a backup of the keystore beforehand and do a test restoration) remove the two self-signed certificates so that only my letsencrypt cert remained.  systemctl restart openfire, and the letsencrypt cert was the cert that was used for everything that used a cert, including the administrative web interface, and xmpp (I checked the cert provided in pidgin and it was using the same letsencrypt cert the browser was using).  I’m sure there’s a way to tell Openfire, “Of the certificates you have in your store, use X cert for Y service,” I’m just not sure where that is in the control panel.

UPDATE:  confirmed Openfire Meetings can be used to transmit hand farts over the internet

LSI 9207-8i Ordered

Migrating from my current Adaptec 6805 RAID controller to an LSI 9207 HBA.  Since I’m using ZFS, the RAID functionality kinda gets in the way.  It’s an added layer of complexity that I’m not using, and they’re expensive to replace.  It also locks me into a specific make (at least) of controller.  I should have used an HBA from the start, but I just went with what I had at the time.  Currently doing a full backup of /.  Hopefully things will go smoothly!

Replacing a potentially bad drive

Hot spare kicked in!  Though I did kinda force it to by yanking a drive and then offline-ing it..  I’ll make the potentially bad drive the new spare, and order a new 2TB.  Stuck labels on the ‘old spare’ (scsi-22f6baa1200d00000) as well as the ‘new spare’ (scsi-22f4b4ac400d00000).

Update:  ..and thats what I did!  Also considering ditching the SSD cache device so that I can stick another spare in that slot.  Easy way to keep everything together, and it’ll already be in the server when I need it..  Plus, that frees up the SSD for other more interesting uses.