I was reminded to data about the availability issues with storage. Of course, I live it every day on the “sell side” working for Pure Storage. But for my home lab and office, I tend to have “less than enterprise” requirements.
Even so, my primary use of my DroboElite has been for secondary copies (backups) of data on VMs. These include my primary storage (VM boot images, as well as file shares where I store all of my music, video and of course photos).
The iSCSI DroboElite works with VMware ESXi (VSphere 5.5) clusters, even though it is only qualified with ESXi 4.x. The DroboElite isn’t very fast, but it’s worked well for me for a few years. It’s survived power outages despite lack of a UPS, and in general I’ve been happy to have it, although I’ve been frustrated at times with the performance as well as with it’s predilection for rebooting when being touched by non-qualified OSes such as FreeBSD.
Today I deleted a 4TB volume from my DroboElite, removing it from my cluster. It had about 2TB of data on it, but it was a full ZFS send/receive copy of data on another DroboElite volume. I was in the process of moving data from a block-aligned FreeBSD (virtualized) ZFS pool to a 4k-aligned ZFS pool (using these instructions, which I’ve been successful with in the past). I wanted to start the process from scratch, so I deleted the old 4k-aligned pool from FreeBSD, removed it from the Drobo, created a new 4TB volume and exported it to the host and consumed it as an RDM in my guest VM.
I noticed that space on the Drobo, which is scavenged after a volume deletion asynchronously as a background operation, was very slow to free up. I had plenty of space on the Drobo, so started creating my new pool.
During this operation, which has never given me issues in the past, the Drobo rebooted (I could tell by the change in fan speed). It then went into endless reboot cycling. The blue lights, which light up one by one, were my benchmark for how far I was getting. The Drobo would get to light 8 and then reboot.
I followed the instructions at Drobo’s KB article #102 but that was no help.
The worst part about this was that I had broken my own rule, and had two “necessary” VMs living on the Drobo: One of my routers (my primary PFSense router instance) and the vCenter Appliance.
Lucky for me I was running CARP redundancy and my other router (another VM running under VMware Fusion on a Mac mini in my garage) took over.
I lost one two pools on my FreeBSD file server, but the second was just a copy of the first, and the first had only backups of my all-SSD data pool, plus TimeMachine backups from our desktops and laptops. Do no primary data from that VM was lost. I also have been toying around with ZFS in another instance, using Passthrough disks, as an NFS server to replace my iSCSI Drobo, so I had yet another copy of everything.
This was a good scare. Time for me to go back and rebuild my backup strategy.
Also time for me to look into getting a Synology or other dedicated hardware array for my office. I don’t trust the Drobo, and am going to want something with a bit more performance.