Now I’m a little bit crazy. I’m not your typical sysadmin with a whole lotta training classes and certification under my belt, but what I do have is imagination. I had it in my imagination to get enough storage to backup our existing file server, which has about 18TB currently. Now the “oringinal” idea that was pitched by my boss was to buy consumer grade disks and put them into a bunch of workstations we have and use something like Lustre or some other cluster file system to create a space large enough to do a backup.
On a side note to those wondering why we were thinking this is that our existing tape library is well on the fritz. It was dropped on its head at birth and has suffered other damage by some other idiot techs who manhandled it on install. So its always been a bit moody since then. Anyways back to the brainstorming. Since I would be the one tasked with setting this up and managing it, I was thinking, “OMG there has to be a better way then this mess.” And thus comes in all the useless knowledge I gather by reading tech sites like Engadget, Gizmodo, Slashdot, etc.
Drobo! I’ve read about it and have dreamed about it and would love nothing better then to have a DroboFS of my own to replace this silly 500GB usb drive I currently have at home that is shared out via my desktop. Looking at their site, I see a deal for two DroboPro plus 2TB disks to fill it all up. With Drobo’s overhead, each DroboPro comes to about 14TB each. They also come with the ability to connect over Ethernet via iSCSI. All this for a low great price. Well low enough when compared to enterprise level storage from Sun(which is what our file server uses anyways).
Some more side story: In the beginning our file server was setup using Sun’s file system, whatever it is. I wasn’t the sysadmin then, in fact I’m just an undergrad right now working towards my math degree. Again back on topic. Some CS student wrote some nasty program that generated millions of empty files that ate up all the inodes and poof file server down. At this point ZFS had come out and we though, sure why not. It doesn’t have inodes, so no problem. ZFS is in my opinion the best file system out there. They just need to add support for a cluster version to allow for system failover like in Lustre or GFS and that would be the sweet spot.
So I sign for the packages and get to work right away. At this point I wasn’t really sure of what I was going to do or how to do it. It every much was a trial and error kind of thing. I started off with CentOS 5.5 and used LVM to take the standard 7x2TB luns that are the default for DroboPro, and formated the whole thing with XFS. I tried out XFS since ext3, which is the only Linux filesystem “offically” supported(its states beta), but ext3 doesnt do well for very large filesystems. But turns out XFS wasnt much good either. I wanted to test out the repair functionality of xfs, but xfs_repair -n would crap out with an I/O error. I think it was expecting more of the underlying hardware to be exposed, but Drobo stands in the way with their BeyondRAID tech that runs internal to the array.
Next up was Solaris, but that ended rather quickly. Their iSCSI initatior didn’t get along with Drobo. It would complain about some unsupported/unknown opt code 0xE0, and the Drobo itself would flash its lights as if it was constantly booting on and off. With all that going on, the zpool create command was running terabily slow. So that failed.
I was reading up on ZFS and Linux when I read ZFS had been ported to FreeBSD. Now I’ve never touched BSD before but I’m always up to a challenge. Installed FreeBSD rather quickly, ok. Got iSCSI working with the Drobo, great. Ran zpool create, and it worked flawlessly, awesome! Now I’m ready to start figuring out the best way to get data from the file server to this machine. I could use zfs send | ssh zfs receive but ssh encryption would be too much overhead. I think I’ll go with setting up an NFS server on this machine and pass zfs snapshots into files onto the NFS share. Then at my leisure, pass the snapshot files back into zfs on the backup machine.
So thats things so far. I’ll post later when I get the backup system I’ve thought up working out.