Welp, after a few harrowing days, I managed to recover all my data from a failing ZFS pool. It had been serving me well, with no significant issue since 2013, about 10 years at this point. But the time has come to
destroy it, which I did. Goodbye old friend. 😢
On the bright side, I have two shiny new pools with shiny new disks, and a bunch of old hardware that needs a new use.
Also, I have to say that using a tool that automatically takes ZFS snapshots on some kind of schedule is a must-have. I have a filesystem that was badly corrupted but one of the snapshots survived intact. I was lucky that it was mostly a slowly-changing filesystem and the snapshot was up to date. I use
sanoid for this purpose but I suppose there are many.
@email@example.com phu ZFS. Long time ago that I did that Solaris Admin Course 1 and 2. It is nice to work with ZFS when it works and you know how to set them up
@firstname.lastname@example.org Oof, a long story. One disk went bad, and I replaced the disk. As that disk was resilvering, a second disk started dropping off the bus and then reappearing. This array is 6 disks arranged in 3 mirrors (so, 2 disks for each mirror), and that meant I had two mirrors with only one disk each supporting them for awhile 😱 Anyway, something about that disk disappearing and reappearing threw the entire array into…..disarray (pardon the pun). I can’t even explain what happened but it was really in a bad state and the resilvering just wouldn’t complete.
I bought some new disks, made a new array, and used
zfs send to get as many filesystems as possible from the old array to the newly-built one. One filsesystem was cranky, so I used an older snapshot of that one instead and was lucky that it worked fine. Finally, I
rsynced a few directories that seemed like they might have been out of date in that old snapshot, and that worked too.
Having a tool that automatically takes snapshots on a regular interval saved my ass.
@email@example.com Backups were at least an hour old whereas I sent snapshots from the existing and fully up to date file systems.
@firstname.lastname@example.org Yeah I try to minimize TLAs