Yesterday I made a very bad mistake. A big-ol' oopsie-woopsie on my part.
I overwrote /dev/sda with a linux installer ISO.
sudo dd if=Downloads/trisquel-something-something.iso of=/dev/sda status=progress
I've read stories of similar things happening before. The first one that comes to mind is the tale of a newbie sysadmin rm -rf'ing / on a VAX system, and recovering using only GNU emacs and a hand-assembled tar binary.
In their case, the unfortunate sysadmin still had some of /usr and a working shell. I had neither.
For one thing, I don't have a traditional linux distro setup. I keep my rootfs read-only on a squashfs image, with my archive of images stored on an encrypted SSD with btrfs. So by dd'ing sda, I had borked the first few gigs of my image archive, which wound up containing most of my rootfs image. sda was unsalvageable and I couldn't even get a shell running.
Where do we go from here?
Well my first step, of course, was to run dd again! This time from another computer, to burn a copy of the latest Gentoo administration CD.
(This time I double checked I was writing to the write disk :) )
The choice of livecd didn't matter too much, I just needed something with cryptsetup >= 2, and I'm pretty familiar with Gentoo.
So, step one is to keep calm.
Step two is to get my encryption keys off my boot key.
cryptsetup open /dev/sdf6 boot
mount /dev/mapper/boot /mnt/key
cp /mnt/key/path/to/initrd.cpio.gz /tmp
gzip -d initrd.cpio.gz
A funny quirk of the Gentoo livecd is that it doesn't have cpio, but it does have busybox, which includes a cpio applet. Not quite sure why, but I wasn't going to go through the effort of burning another livecd that doesn't have that quirk.
cpio -i < initrd.cpio
bash: cpio: command not found
busybox cpio -i < initrd.cpio
alias cpio="busybox cpio"
cryptsetup open --header=keys/hiigara-root.header --key-file=keys/hiigara-root.keyfile /dev/sda hroot
cryptsetup open --header=keys/hiigara-home.header --key-file=keys/hiigara-home.keyfile /dev/sda hhome
I open /dev/sda because, of course, I need to re-format it with btrfs and put back my images.
mount /dev/mapper/hroot /mnt/hroot
mount /dev/mapper/hhome /mnt/hhome
Now let's get to restoring those images! I opened my /home because that's where my image development environment is. With my development environment, I could just place the most recent image in /mnt/hroot and things would be mostly fine. I would still need to re-write some files in /etc.
cp vroot.x86_64.sfs /mnt/hroot
btrfs subvol create /mnt/hroot/rw
mount /mnt/hroot/vroot.x86_64.sfs /mnt/temp
cp -a /mnt/temp/rw/etc /mnt/hroot/rw
cp -a /mnt/temp/rw/var /mnt/hroot/rw
echo hiigara > /mnt/hroot/rw/etc/hostname
I also need to add my user account. To do this, I'll just add it to the livecd and copy the last line to the almost-rescued system.
tail -n 1 /etc/passwd >> /mnt/hroot/rw/etc/passwd
tail -n 1 /etc/group >> /mnt/hroot/rw/etc/group
tail -n 1 /etc/shadow >> /mnt/hroot/rw/etc/shadow
That should be enough, I've got my system image restored and my day-to-day user account added! One reboot later and...
hiigara login: [eostr - eos - iausdgiuasyhfgiuasgf - why isnt my keyboard working?]
Huh. That's weird. My keyboard isn't working, my display is running at the standard 640x480, and lightdm isn't starting.
Hey wait a minute, did I add my kernel modules?
[reboot into the gentoo livecd, decrypt and mount my drives, etc]
bash: make: command not found
The Gentoo livecd doesn't include make!?
No, of course not. I suppose a minimal gentoo install environment doesn't need much beyond some filesystem tools and tar.
Luckily, we have my image development environment! We can use the chroot I already have built!
cd /mnt/hhome/eostre/Documents/Projects/Project\ Abyssal\ Penguin\ x64/vroot.x64 # it's an old codename
mount --bind /dev dev
mount --bind /sys sys
mount --bind /proc proc
mount --bind ../../linux-5.4.15 mnt
cp /etc/resolv.conf etc/resolv.conf
And now, within the void chroot
make: error: something something something some library not found
After a lot of troubleshooting, I figured out I needed some elf-related libraries.
xbps-install elfutils elfutils-devel libelf etc.
[success! after a lot of compiling]
mksquashfs modules modules5.4.15abigail3.sfs
[and now from the livecd]
cp usr/lib/modules5.4.15abigail3.sfs /mnt/hroot/
One reboot later and... everything came back up!
Mostly. I had a few more issues, but nothing that warrants a mention in this blog post.
TL;DR reformat sda, replace the system image, config some stuff in /etc, and make sure to reinstall your kernel modules!