e9d.org

contact
whoami
blog/
dist/

wiki/
FAQ

résumé
donate

legal


Yesterday I made a very bad mistake. A big-ol' oopsie-woopsie on my part.
I overwrote /dev/sda with a linux installer ISO.
	sudo dd if=Downloads/trisquel-something-something.iso of=/dev/sda status=progress
	[enter]
---
Oops!
--
I've read stories of similar things happening before. The first one that comes to mind is the tale of a newbie sysadmin rm -rf'ing / on a VAX system, and recovering using only GNU emacs and a hand-assembled tar binary.
In their case, the unfortunate sysadmin still had some of /usr and a working shell. I had neither.
For one thing, I don't have a traditional linux distro setup. I keep my rootfs read-only on a squashfs image, with my archive of images stored on an encrypted SSD with btrfs. So by dd'ing sda, I had borked the first few gigs of my image archive, which wound up containing most of my rootfs image. sda was unsalvageable and I couldn't even get a shell running.
Where do we go from here?
---
Well my first step, of course, was to run dd again! This time from another computer, to burn a copy of the latest Gentoo administration CD.
(This time I double checked I was writing to the right disk :) )
The choice of livecd didn't matter too much, I just needed something with cryptsetup >= 2, and I'm pretty familiar with Gentoo.
So, step one is to keep calm.
Step two is to get my encryption keys off my boot key.

	cryptsetup open /dev/sdf6 boot
	mount /dev/mapper/boot /mnt/key
	cp /mnt/key/path/to/initrd.cpio.gz /tmp
	cd /tmp
	gzip -d initrd.cpio.gz

A funny quirk of the Gentoo livecd is that it doesn't have cpio, but it does have busybox, which includes a cpio applet. Not quite sure why, but I wasn't going to go through the effort of burning another livecd that doesn't have that quirk.

	cpio -i < initrd.cpio
	bash: cpio: command not found
	busybox cpio -i < initrd.cpio
	alias cpio="busybox cpio"
	cryptsetup open --header=keys/hiigara-root.header --key-file=keys/hiigara-root.keyfile /dev/sda hroot
	cryptsetup open --header=keys/hiigara-home.header --key-file=keys/hiigara-home.keyfile /dev/sda hhome

I open /dev/sda because, of course, I need to re-format it with btrfs and put back my images.
mkfs.btrfs /dev/mapper/hroot mount /dev/mapper/hroot /mnt/hroot mount /dev/mapper/hhome /mnt/hhome Now let's get to restoring those images! I opened my /home because that's where my image development environment is. With my development environment, I could just place the most recent image in /mnt/hroot and things would be mostly fine. I would still need to re-write some files in /etc. cp vroot.x86_64.sfs /mnt/hroot btrfs subvol create /mnt/hroot/rw mount /mnt/hroot/vroot.x86_64.sfs /mnt/temp cp -a /mnt/temp/rw/etc /mnt/hroot/rw cp -a /mnt/temp/rw/var /mnt/hroot/rw umount /mnt/temp echo hiigara > /mnt/hroot/rw/etc/hostname I also need to add my user account. To do this, I'll just add it to the livecd and copy the last line to the almost-rescued system. useradd eostre passwd eostre [hunter2] tail -n 1 /etc/passwd >> /mnt/hroot/rw/etc/passwd tail -n 1 /etc/group >> /mnt/hroot/rw/etc/group tail -n 1 /etc/shadow >> /mnt/hroot/rw/etc/shadow That should be enough, I've got my system image restored and my day-to-day user account added! One reboot later and... hiigara login: [eostr - eos - iausdgiuasyhfgiuasgf - why isnt my keyboard working?] --- Huh. That's weird. My keyboard isn't working, my display is running at the standard 640x480, and lightdm isn't starting. Hey wait a minute, did I add my kernel modules? [reboot into the gentoo livecd, decrypt and mount my drives, etc] cd /mnt/hhome/eostre/Documents/Projects/linux-5.4.15 make modules_install bash: make: command not found The Gentoo livecd doesn't include make!? No, of course not. I suppose a minimal gentoo install environment doesn't need much beyond some filesystem tools and tar. Luckily, we have my image development environment! We can use the chroot I already have built! cd /mnt/hhome/eostre/Documents/Projects/Project\ Abyssal\ Penguin\ x64/vroot.x64 # it's an old codename mount --bind /dev dev mount --bind /sys sys mount --bind /proc proc mount --bind ../../linux-5.4.15 mnt cp /etc/resolv.conf etc/resolv.conf chroot . And now, within the void chroot xbps-install -Su cd /mnt make clean make -j10 make: error: something something something some library not found After a lot of troubleshooting, I figured out I needed some elf-related libraries. xbps-install elfutils elfutils-devel libelf etc. make clean make -j10 [success! after a lot of compiling] make modules_install cd /usr/lib mksquashfs modules modules5.4.15abigail3.sfs exit [and now from the livecd] cp usr/lib/modules5.4.15abigail3.sfs /mnt/hroot/ One reboot later and... everything came back up! Mostly. I had a few more issues, but nothing that warrants a mention in this blog post. --- TL;DR reformat sda, replace the system image, config some stuff in /etc, and make sure to reinstall your kernel modules! ________________________________________________________________________________ (C)/AGPLv3