Xubuntu on HP z420 (rebuild post crash)



Introduction

The bad kernel release for the Spectre/Meltdown bugs hit me pretty hard. Somehow I ended up with a very corrupt root FS. Oddly it only affected some files and directories but the amount of broken stuff was too much to fix and have any confidence of the system running well again.

I grabbed my Xubuntu 16.04.3 LTS USB stick and booted the live OS. For some reason, I needed to modify the kernel boot parameters to get it up. Hit 'e' delete 'quiet splash' and replace with 'verbose nomodeset' then F10.

After booting up I brought up some terminals and mounted an external USB harddrive to catch and carry any data I could save off the old system.

Save off old data

I wish I would have logged this step more, but basically I had to

vgchange -a y MYHOSTNAME-vg

the mounted the root LVM LV on /mnt/root. I checked each directory that I wanted to try to save with:

find . -type f -exec cksum {} > /dev/null \;

Bad files would give an Input/Output error and meant they were unrecoverable.

I make daily backups to a locally ecrypt'ed directory which I then rsync over to another encrypted directory on a remote CIFS server. The CIFS server files were corrupt but I'm not sure why. The local files were not corrupted so I needed to get them over to the USB HD.

Luckily I had this step documented on a different computer, but basically you mount on top of the mount with the right options:

ECRYPT_OPTIONS="key=passphrase,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_passthrough=n,ecryptfs_enable_filename_crypto=n"

mount -t ecryptfs "/scratch/myusername/encrypted" "/scratch/myusername/encrypted" -o "${ECRYPT_OPTIONS}"

After moving a few more non-encrypted directories over to the USB HD, I unmounted everything and rebooted. This time I removed all of the USB HDs other than the USB stick for the Live OS. Again booting the Live OS with nomodeset.

Prep new filesystems

My goal now is to whack and re-create the LVM setup that I had before. First, for pedantic state, I removed all of the existing LVs, then the VG, then the PVs. Next I zero'ed out the first 8M of each PV partition:

dd if=/dev/zero of=/dev/sda1 bs=8M count=1 #future LVM cache

dd if=/dev/zero of=/dev/sdc1 bs=8M count=1

dd if=/dev/zero of=/dev/sdd3 bs=8M count=1

Now create the PVs

pvcreate -v /dev/sdc1 /dev/sdd3

Now create the VG

vgcreate -v myhost-vg /dev/sdd3 /dev/sdc1

Now the LVs

lvcreate -L 32G -n swap_1 myhost-vg

lvcreate -l 100%FREE -n root myhost-vg

Reboot and choose Install Xubuntu with nomodeset kernel option

After the installer starts, hit Super-L T for a terminal.

sudo su -

vgchange -a y

mkfs.ext4 -L host-root /dev/mapper/hostname--vg-root

mkswap /dev/mapper/hostname--vg-swap_1

exit # and close terminal

Choose Other for filesystem layout. Mount LV root as root and format as XFS, swap as swap, non-LV ext2 partition as /boot and format.

Reboot and cross fingers.

Data Restoration

Mount the encrypted external USB HDs. Create a temporary holding space on the root FS.

sudo mkdir /recovery

sudo cp -a /path/to/hd/recovery/* /recovery

sudo su -

mkdir /tmp/restore

cd /tmp/restore

tar -j --checkpoint=1024 --extract --listed-incremental=/dev/null --file /MYHOSTNAME-recovery/MYHOSTNAME-backups/MYUSERNAME.homedir.0.tbz2

tar -j --checkpoint=1024 --extract --listed-incremental=/dev/null --file /MYHOSTNAME-recovery/MYHOSTNAME-backups/MYUSERNAME.homedir.1.tbz2

tar -j --checkpoint=1024 --extract --listed-incremental=/dev/null --file /MYHOSTNAME-recovery/MYHOSTNAME-backups/MYUSERNAME.homedir.2.tbz2

tar -j --checkpoint=1024 --extract --listed-incremental=/dev/null --file /MYHOSTNAME-recovery/MYHOSTNAME-backups/MYUSERNAME.homedir.3.tbz2

cd /home/MYUSERNAME

mkdir FACTORY-FRESH

mv * FACTORY-FRESH

mv .???* FACTORY-FRESH

mv /tmp/restore/MYUSERNAME/* /tmp/restore/MYUSERNAME/.???* .

Install symlinks to make sure the restored links look good

apt-get install symlinks

symlinks -r . | grep dangling

Post-installation Configuration (Part 1)

Drivers and Kernel Parameters

Settings -> Additional Drivers

Nvidia

Intel

Settings -> Ubuntu Software

Ubuntu Software

Source Code

Fix default kernel parameters

edit /etc/default/grub

## remove nomodeset (from install)

## change quiet splash to verbose

sudo update-grub

Lightdm Configuration

edit /etc/lightdm/lightdm-gtk-greeter.conf

## add this line to the end of the file

display-setup-script = xrandr --output DP-2 --primary --output DP-4 --right-of DP-2 --rotate left --output DVI-I-0 --off --output DVI-I-1 --off

reboot

Install SSHD

Make the system remotely accessible

sudo apt-get install openssh-server

Configure LVM Cache

fdisk -l /dev/sda

Disk /dev/sda: 447.1 GiB, 480103981056 bytes, 937703088 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 11224C49-D9C5-45E2-82A8-31BB652E2207

Device Start End Sectors Size Type
/dev/sda1 2048 419432447 419430400 200G Linux LVM
/dev/sda2 419432448 937699327 518266880 247.1G Microsoft basic data

pvcreate /dev/sda1

vgextend -v MYHOSTNAME-vg /dev/sda1

lvcreate -n temp -l 38 MYHOSTNAME-vg /dev/sda1

lvcreate -n CacheMetaLV -L 150M MYHOSTNAME-vg /dev/sda1

lvcreate -n CacheDataLV -l 100%FREE MYHOSTNAME-vg /dev/sda1

lvremove MYHOSTNAME-vg/temp

lvconvert --chunksize 512K --type cache-pool --poolmetadata MYHOSTNAME-vg/CacheMetaLV MYHOSTNAME-vg/CacheDataLV

lvconvert --type cache --cachepool MYHOSTNAME-vg/CacheDataLV MYHOSTNAME-vg/root

lvs -a MYHOSTNAME-vg

echo "dm_cache" >> /etc/initramfs-tools/modules

echo "dm_cache_mq" >> /etc/initramfs-tools/modules

echo "dm_persistent_data" >> /etc/initramfs-tools/modules

echo "dm_bufio" >> /etc/initramfs-tools/modules

## watch for borked copy and paste since sometimes spaces aren't spaces, though they should be

##
cat > /etc/initramfs-tools/hooks/lvmcachechech.sh <<'EOF'

#!/bin/sh

PREREQ="lvm2"

prereqs()

{

echo "$PREREQ"

}

case $1 in

prereqs)

prereqs

exit 0

;;

esac

if [ ! -x /usr/sbin/cache_check ]; then

exit 0

. /usr/share/initramfs-tools/hook-functions

copy_exec /usr/sbin/cache_check

manual_add_modules dm_cache dm_cache_mq

EOF

chmod 700 /etc/initramfs-tools/hooks/lvmcachechech.sh

apt-get install thin-provisioning-tools

update-initramfs -u -k all

sudo lvs -a -o+chunksize

Network configuration

update /etc/dhcp/dhclient.conf

## add send host-name

send host-name = "MYHOSTNAME.gearboxsoftware.com";

Update Network Manager configuration

/etc/NetworkManager/system-connections/Wired connection 1

[ipv4]

dhcp-client-id=MYHOSTNAME.gearboxsoftware.com

may-fail=false

Sound Configuration

edit /etc/pulse/default.pa

### Should be after module-*-restore but before module-*-detect

load-module module-switch-on-port-available

load-module module-switch-on-connect

Administrative Mail

## add MAILTO="MYFIRSTNAME.MYLASTNAME@gearboxsoftware.com" to

/etc/cron.d/anacron

/etc/crontab

/etc/cron.d/sysstat

System Variables

# edit /etc/default/rcS and add:

UTC=no

VERBOSE=yes

FSCKFIX=yes

# edit /etc/sysctl.conf and add:

fs.inotify.max_user_watches = 524288

Additional Software Installation

Google Chrome

http://askubuntu.com/questions/510056/how-to-install-google-chrome

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -

sudo sh -c 'echo "deb https://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'

sudo apt-get update

sudo apt-get install google-chrome-stable

# since the google package installs its own source file whack ours

sudo rm /etc/apt/sources.list.d/google.list

Perforce

https://www.perforce.com/downloads/helix-visual-client-p4v

https://www.perforce.com/downloads/helix-command-line-client-p4

cd /usr/local

(umask 022; sudo tar zxvf /tmp/p4v.tgz)

export P4VER=p4v-2017.3.1601999

export ALT_VER=20180112

sudo cp /tmp/p4 /usr/local/${P4VER}/bin

sudo chmod 0755 /usr/local/${P4VER}/bin/p4

sudo chown -R bin:bin ${P4VER}

sudo mkdir -m 0755 -p /usr/share/icons/perforce

sudo update-alternatives --install /usr/local/bin/p4 p4 /usr/local/${P4VER}/bin/p4 ${ALT_VER}

sudo update-alternatives --install /usr/local/bin/p4v p4v /usr/local/${P4VER}/bin/p4v ${ALT_VER} \
--slave /usr/share/icons/perforce/p4merge.svg p4merge.svg /usr/local/${P4VER}/lib/p4v/P4VResources/icons/p4merge.svg \
--slave /usr/share/icons/perforce/p4v.svg p4v.svg /usr/local/${P4VER}/lib/p4v/P4VResources/icons/p4v.svg \
--slave /usr/share/icons/perforce/p4admin.svg p4admin.svg /usr/local/${P4VER}/lib/p4v/P4VResources/icons/p4admin.svg

sudo update-alternatives --display p4v

sudo update-alternatives --display p4

Jetbrains IDEs

Rubymine

https://www.jetbrains.com/ruby/download/download-thanks.html?platform=linux

cd /usr/local

export RMVER=2017.3.1

export RM_ALT_VER=20180112

(umask 022; sudo tar zxvf /tmp/RubyMine-${RMVER}.tar.gz )

sudo chown -R bin:bin RubyMine-${RMVER}

sudo mkdir -p -m 0755 /usr/share/icons/rubymine

sudo update-alternatives --install /usr/local/bin/rubymine.sh rubymine.sh /usr/local/RubyMine-${RMVER}/bin/rubymine.sh ${RM_ALT_VER} \
--slave /usr/share/icons/rubymine/rubymine.png rubymine.png /usr/local/RubyMine-${RMVER}/bin/rubymine.png \
--slave /usr/share/icons/rubymine/RMlogo.svg RMlogo.svg /usr/local/RubyMine-${RMVER}/bin/Rmlogo.svg

sudo update-alternatives --display rubymine.sh

Goland

https://www.jetbrains.com/go/download/download-thanks.html?platform=linux

cd /usr/local

export GOLANDVER=2017.3

export GOLANDMINOR=1

export GOLANDALTVER=20180112

(umask 022; sudo tar zxvf /tmp/goland-${GOLANDVER}.${GOLANDMINOR}.tar.gz )

sudo chown -R bin:bin GoLand-${GOLANDVER}

sudo mkdir -p -m 0755 /usr/share/icons/goland

sudo chown root:root /usr/share/icons/goland

sudo update-alternatives --install /usr/local/bin/goland.sh goland.sh /usr/local/GoLand-${GOLANDVER}/bin/goland.sh ${GOLANDALTVER} \
--slave /usr/share/icons/goland/goland.png goland.png /usr/local/GoLand-${GOLANDVER}/bin/goland.png

sudo update-alternatives --display goland.sh

Pycharm

https://www.jetbrains.com/pycharm/download/download-thanks.html?platform=linux&code=PCC

cd /usr/local

export PYVER=2017.3.2

export PYALTVER=20180112

(umask 022; sudo tar zxvf /tmp/pycharm-community-${PYVER}.tar.gz )

sudo chown -R bin:bin pycharm-community-${PYVER}

sudo mkdir -p -m 0755 /usr/share/icons/pycharm

sudo update-alternatives --install /usr/local/bin/pycharm.sh pycharm.sh /usr/local/pycharm-community-${PYVER}/bin/pycharm.sh ${PYALTVER} \
--slave /usr/share/icons/pycharm/pycharm.png pycharm.png /usr/local/pycharm-community-${PYVER}/bin/pycharm.png

sudo update-alternatives --display pycharm.sh

Crossover

https://www.codeweavers.com

sudo dpkg -i crossover_17.0.0-1.deb

sudo apt-get -f install

Slack

https://slack.com/downloads/instructions/ubuntu

sudo dpkg -i /tmp/slack-desktop-3.0.2-amd64.deb

sudo bash /etc/cron.daily/slack

sudo chmod 0644 /etc/apt/sources.list.d/slack.list

Oracle Java

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

export JAVA_MINOR=152

export JAVA_ALT_VER=20180112

sudo mkdir -p /usr/lib/jvm

sudo chmod 0755 /usr/lib/jvm

cd /usr/lib/jvm

(umask 022; sudo tar zxvf /tmp/jdk-8u${JAVA_MINOR}-linux-x64.tar.gz)

sudo gzip -9v jdk1.8.0_${JAVA_MINOR}/man/man1/*

sudo chown -R bin:bin jdk1.8.0_${JAVA_MINOR}

export LINKDIR=/usr/bin

export MANLINK=/usr/share/man/man1

export JRELINK=/usr/lib/jvm/jdk1.8.0_${JAVA_MINOR}

export JHOME=${JRELINK}

export JREDIR=${JHOME}/jre/bin

export JDKDIR=${JHOME}/bin

export MANDIR=${JHOME}/man/man1

bash -s <<'EOF'

sudo update-alternatives --install ${LINKDIR}/java java ${JREDIR}/java ${JAVA_ALT_VER} \

--slave ${LINKDIR}/jcontrol jcontrol ${JREDIR}/jcontrol \

--slave $MANLINK/java.1.gz java.1.gz ${MANDIR}/java.1.gz

sudo update-alternatives --install ${LINKDIR}/jjs jjs ${JREDIR}/jjs ${JAVA_ALT_VER} \

--slave $MANLINK/jjs.1.gz jjs.1.gz ${MANDIR}/jjs.1.gz

sudo update-alternatives --install ${LINKDIR}/policytool policytool ${JREDIR}/policytool ${JAVA_ALT_VER}

sudo update-alternatives --install ${LINKDIR}/servertool servertool ${JREDIR}/servertool ${JAVA_ALT_VER} \

--slave $MANLINK/servertool.1.gz servertool.1.gz ${MANDIR}/servertool.1.gz

sudo update-alternatives --install ${LINKDIR}/keytool keytool ${JREDIR}/keytool ${JAVA_ALT_VER} \

--slave $MANLINK/keytool.1.gz keytool.1.gz ${MANDIR}/keytool.1.gz

sudo update-alternatives --install ${LINKDIR}/orbd orbd ${JREDIR}/orbd ${JAVA_ALT_VER} \

--slave $MANLINK/orbd.1.gz orbd.1.gz ${MANDIR}/orbd.1.gz

sudo update-alternatives --install ${LINKDIR}/pack200 pack200 ${JREDIR}/pack200 ${JAVA_ALT_VER} \

--slave $MANLINK/pack200.1.gz pack200.1.gz ${MANDIR}/pack200.1.gz

sudo update-alternatives --install ${LINKDIR}/rmid rmid ${JREDIR}/rmid ${JAVA_ALT_VER} \

--slave $MANLINK/rmid.1.gz rmid.1.gz ${MANDIR}/rmid.1.gz

sudo update-alternatives --install ${LINKDIR}/rmiregistry rmiregistry ${JREDIR}/rmiregistry ${JAVA_ALT_VER} \

--slave $MANLINK/rmiregistry.1.gz rmiregistry.1.gz ${MANDIR}/rmiregistry.1.gz

sudo update-alternatives --install ${LINKDIR}/tnameserv tnameserv ${JREDIR}/tnameserv ${JAVA_ALT_VER} \

--slave $MANLINK/tnameserv.1.gz tnameserv.1.gz ${MANDIR}/tnameserv.1.gz

sudo update-alternatives --install ${LINKDIR}/unpack200 unpack200 ${JREDIR}/unpack200 ${JAVA_ALT_VER} \

--slave $MANLINK/unpack200.1.gz unpack200.1.gz ${MANDIR}/unpack200.1.gz

sudo update-alternatives --install ${LINKDIR}/javaws javaws ${JREDIR}/javaws ${JAVA_ALT_VER}

EOF

cd /usr/lib/jvm/jdk1.8.0_${JAVA_MINOR}/jre/lib/security

sudo unzip -j /tmp/jce_policy-8.zip

sudo chown -R bin:bin /usr/lib/jvm/jdk1.8.0_${JAVA_MINOR}

Docker

sudo apt-get install docker docker.io

sudoedit /lib/systemd/system/docker.service

## add '--dns MYDNSSERVER' before -H

sudo systemctl daemon-reload

sudo systemctl restart docker.service

cd /tmp

curl -sL https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` > /tmp/docker-compose

sudo mv /tmp/docker-compose /usr/local/bin

sudo chown bin:bin /usr/local/bin/docker-compose

sudo chmod 0755 /usr/local/bin/docker-compose

sudoedit /etc/group

## add your user to docker group

Additional Repository Software

sudo apt-get install xscreensaver xscreensaver-data-extra xscreensaver-gl vim nedit fonts-inconsolata fonts-ubuntu-font-family-console fonts-ubuntu-title ttf-ubuntu-font-family ttf-xfree86-nonfree xfonts-base curl rxvt-unicode-256color rxvt screen ctags xdaliclock fortune-mod fortunes-ubuntu-server fortunes-off fortunes-min xfonts-100dpi xscreensaver-data openconnect network-manager-openconnect{,-gnome} openjdk-8-jdk libbonobo2-bin libgnomevfs2-bin libgnomevfs2-extra gnome-mime-data libice-doc libsm-doc libxcb-doc libxt-doc openjdk-8-demo ruby awscli xfce4-artwork xfce4-battery-plugin xfce4-cpufreq-plugin xfce4-datetime-plugin xfce4-diskperf-plugin xfce4-goodies xfce4-sensors-plugin seahorse rdesktop xfsprogs btrfs-tools openssh-server dump ttf-bitstream-vera ri ruby-dev bundler xsensors attr quota cifs-utils xfonts-terminus xfonts-terminus-oblique ntp devilspie2 dos2unix terminator vinagre iperf sysstat

sudo apt-get -f install

sudo fc-cache -f -v

sudo apt-get install virtualbox virtualbox-dkms virtualbox-qt

sudo apt-get install virtualbox-ext-pack

sudo dpkg --add-architecture i386

sudo apt-get update && sudo apt-get install steam

sudo apt-get install pulseaudio-module-bluetooth

sudoedit /etc/pulse/default.pa

## add load-module module-switch-on-connect

## after load-module module-switch-on-port-available

sudo apt-get install xfsdump lzop

Golang 1.9

export GOVER=1.9

export GOMINOR=2

cd /tmp

wget https://storage.googleapis.com/golang/go${GOVER}.${GOMINOR}.linux-amd64.tar.gz

(umask 022; tar zxf go${GOVER}.${GOMINOR}.linux-amd64.tar.gz)

mv go go${GOVER}

sudo mv go${GOVER} /usr/local

sudo chown -R bin:bin /usr/local/go${GOVER}

vim ~/.bashrc

export GOROOT=/usr/local/go1.9.2

export PATH=${PATH}:${GOROOT}/bin

Miscellaneous Software

sudo apt-get install icedtea-plugin ## for cisco ASDM

sudo apt install redis-tools mysql-client mysql-common libmysqlclient-dev

sudo apt-get install alpine gedit gedit-plugins pbzip2 pixz zlib1g-dev bsd-mailx

sudo apt-get install libnss-mdns:i386 libxml2:i386

User Directories on /

sudo mkdir -p /mnt/dos /mnt/cifs /scratch

sudo chown myusername:mygroupname /mnt/dos /mnt/cifs /scratch /Steam

sudo mkdir -p /Steam/{common,steam021,steam022,steam023,steam024}

sudo chown -R myusername:mygroupname /Steam

Nvidia Dmesg Warning Fix

NVRM: Your system is not currently configured to drive a VGA console
on the primary VGA device. The NVIDIA Linux graphics driver
requires the use of a text-mode VGA console. Use of other console
drivers including, but not limited to, vesafb, may result in
corruption and stability problems, and is not supported.

sudoedit /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="verbose video:vesa:off vga=normal"

sudo update-grub

sudo init 6

Search This Blog

Technical Stuff