Migrating to new host#

Prepare salt configuration for migration#

Ensure that salt-master, loadbalancer, and host in question can be brought up with vagrant locally, and that their health check for the relevant service is failing in haproxy after the host is fully up

To view haproxy status:

vagrant up the salt-master, loadbalancer, and host in question (vagrant up)
Prepare an ssh configuration file to access the host with native ssh commands: vagrant ssh-config salt-master loadbalancer >> vagrant-ssh
Open an ssh session with port forwarding to the haproxy status page: ssh -L 4646:127.0.0.1:4646 -F vagrant-ssh loadbalancer
view the haproxy status page in your browser http://localhost:4646/haproxy?stats

Edit pillar data for roles.sls to include both old and new hostnames (ex. hostname*)

diff --git a/pillar/prod/roles.sls b/pillar/prod/roles.sls
index 68387c9..7a8ace1 100644
--- a/pillar/prod/roles.sls
+++ b/pillar/prod/roles.sls
@@ -35,7 +35,7 @@ roles:
     purpose:  "Builds and serves CPython's documentation"
     contact:  "mdk"
   downloads:
-    pattern:  "downloads.nyc1.psf.io"
+    pattern:  "downloads*.nyc1.psf.io"
     purpose:  "Serves python.org downloads"
     contact:  "CPython Release Managers"
   hg:

ssh into the salt-master server ssh salt.nyc1.psf.io
Navigate to srv/psf-salt user@salt:~$ cd /srv/psf-salt
Run user@salt:/srv/psf-salt$ sudo git pull
Run highstate to update the roles settings to reflect the new matchng pattern, as well as additional changes to support migration: user@salt:/srv/psf-salt$ sudo salt-call state.highstate

Start a new droplet in digital ocean, and check resources being used on old host to see if we are over or under spending on resources
Create a new droplet with new version of ubuntu, appropriate resources, and name it according to hostname + 2004

ssh into new-host via the IP address provided by DigitalOceanssh root@NNN.NNN.NNN.NNN
Add Salt repositories for our current target version (add the apt-repo and install salt-minion package)

user@new-host:~$ wget --quiet -O /usr/share/keyrings/salt-archive-keyring.gpg https://repo.saltproject.io/py3/ubuntu/20.04/$(dpkg --print-architecture)/3004/salt-archive-keyring.gpg
user@new-host:~$ echo "deb [signed-by=/usr/share/keyrings/salt-archive-keyring.gpg arch=$(dpkg --print-architecture)] https://repo.saltproject.io/py3/ubuntu/20.04/$(dpkg --print-architecture)/3004 focal main" > /etc/apt/sources.list.d/salt.list

user@new-host:~$ apt-get update -y && apt-get install -y --no-install-recommends salt-minion
On the old-host, look through /etc/salt/minion.d* to set up salt-minion configuration files to match on new-host:
- run user@old-host:~$ for file in /etc/salt/minion.d/*; do echo -e "cat > $file <<EOF"; sudo cat $file; echo "EOF"; done to generate bash that will create these files

Restart the salt-minion service on the new host to pickup the configuration and register with salt-master: user@new-host:~$ sudo salt-call service.restart
On salt-master, accept the key for the new-host: user@new-host:~$ sudo salt-key -a new-host
On the new-host, run highstate sudo salt-call sate.highstate
Log out of root session
Ensure that the new host is not passing health checks in the loadbalancer: ssh -L 4646:127.0.0.1:4646 lb-a.nyc1.psf.io then open http://localhost:4646/haproxy?stats in your browser.
Run hightstate on the salt-master to create a public dns record for the new-host user@salt:/srv/psf-salt$ sudo salt-call state.highstate

laptop:psf-salt user$ ssh -A new-host into new host to enable forwarding of ssh-agent
Stop cron jobs user@new-host:~$ sudo service cron stop
Stop public-facing services, like nginx, or the service the health check is looking for. Use this command as an example: user@new-host:~$ sudo service nginx stop
Ensure that any additional volumes are mounted and in the correct location:

Check what disks are currently mounted and where: df
Determine where any additional disks should be mounted (based on salt configuration of services, for example docs and downloads roles need a big /srv for their data storage
Ensure mounting of any external disks are in the right location using mount command with appropriate arguments
Ensure that the volumes will be remounted on startup by configuring them in /etc/fstab

Run rsync once to move bulk of data and as necessary to watch for changes user@new-host:~$ sudo -E -s rsync -av --rsync-path="sudo rsync" username@old-host:/pathname/ /pathname/

The /pathname/ can be determined by looking at the pillar data for backups, pillar/prod/backup using the source_directory path for the given host (example: the downloads host uses /srv/)

ssh into old-host ( laptop:psf-salt user$ ssh old-host )
Stop cron jobs user@old-host:~$ sudo service cron stop
Stop public-facing services, like nginx, or the service the health check is looking for ex) user@old-host:~$ sudo service nginx stop

Run rsync once more to finalize data migration user@new-host:~$ sudo -E -s rsync -av --rsync-path="sudo rsync" username@hostname: /pathname/ /pathname/
Start cron jobs user@new-host:~$ sudo service cron start
Start public-facing services involved with healthcheck, like nginx, user@new-host:~$ sudo service nginx start
Ensure that the new-host is live and serving traffic by viewing loadbalancer page:

view the haproxy status page in your browser http://localhost:4646/haproxy?stats

user@new-host:~$ for user in /home/psf-users/*; do sudo -E -s rsync --delete -av --progress --rsync-path="sudo rsync" user@old-host:$user/ $user/migrated-from-ubuntu-1804-lts-host/; done

On old-host, stop the old-host by running, user@old-host:~$ sudo shutdown -h now
Destroy the old-host in DigitalOcean
Change the new-host name in DigitalOcean by removing the suffix or similar that was used to differentiate it from the old-host.
Run user@salt:~$ sudo salt-key -L to list out and user@salt:~$ sudo salt-key -d old-host to remove old keys
On new-host, run user@new-host:~$ sudo hostname new-host to rename
Update new-host name in /etc/hostname, /etc/salt/minion_id, and /etc/hosts
Restart the salt minion user@new-host:~$ sudo salt-call service.restart
Restart datadog user@new-host:~$ sudo service datadog-agent restart
Run user@salt:~$ sudo salt-key -a new-host to accept new keys
Run highstate user@salt:~$ sudo salt-call state.highstate on salt-master to update domain name as well as known_hosts file