Patching Unix Servers using ansible

 

The below command will install the systemback package & check if we have enough space on servers for OS-level backup:

1ansible -m command -a "hostname" all 2ansible -m command -a 'sudo add-apt-repository "deb http://ppa.launchpad.net/nemh/systemback/ubuntu xenial main"' all 3ansible -m command -a 'sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 382003C2C8B7B4AB813E915B14E4942973C62A1B' all 4ansible -m command -a 'sudo apt update' all 5ansible -m command -a 'sudo apt install systemback -y' all 6ansible -m command -a 'systemback-cli -s' all (Check the default directory for systemback generally its /home) 7ansible -m command -a 'sudo df -Th /' all (Check we have enough space before systemback atleast 15 Gb) 8ansible -m command -a 'sudo systemback-cli -n' all (To create Systemback restore point) 9ansible -m command -a 'sudo du -sh /home/*' all (check size of /home/Systemback ) 10ansible -m command -a 'sudo df -Th' all (Check we have enough space after systemback) 11 12ansible -m command -a "uname -r" preprod 13ansible -m command -a "df -Th" preprod

Always make sure to have enough space under/before running systemback-cli restore point creation.

• The below command will create a new restore point on all the machines:

1ansible -m command -a 'systemback-cli -n' all

 

• Once we have the restore point created using systemback-cli proceed to create cloud level snapshot

• While taking snapshots make sure MySQL is stopped so that everything is committed to the database.

1ansible -m command -a 'systemctl stop mysql' preprod

1We can run commands by logging to each server individually too: 2 3uname -a 4Linux testdbAS-1AS 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 5 6apt-get update 7Hit:1 http://repo.percona.com/prel/apt focal InRelease 8Hit:2 http://de.archive.ubuntu.com/ubuntu focal InRelease 9Get:16 10Fetched 12.2 MB in 3s (4,812 kB/s) 11Reading package lists... Done 12 13 14apt list --upgradable 15Listing... Done 16alsa-ucm-conf/focal-updates,focal-updates 1.2.2-1ubuntu0.13 all [upgradable from: 1.2.2-1ubuntu0.5] 17apt-utils/focal-updates 2.0.9 amd64 [upgradable from: 2.0.4] 18apt/focal-updates 2.0.9 amd64 [upgradable from: 2.0.4] 19base-files/focal-updates 11ubuntu5.6 amd64 [upgradable from: 11ubuntu5.3]]

 

• Install the below packages if missing

1ansible -m command -a 'apt-get update' preprod1 2ansible -m command -a 'sudo apt-get install needrestart -y' preprod1 3ansible -m command -a 'sudo apt-get install debian-goodies -y' preprod1

 

 

• List of packages to be upgraded:

1ansible -m command -a 'apt list --upgradable' preprod1

 

• Below command can apply pending patches:

1ansible -m command -a 'sudo apt upgrade -y' preprod1

 

To check which daemons need to be restarted after library upgrades

 

1ansible -m command -a 'sudo needrestart' preprod1 2ansible -m command -a 'sudo checkrestart' preprod1

 

In case patches require reboot then before reboot just make sure your application services are stopped gracefully. Please do not proceed with a reboot unless the service stops.

Mostly it would be MySQL service in database servers.

1ansible -m command -a 'systemctl status mysql' all

1systemctl stop mysql

Once the server is up let’s make sure to start the service.

1systemctl start mysql

 

Once we have systemctl and IONOS backup done below playbooks can do the needful. We can run the playbook against each server one by one and make sure it’s up properly by watching the ionos console.

 

1ansible-playbook update.yaml -i preprod1, 2ansible-playbook update.yaml --limit 'preprod' 3 4ansible-playbook update.yaml -i prod-1, ( 5ansible-playbook update_pmm.yaml -i prod-2,

Make sure to close any other sessions on the machines before running the playbook.e.g. session used for backup can cause the playbook to fail.

• 
  

update.yaml

1- name: Update your Debian or Ubuntu box in Ansible 2 hosts: all 3 tasks: 4 - name: Update all packages 5 apt: 6 update_cache: yes 7 upgrade: dist 8 become: true 9 10 - name: Stop service mysql, if running 11 service: 12 name: mysql 13 state: stopped 14 enabled: true 15 become: true 16 17 - name: Reboot box if kernel/libs updated and requested by the system 18 shell: sleep 10 && /sbin/shutdown -r now 'Rebooting box to update system libs/kernel as needed' 19 args: 20 removes: /var/run/reboot-required 21 async: 300 22 poll: 0 23 ignore_errors: true 24 25 - name: Wait for system to become reachable again 26 wait_for_connection: 27 delay: 60 28 timeout: 300 29 30 - name: Start service mysql, if not started 31 service: 32 name: mysql 33 state: started 34 enabled: true 35 become: true 36 37 38 - name: Verify new update (optional) 39 command: uname -mrs 40 register: uname_result 41 - name: Display new kernel version 42 debug: 43 var: uname_result.stdout_lines

 

 

For Centos (in the case of proxy servers only) below commands can be used:

1cd /root 2ansible -m command -a "hostname" all --limit @hostlimits.txt 3ansible -m command -a "sudo yum -r needs-restarting" all --limit @hostlimits.txt 4ansible -m command -a "sudo yum --security check-update" all --limit @hostlimits.txt 5ansible -m command -a "sudo yum --security update -y" all --limit @hostlimits.txt

Please note you can limit the hosts by using hostnames in @hostlimits.txt

 

Post reboot Validation:

• Make sure you are able to list databases on mysql server post server reboot.

Sample Output:

1Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. 2 3mysql> show databases; 4+--------------------+ 5| Database | 6+--------------------+ 7| d1 | 8| infon_schema | 9| mysql | 10| perfschema | 11| sys | 12+--------------------+ 135 rows in set (0.01 sec) 14 15mysql> exit

 

 

• We have to make sure Replication works as expected on all the databases and the below commands:

Log in to MySQL command line and run:

run SHOW SLAVE STATUS\G first to check if it is running.

1SHOW SLAVE STATUS\G

Check the output of this to see if it is working:

1START SLAVE;

it won't do any harm to run START SLAVE; if it is already running

 

As recommended by DBA, we should reboot the replicas first, make sure they come back correctly, and then do the masters.

 

On slave servers just stop the slave before reboot and make sure it comes up after reboot.

1stop slave;

1start slave;