Saltstack Windows Patching

CONFIGURING SALT

Salt configuration is very simple. The default configuration for the master will work for most installations and the only requirement for setting up a minion is to set the location of the master in the minion configuration file.
The configuration files will be installed to /etc/salt and are named after the respective components, /etc/salt/master, and /etc/salt/minion.

MASTER CONFIGURATION

By default the Salt master listens on ports 4505 and 4506 on all interfaces (0.0.0.0). To bind Salt to a specific IP, redefine the "interface" directive in the master configuration file, typically /etc/salt/master, as follows:

- #interface: 0.0.0.0
+ interface: 10.0.0.1

After updating the configuration file, restart the Salt master. See the master configuration reference for more details about other configurable options.

MINION CONFIGURATION

Although there are many Salt Minion configuration options, configuring a Salt Minion is very simple. By default a Salt Minion will try to connect to the DNS name "salt"; if the Minion is able to resolve that name correctly, no configuration is needed.
If the DNS name "salt" does not resolve to point to the correct location of the Master, redefine the "master" directive in the minion configuration file, typically /etc/salt/minion, as follows:

- #master: salt
+ master: 10.0.0.1

After updating the configuration file, restart the Salt minion. See the minion configuration reference for more details about other configurable options.

PROXY MINION CONFIGURATION

A proxy minion emulates the behaviour of a regular minion and inherits their options.
Similarly, the configuration file is /etc/salt/proxy and the proxy tries to connect to the DNS name "salt".
In addition to the regular minion options, there are several proxy-specific - see the proxy minion configuration reference.

RUNNING SALT

1. Start the master in the foreground (to daemonize the process, pass the -d flag):
   

   salt-master
   

2. Start the minion in the foreground (to daemonize the process, pass the -d flag):
   

   salt-minion
   

Having trouble?
The simplest way to troubleshoot Salt is to run the master and minion in the foreground with log level set to debug:

salt-master --log-level=debug

For information on salt's logging system please see the logging document.
Run as an unprivileged (non-root) user
To run Salt as another user, set the user parameter in the master config file.
Additionally, ownership, and permissions need to be set such that the desired user can read from and write to the following directories (and their subdirectories, where applicable):

• /etc/salt
• /var/cache/salt
• /var/log/salt
• /var/run/salt

More information about running salt as a non-privileged user can be found here.
There is also a full troubleshooting guide available.

KEY IDENTITY

Salt provides commands to validate the identity of your Salt master and Salt minions before the initial key exchange. Validating key identity helps avoid inadvertently connecting to the wrong Salt master, and helps prevent a potential MiTM attack when establishing the initial connection.

MASTER KEY FINGERPRINT

Print the master key fingerprint by running the following command on the Salt master:

salt-key -F master

Copy the master.pub fingerprint from the Local Keys section, and then set this value as the master_finger in the minion configuration file. Save the configuration file and then restart the Salt minion.

MINION KEY FINGERPRINT

Run the following command on each Salt minion to view the minion key fingerprint:

salt-call --local key.finger

Compare this value to the value that is displayed when you run the salt-key --finger <MINION_ID> command on the Salt master.

KEY MANAGEMENT

Salt uses AES encryption for all communication between the Master and the Minion. This ensures that the commands sent to the Minions cannot be tampered with, and that communication between Master and Minion is authenticated through trusted, accepted keys.
Before commands can be sent to a Minion, its key must be accepted on the Master. Run the salt-key command to list the keys known to the Salt Master:

[root@master ~]# salt-key -L
Unaccepted Keys:
alpha
bravo
charlie
delta
Accepted Keys:

This example shows that the Salt Master is aware of four Minions, but none of the keys has been accepted. To accept the keys and allow the Minions to be controlled by the Master, again use the salt-key command:

[root@master ~]# salt-key -A
[root@master ~]# salt-key -L
Unaccepted Keys:
Accepted Keys:
alpha
bravo
charlie
delta

The salt-key command allows for signing keys individually or in bulk. The example above, using -A bulk-accepts all pending keys. To accept keys individually use the lowercase of the same option, -a keyname.
See also
salt-key manpage

SENDING COMMANDS

Communication between the Master and a Minion may be verified by running the test.version command:

[root@master ~]# salt alpha test.version
alpha:
    2018.3.4

Communication between the Master and all Minions may be tested in a similar way:

[root@master ~]# salt '*' test.version
alpha:
    2018.3.4
bravo:
    2018.3.4
charlie:
    2018.3.4
delta:
    2018.3.4

Each of the Minions should send a 2018.3.4 response as shown above, or any other salt version installed



I was going through few of DEVOPS tools for automation work out of personal interest and was comparing them with respect to our inhouse environments and while comparing the Puppet Vs Chef Vs Ansible Vs SaltStack I see Salt is a very good open source tool and have the capability to manage cloud environments without buying the enterprise edition and can easily manage 1000+ servers with single master itself plus its capable of handling Docker images also.
One better thing is its marginable faster than all the configuration management tools when it comes to large deployments of 1000+ slaves connected and managed by one master. It will be really helpful for curbing out lot of man hours of manual work specially in hosted environments including the inhouse environments.
As an immediate demo in this article, we will try to show how we can eliminate pain points like:

1. Windows Security fixes (KB) patching to 100+ servers takes huge time and weekends time to login manually to each server and do the patching.
2. Monitoring Service status and ensuring services are running fine, network status, etc. eats out time from daily work hours.
3. Env’s configurations states are not in control and reverting back and tracking a single change is not possible. Due to which many issues gets raised by users.

To quickly summarize the article:

As an initial step towards salts capabilities, I have setup Saltstack(Master & Minion[slave]) doing basic windows security patching successfully. We can easily add any windows, linux, solaris boxes as minions to the same salt master for any kind of patching and configuration management tasks.
We can centralize the salt master to make all kind of patching and maintenance activities risk free and automated saving lot of hours of manual activities. Salt is well capable to manage data centers as well as cloud environment’s and amazon VM’s including our local VM environments.

Patching windows security fixes to all slaves from a single master server:

Objectives of example:

– Installation of salt master and saltstack minion (slave). I have taken my laptop as a minion or slave for testing, we can add 1k+ minions to one salt master also.
– Sample steps to change configuration in all minions automatically from one master.
– Sample steps to patch the security fixes to all the minions (Example: cloud servers)

SALT BOOTSTRAP

The Salt Bootstrap Script allows a user to install the Salt Minion or Master on a variety of system distributions and versions.
The Salt Bootstrap Script is a shell script is known as bootstrap-salt.sh. It runs through a series of checks to determine the operating system type and version. It then installs the Salt binaries using the appropriate methods.
The Salt Bootstrap Script installs the minimum number of packages required to run Salt. This means that in the event you run the bootstrap to install via package, Git will not be installed. Installing the minimum number of packages helps ensure the script stays as lightweight as possible, assuming the user will install any other required packages after the Salt binaries are present on the system.
The Salt Bootstrap Script is maintained in a separate repo from Salt, complete with its own issues, pull requests, contributing guidelines, release protocol, etc.
To learn more, please see the Salt Bootstrap repo links:

• Salt Bootstrap repo
• README: includes supported operating systems, example usage, and more.
• Contributing Guidelines
• Release Process

Note
The Salt Bootstrap script can be found in the Salt repo under the salt/cloud/deploy/bootstrap-salt.sh path. Any changes to this file will be overwritten! Bug fixes and feature additions must be submitted via the Salt Bootstrap repo. Please see the Salt Bootstrap Script's Release Process for more information.

Setup of Salt (Master):

Note: I have used a RHEL 6  server to setup the salt master.

1. Run below command to download the script to install the master.

curl -L https://bootstrap.saltstack.com -o install_salt.sh 

2. Run below command to start the install of salt master

install_salt.sh -M 

[root@saltmaster opt]# sudo sh install_salt.sh –M

*  INFO: sh install_salt.sh -- Version 2015.05.07
*  INFO: System Information:
*  INFO:   CPU:          GenuineIntel
*  INFO:   CPU Arch:     x86_64
*  INFO:   OS Name:      Linux
*  INFO:   OS Version:   2.6.18-348.el5
*  INFO:   Distribution: Red Hat Enterprise Server 5.9
*  INFO: Installing minion
*  INFO: Installing master
*  INFO: Found function install_red_hat_enterprise_server_stable_deps
*  INFO: Found function install_red_hat_enterprise_server_stable
*  INFO: Found function install_red_hat_enterprise_server_stable_post
*  INFO: Found function install_red_hat_enterprise_server_restart_daemons
*  INFO: Found function daemons_running
*  INFO: Running install_red_hat_enterprise_server_stable_deps()
*  INFO: Adding SaltStack's COPR repository

Loaded plugins: product-id, security, subscription-manager

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Setting up Install Process
Package python26-PyYAML-3.08-4.el5.x86_64 already installed and latest version
Package python26-m2crypto-0.21.1-5.el5.x86_64 already installed and latest version
Package python26-2.6.8-2.el5.x86_64 already installed and latest version
Package python26-requests-1.1.0-5.el5.noarch already installed and latest version
Package python26-crypto-2.3-6.el5.x86_64 already installed and latest version
Package python26-msgpack-0.4.5-1.el5.x86_64 already installed and latest version
Package python26-zmq-14.5.0-1.x86_64 already installed and latest version
Package python26-jinja2-2.5.5-6.el5.noarch already installed and latest version

Nothing to do


*  INFO: Running install_red_hat_enterprise_server_stable()
Loaded plugins: product-id, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Setting up Install Process

Package salt-minion-2014.7.5-2.noarch already installed and latest version

Resolving Dependencies

--> Running transaction check

---> Package salt-master.noarch 0:2014.7.5-2 set to be updated

--> Finished Dependency Resolution

 

Dependencies Resolved

================================================================================

Package           Arch         Version          Repository                Size

================================================================================

Installing:

salt-master       noarch       2014.7.5-2       saltstack-salt-el5       765 k

 

Transaction Summary

================================================================================

Install       1 Package(s)

Upgrade       0 Package(s)

 

Total download size: 765 k

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

  Installing     : salt-master                                              1/1

 

Installed:

  salt-master.noarch 0:2014.7.5-2

 

Complete!

*  INFO: Running install_red_hat_enterprise_server_stable_post()

*  INFO: Running install_red_hat_enterprise_server_restart_daemons()

Starting salt-minion daemon:                               [  OK  ]

Starting salt-master daemon:                               [  OK  ]

*  INFO: Running daemons_running()

 

-------------------------------------------------------

3. Once installation completed run below command to check for the versions installed.

4. Run below command to restart the salt master service.

service salt-master restart

5. Now download the saltstack windows minion (slave) software for the servers to manage from below url:

SaltStack Windows Installation

6. Install the saltstack windows minion in client node with default settings and update below details while installing.

                 IP address of master and FQDN of client node.

7. Make sure the saltstack windows salt-minion service is up and running on client node which you want to manage.

8. Now go back to master server and run below command to see the authentication certificate sent by all minions.

[root@saltmaster opt]# sudo salt-key

Accepted Keys:

Unaccepted Keys:

SaltMinion-WindowsLaptop1

Rejected Keys:

9. Now accept the key to authenticate the saltstack windows minion to master.

[root@saltmaster opt]# salt-key -a SaltMinion-WindowsLaptop1

The following keys are going to be accepted:

Unaccepted Keys:

SaltMinion-WindowsLaptop1

Proceed? [n/Y] Y

Key for minion SaltMinion-WindowsLaptop1 accepted.

[root@saltmaster opt]# sudo salt-key

Accepted Keys:

SaltMinion-WindowsLaptop1

Unaccepted Keys:

Rejected Keys:

10. Run below command to check the status of all saltstack windows minions added to salt master. (Here we have added only one minion – my laptop)

Note : you can add 1k+ minions to a single salt master.

salt '*' test.ping

11. We can check if the salt master is able to control the saltstack windows minions by running below command to change the minion computers descriptions.

11. a. Check the computers description before running below command to change.

11.b. Now run the below command to change the description of the saltstack windows minion/slave computer from master.

       salt '*' system.set_computer_desc 'Office Laptop Of Ramakanta'

                          11.c. Check the saltstack windows minion’s computer description to make sure the master is able to control the minion.

12. We are now sure that master is able to control the saltstack windows minion so we will go ahead with one more example of auto downloading and installing security fixes automatically.

Steps to Auto patch windows KB3011780 security update from master to all the salt minion servers.

1. Before starting the installation, check the latest KB installed in the minion to keep note of it for future verifications of if the KB got installed successfully or not.

Latest KB = KB3010788

2. Run below command from salt master to download the KB from Microsoft website in all the saltstack windows minion servers.

It will download and store the file in C:\TEMP as mentioned in below command:

salt '*' cp.get_url 'http://download.microsoft.com/download/C/C/3/CC36FA0C-974B-444A-B2C6-8E368250E37F/Windows6.1-KB3011780-x64.msu' 'C:\TEMP\Windows6.1-KB3011780-x64.msu'  -l debug

Note: Here “*” is used to let salt order all registered saltstack windows minions to download the files. It can be customized further to suit your requirement of patching few servers or servers starting with some special patterns etc.

On The saltstack windows minion C:\TEMP the file got downloaded automatically.

3. To install the downloaded KB from C:\TEMP in all saltstack windows minions run below command.

salt -t 900 '*' cmd.run 'wusa.exe C:\TEMP\Windows6.1-KB3011780-x64.msu /quiet /norestart' -l debug

Now to check whether the KB got installed successfully run below command:
Below output shows the 226 number KB got installed successfully in my laptop (minion/slave)

salt -t 200 '*' cmd.run 'systeminfo | find "KB3011780"'

You can see the security fix KB3011780 is now installed successfully on my laptop (saltstack windows minion) without any manual intervention.


So if we implement the SaltStack we can manage any configurations, patching and upgrades in any of hosted or local environment’s easily and automatically saving a lot of man hours of manual work and a single person will be able to handle many environments easily from a master.


Some Important Commands:
Extract powershell modules here:
C:\Windows\System32\WindowsPowerShell\v1.0\Modules

Set-ExecutionPolicy Unrestricted
Import-Module PSWindowsUpdate
salt-call state.highstate -l debug
telnet 172.19.8.25 4505
telnet 172.19.8.25 4506
install-windowsfeature "telnet-client"
sudo salt '*' cmd.run 'get-eventlog system | where-object {$_.eventid -eq 6006} | select -first 1' shell=powershell
get-hotfix | select-object -property PScomputername,hotfixid,installedon|findstr -i KB4512506

https://github.com/jborean93/ansible-windows/blob/master/scripts/Upgrade-PowerShell.ps1
https://docs.ansible.com/ansible/latest/user_guide/windows_setup.html