Kubernetes - Low disk space alerts

 

we need to find the pod to exec in into, do this by running :

 kubectl get pods -n <namespacename>

kubectl exec -it -n vault vault-0 -- /bin/sh'

we exec into the pod to discover what is using the disk. Do this by running

 Run ‘df –Th’ & du –shc /*' to see the disk space being used, from this cd to the directory which is using the disk space. Using the above, we can see this is /vault/audit

At this point depending on where the disk usage is, you can follow the determined process below

vault - vault_audit.log

The vault_audit.log file which is as an example taking more space can be moved from the pod to your local machine then uploaded to S3.

1. This assumes you have followed the above and are in the /vault/audit folder on vault-0.

2. Rename the file, by running the following command

   1. 'mv vault_audit.log vault_audit_mm-DD-yyyy.log'

   2. e.g. 'mv vault_audit.log vault_audit_10-21-2021.log'

3. Transfer the file from the pod to your local machine by running the follow command in a new Terminal window

   1. 'k cp vault/vault-0:/vault/audit/vault_audit_mm-DD-yyyy.log your local folder here'
      e.g. ‘k cp vault/vault-0:/vault/audit/vault_audit_10-21-2021.log /home/joe/vault_audit_10-21-2021.log'
      'metricgaming-eu-west-2-prod-logs > topics/ > vault-audit-log/' and upload the file.
      Filesystem Type Size Used Available Use% Mounted on
/dev/vdb ext4 9.8G 9.0G 312.4M 97% /vault/audit
du -shc /vault/*
20.0K /vault/audit
12 /bin/vault /vault/audit/vault_audit.log.old (deleted)

4. This will start the transfer, to see it in realtime, open up another Terminal tab and run the commands

   1. 'cd your.local.path.here' i.e. 'cd /home/joe'

   2. ‘watch ls -lah'

5. Once the file has transferred, log in to AWS, go to S3 and then to
   

6. Once the upload is complete, the file can be deleted on the pod by running the following commands

   1. ‘rm vault_audit_mm-DD-yyyy.log'

   2. e.g. ‘rm vault_audit_10-21-2021.log'

   3. In case du & df usage inside the container shows the different output then you need to kill the process holding it.

   4. Verify if different disk space usage is reported by du and df commands:

      df -Th
      

   5. Find out deleted files:

      ~# lsof 2>/dev/null | grep deleted
      

   6. Kill processes holding the files:

      lsof | grep "deleted" | awk '{print $2}' | xargs kill -9

      kill -9 12

   7. Verify disk space usage with the df -Th command /dev/vdb ext4 9.8G 36.0M 9.2G 0% /vault/audit

7. Ensure the new log file is created by running the following command, whilst still in the /vault/audit folder

   1. ‘watch ls -lah'