VCSA disks become full over time

I’ve recently spoken with a number of VMware vCenter Server Appliance 6 (VCSA) users that have had issues with the root filesystem of VCSA running out of space.

This situation seems to be occurring more often now due to a combination of when the VCSA 6 went mainstream (18 to 24 months ago) and the default 365 day password expiration. The combination is just long enough for the root password to expire and after about 6 months (depending entirely on the size and activity of the vSphere environment) the /dev/sda3 disk fills!

What actually happens is: the root password of the VCSA expires, cron jobs do not run, and log files continue to grow that would normally be compressed, rotated and flushed. Eventually, the /var/log/audit.log file grows (sometimes to many Gigabytes) and the entire /dev/sda3 disk fills to 100% use. This is a critical situation which will eventually crash the VCSA!

Dealing with the expired root password on VCSA

You will need to connect to your VCSA with SSH. The first thing you might see is a password change request. This is a sure sign that the root password has expired and you need to fix it. Even if there is no change request, the root password may still be expired!

Because the VCSA 6 remembers 5 passwords by default, you will need to pick a different password from the one you had previously specified.

Recently the NIST has published new Digital Identity Guidelines debunking both password aging and any prohibition on re-use of old passwords, especially for service accounts. If you do not want to permanently change the VCSA root password, and know the original password meets all applicable standards and not compromised in any way, simply set 5 dummy passwords (MyP@$$w0rd1, MyP@$$w0rd2, MyP@$$w0rd3, MyP@$$w0rd4, MyP@$$w0rd5), followed by your original secure password.

You can change the root password with the command: passwd root

Now, access your VCSA at its VAMI interface (https://vcsa.yourdomain.tld:5480) and set your desired “Root password expires setting”

Checking the VCSA disk utilization

Once you are successfully logged-in, and the root password updated, check your disk utilization by running: df -h

Any utilization over 60% should be considered a warning sign, and a likely indicator that cron jobs have not been running.

Here we see that /dev/sda3 (mounted on /) is 78% utilized

It is almost a certainty that the space consumption is in: /var/log, so change to that directory  (cd /var/log) and  run: du –h –d 1 This command will show disk utilization by folder, limiting the depth to 1 folder.

Now we see that ./audit is using 2.6G of the 3.1G total consumed by ./log

Now, we change directory to ./audit (cd audit) and run: ls -la

What we see is that the file audit.log is 2.6G in size and the cron job that compresses and rotates it to a *.bz2 has not run for months.

If the root filesystem of your VCSA is completely full, you will need to reclaim space or grow the root partition (/dev/sda3) before cron can run successfully.

If you were able to update the root password on your VCSA before /dev/sda3 ran completely out of space, then cron will run and the problem will take care of itself. The cron job runs every 15 minutes. Allowing time for the compression to take place, your /var/log/audit will look like the above screenshot in about 30 minutes.

Just to be sure, run: df -h

And the overall filesystem should show no disks or partitions with more than 60% utilization.