Secure your VMware ESXi hosts against ransomware in three simple steps!

Using TPM 2.0, Secure Boot and execInstalledOnly

  • We have recently seen an increase in ransomware targeting VMware vSphere ESXi hosts and encrypting all virtual machines at once, quickly and effectively. An example is RansomEXX a.k.a. Defray777. More info can be found in this Crowdstrike writeup.
  • This attack vector is possible because once attackers get control of an ESXi host, they can upload and execute any custom binaries they want.
  • We can fairly easily prevent this by combining the use of TPM 2.0, UEFI Secure Boot for the ESXi hosts and the relatively unknown ESXi setting VMkernel.Boot.execInstalledOnly, which is described in the ‘Three steps to protect ESXi against ransomware’ section below. (Jump straight there)

Background

We have recently seen an increase in ransomware attacks where the encryption is executed from the virtualization platform (ESXi or Hyper-V hosts) rather than from inside each guest operating systems (Windows, Linux etc). The benefit of this method from the attackers’ side is that they can encrypt numerous systems without having to reach them all over the network and obtain administrative privileges. This can greatly increase the scope and speed of the attack, which is bad news for us.

This blog post won’t go into the technical details on how the attacker gets into the ESXi hosts to execute the actual ransomware. This could for example be done through an RCE vulnerability such as the one for SLP in ESXi or through Active Directory->vCenter Server->ESXi, but also in other ways. A future blog post will analyze this in more detail and provide more suggested protections.

Screenshot of ESXi virtual machine files encrypted by RansomEXX/Defray777
Screenshot of ESXi virtual machine files encrypted by RansomEXX/Defray777

The ransomware will encrypt all virtual machines’ vmdk files on all attached datastores. It will also encrypt the ESXi host itself including all log files, so unless you have central tamper-proof logging in place it will be very difficult to secure forensic evidence regarding how the attack was carried out.

Despite the encryption, the ESXi hosts will remain running since they have already loaded the system files into memory. However, they will usually not survive a reboot, and will need a complete reinstallation. As long as the host is still running, the ransomware monitors the virtual machines and will encrypt any new vmdk or other virtual machine files that are put on shared datastores that it can reach.

Screenshot of reverse engineering of the RansomEXX/Defray777 ESXi ransomware, displaying the strings found in it.
Reverse engineering of the RansomEXX/Defray777 ESXi ransomware, displaying the strings found.

VMware has a good technical post about this ransomware at Deconstructing Defray777 Ransomware, which goes through the technical details, but doesn’t mention specifically how to protect the ESXi hosts.

They also have some good videos covering the basics of vSphere ESXi security here:

When we at Truesec perform security health checks of customer vSphere environments, we always give everyone the following fundamental recommendations, so do make sure you have these in place:

 

  • Keep your systems (vCenter Server, ESXi hosts, VMware Tools etc.) up to date when there are security patches released.
  • Use unique, strong passwords for administrative accounts and handle them securely.
  • Segment your networks so that vCenter Server and ESXi administrative interfaces are not reachable for non-administrative computers and users. Also use dedicated workstations for administrators.
  • Configure central logging so that you have tamper-proof logs of all administrative actions and changes in your environment.
  • Make frequent backups, and make sure they can not be deleted even if the attacker gets complete control over the rest of your environment.

 

Three steps to protect ESXi against ransomware

To prevent the actual ransomware execution, we recommend our customers to take the following steps for all ESXi hosts:

  • Use TPM 2.0 chips
  • Enable UEFI Secure Boot on the physical servers
  • Prohibit execution of custom code inside ESXi (VMkernel.Boot.execInstalledOnly)

Note: The ‘execInstalledOnly’ setting described below is not documented by VMware for vSphere versions 6.5 and 6.7, although it exists and works fine in our lab environment. From vSphere 7.0 the setting is recommended in the vSphere Security Configuration Guide. We recommend all customers to upgrade to vSphere 7.0 before implementing these recommendations.

In vSphere 7.0 U2 and newer versions, there are additional protections added which utilize encryption and TPM 2.0 to prevent tampering with these settings and the general ESXi configuration.

As always, make sure you test and evaluate the consequence of any upgrades and changes on a non-critical part of your environment before rolling it out in production. Also make sure your third-party vendors support the new version.

Step 1: TPM 2.0

TPM 2.0 is a hardware chip that most modern physical servers have. It allows the operating system (ESXi in our case) to store secrets, keys, measurements etc. in a secure manner. This is used by vCenter Server to make sure the ESXi hosts’ boot files haven’t been tampered with, and works with vSphere/ESXi 6.7 and newer. More info here: https://blogs.vmware.com/vsphere/2018/04/vsphere-6-7-esxi-tpm-2-0.html

There isn’t any specific configuration that needs to be made. If your ESXi hosts have active TPM 2.0 chips, vCenter Server will automatically display their attestation status in the ‘Monitor->Security’ tab of the clusters:

vCenter Server attestation status of ESXi hosts using TPM 2.0
vCenter Server attestation status of ESXi hosts using TPM 2.0

In vSphere 7.0 U2 and newer, the TPM 2.0 chip is also used to encrypt the configuration of the ESXi host. This is described in detail in the vSphere documentation.

Step 2: Secure Boot

Secure Boot is a UEFI BIOS feature that strengthens the security of the operating system (ESXi in this case) by making sure that all code that is loaded at boot is digitally signed and has not been tampered with. Unlike some other operating systems, ESXi can have Secure Boot enabled retroactively without having to perform a complete reinstallation.

However, there might be some third-party packages (‘VIBs’, in vSphere language) that have the wrong ‘Acceptance level’ and can prevent Secure Boot from working correctly. To check if your ESXi host already has Secure Boot enabled, and whether there are any obstacles to enabling it, run the following two commands from an ESXi command line (SSH or ESXi Shell):

/usr/lib/vmware/secureboot/bin/secureBoot.py -s 
/usr/lib/vmware/secureboot/bin/secureBoot.py -c

ESXi secureboot.py check showing VIB signatures verified and acceptance levels validated

As we can see in the output above, Secure Boot is currently Disabled, but there are no obstacles preventing us from enabling it. We can now put this ESXi host in Maintenance Mode, reboot it, enter the server’s BIOS setup, enable Secure Boot, and boot up the ESXi host again. If you’re unsure on how to enable Secure Boot, check with your hardware server vendor on how to do this.

In vSphere 7.0 U2, the Secure Boot setting can be protected from tampering using the configuration protection capability. This is set using the following command line:

esxcli system settings encryption set --require-secure-boot=TRUE

More information is available in this VMware Documentation page.

Step 3: execInstalledOnly

This setting prohibits execution of custom code inside ESXi and will make the ESXi host simply refuse to execute anything that was not installed through a signed VIB package from a certified partner.

This is very easy to achieve in ESXi compared to a general purpose operating system like Windows or Linux. ESXi is by design an “appliance” which doesn’t require any custom code to be run on it apart from VMware’s own code and the drivers and utilities of certified partners. Combining execInstalledOnly with TPM and Secure Boot which tamper-proofs the existing VIBs gives us an excellent combination of protective measures against ransomware and other malware executing inside the ESXi hosts.

The setting is found in ESXi under Manage->Advanced Settings at VMkernel.Boot.execInstalledOnly and it can be set without having to open a CLI to each ESXi host. We can set it for individual ESXi hosts using the vSphere Web Client or for multiple ESXi hosts at a time using PowerCLI (VMware’s PowerShell modules). The examples below use the CLI over SSH, since this gives us some additional information that is good to have when explaining the inner workings of the setting.

The default setting of an ESXi host is that execInstalledOnly is set to FALSE. We can verify this by simply listing the setting using

esxcli system settings kernel list -o execinstalledonly

and then checking that we are allowed to execute a custom binary. Rather than executing a real ransomware, we settled for a test binary that displays a ‘Hello world’ message to indicate that it was allowed to run:

Screenshot of ESXi esxcli system settings kernel list -o execInstalledOnly showing false+false

Now, we set the execInstalledOnly setting to TRUE using

esxcli system settings kernel set -s execinstalledonly -v TRUE

Note that the ‘Configured’ value has changed to TRUE, but that the ‘Runtime’ value is still at FALSE. This is because the system requires a reboot for the setting to start working. We can verify that the protection is still not in effect by successfully executing our test binary again.

Screenshot of ESXi esxcli system settings kernel list -o execInstalledOnly showing true+false

After rebooting the system, we can now see that the setting’s ‘Runtime’ value is TRUE, and we can also see that we are no longer allowed to execute our custom test binary.

Screenshot of ESXi esxcli system settings kernel list -o execInstalledOnly showing true+true

In vSphere 7.0 U2, the execInstalledOnly setting can be protected from tampering using the configuration protection capability. This is set using the following command line:

esxcli system settings encryption set –require-exec-installed-only=TRUE

Putting it all together

Putting TPM, Secure Boot and execInstalledOnly together, we can now establish the following order of checks and changes to each ESXi host:

  1. /usr/lib/vmware/secureboot/bin/secureBoot.py -s
  2. /usr/lib/vmware/secureboot/bin/secureBoot.py -c
  3. Make sure there are no warnings or errors above
  4. esxcli system settings kernel set -s execinstalledonly -v TRUE
  5. Put the ESXi host in Maintenance Mode and reboot it
  6. At boot-up, enter the BIOS setup and enable UEFI Secure Boot (if not already enabled according to step 1)
  7. (If running vSphere 7.0 U2 or newer):
    1. esxcli system settings encryption set --require-secure-boot=TRUE
    2. esxcli system settings encryption set --require-exec-installed-only=TRUE
    3. /sbin/auto-backup.sh
  8. Check that everything works, and take the ESXi host out of Maintenance Mode

Again, make sure you properly test these changes on non-critical ESXi hosts before rolling them out to the entire environment.

Once set, monitor any changes to these settings using central logging and set alerts so that you get notified if anyone tries to change them. Also monitor events around VMs getting unexpectedly shut down, since a ransomware needs to perform this step before being able to encrypt the files, since they’re locked by the hypervisor while running.

If you’re using the vSphere 7.0 U2 additional protections, your ESXi hosts will fail to boot and display a purple screen (which is good) if TPM, Secure Boot or execInstalledOnly is tampered with. This is described in the first example in following VMware KB article: Boot time failures due to ESXi configuration encryption (81446)

Screenshot of ESXi host with purple screen displaying 'Unable to restore system configuration. A security violation was detected'
ESXi purple screen: Unable to restore system configuration. A security violation was detected. https://via.vmw.com/security-violation

Fix the problem by going back and re-checking all the above steps:

  • If TPM 2.0 has been disabled, re-enable it.
  • If UEFI secure boot has been disabled, enable it back.
  • If execInstalledOnly boot option is set to FALSE, change it back to its initial value (i.e. TRUE).

If you have any questions or bump into any problems, don’t hesitate to contact us and we will try to help you out. Watch this space for more blog posts and videos about ransomware and vSphere/ESXi security.


By Anders Olsson

Senior Infrastructure Architect