Operations grimoire/Incidents/2023-02-03-ESXi
From Nasqueron Agora
Not tracked on DevCentral as docker-001 is currently down.
Incident timeline
- 2023-02-03
- 16:28:03 Dorian and Dereckson worked on pad.nasqueron.org, connection was stopped
- 16:29:06 Hypervisor Dreadnought is confirmed working, attack against VMWare ESXi detected, server switched in rescue mode
- 00:15:40 hyper-001 provisioned
- 2023-02-04
- 12:58:36 migration from Dreadnought to hyper-001 done for router-001
- 16:05:01 migration from Dreadnought to hyper-001 done for complector
Analysis
External sources:
- https://www.cert.ssi.gouv.fr/alerte/CERTFR-2023-ALE-015/
- https://www.bleepingcomputer.com/news/security/massive-esxiargs-ransomware-attack-targets-vmware-esxi-servers-worldwide/
First findings:
- FreeBSD machines ZFS disks are intact (db-A-001, complector, router-001)
- Linux machines LVM/xfs disks show I/O error (Equatower, docker-001)
- https://enes.dev/ states the ...-flat.vmdk shouldn't be encrypted
- Backup of the .vmx configuration file is available at .vmx~ when a machine was running during the attack
Actionables
- Monitor VMWare ESXi last available patch vs our current version
- Improve backup strategy
- ZFS storage server
- Glacier backup
- Physical backup on site
- [DONE] Install NetBox to document network as we provision the hypervisor it
- Plan HA for critical services, at least for DevCentral