Operations grimoire/Incidents/2023-02-03-ESXi: Difference between revisions
From Nasqueron Agora
(Created page with "Not tracked on DevCentral as docker-001 is currently down. == Incident timeline == ; 2023-02-03 * 16:28:03 Dorian and Dereckson worked on pad.nasqueron.org, connection was stopped * 16:29:06 Hypervisor Dreadnought is confirmed working, attack against VMWare ESXi detected, server switched in rescue mode * 00:15:40 hyper-001 provisioned ; 2023-02-04 * 12:58:36 migration from Dreadnought to hyper-001 done for router-001 == Analysis == External sources: * https://www....") |
(+complector) |
||
| Line 8: | Line 8: | ||
; 2023-02-04 | ; 2023-02-04 | ||
* 12:58:36 migration from Dreadnought to hyper-001 done for router-001 | * 12:58:36 migration from Dreadnought to hyper-001 done for router-001 | ||
* 16:05:01 migration from Dreadnought to hyper-001 done for complector | |||
== Analysis == | == Analysis == | ||
Latest revision as of 16:06, 4 February 2023
Not tracked on DevCentral as docker-001 is currently down.
Incident timeline
- 2023-02-03
- 16:28:03 Dorian and Dereckson worked on pad.nasqueron.org, connection was stopped
- 16:29:06 Hypervisor Dreadnought is confirmed working, attack against VMWare ESXi detected, server switched in rescue mode
- 00:15:40 hyper-001 provisioned
- 2023-02-04
- 12:58:36 migration from Dreadnought to hyper-001 done for router-001
- 16:05:01 migration from Dreadnought to hyper-001 done for complector
Analysis
External sources:
- https://www.cert.ssi.gouv.fr/alerte/CERTFR-2023-ALE-015/
- https://www.bleepingcomputer.com/news/security/massive-esxiargs-ransomware-attack-targets-vmware-esxi-servers-worldwide/
First findings:
- FreeBSD machines ZFS disks are intact (db-A-001, complector, router-001)
- Linux machines LVM/xfs disks show I/O error (Equatower, docker-001)
- https://enes.dev/ states the ...-flat.vmdk shouldn't be encrypted
- Backup of the .vmx configuration file is available at .vmx~ when a machine was running during the attack
Actionables
- Monitor VMWare ESXi last available patch vs our current version
- Improve backup strategy
- ZFS storage server
- Glacier backup
- Physical backup on site
- [DONE] Install NetBox to document network as we provision the hypervisor it
- Plan HA for critical services, at least for DevCentral
