Uncertainty about compromise can feel expensive, especially in the SMB space: do I dare ignore what I see in the logs if my AV is giving me a clean bill of health? Can I really afford to rebuild this server when things seem mostly ok?
What I’ve been helping many partners do is a surgical restore, which minimizes the impact on the business.
The ideal candidate environment for this is, happily, a common one.
Single Exchange server
Virtual machine
C: volume has the OS and the Exchange application
Databases and logs are on other volumes.
If you have a backup from the last week of February, you could just restore the C: volume, leaving the database volumes alone. Mail data isn’t touched. Then you patch it, and you’ve done a lot to reduce uncertainty.
1. On your firewall, block outbound access from your Exchange server on anything except port 25.
2. Don’t take your Exchange server offline yet. Start restoring a copy of your OS volume from 2/25-2/27 or thereabouts and put it in the same volume as your current C: VMDK.
3. While that is happening, download/copy the .iso of CU version that you need to have and put it on one of your data (non-OS) drives.
4. When you’ve finished with the restore, shut your VM down, and in the settings, swap the C: drive you restored out with the previous one, so you’ll be booting from your restore.
5. Block inbound access to your server on 443. It will be vulnerable after the restore, till you’ve patched it.
6. When the server boots back up, it shouldn’t have any trust issues with the domain, but if it does, resolve it by logging on as the local admin and using your contextualized version of this command:
netdom.exe resetpwd /s:mydomaincontroller /ud:domain\user /pd:password7. Give the server a few minutes to let services start up, and download any queued mail, and the run the Cumulative Update that you had copied to the data drive. Then run the Hafnium patch.
If the patch fails (which I’ve seen it do sometimes) it is usually due to an Exchange service that failed to stop when asked. Check Services–if you see a service in the Stopping state, kill it with Task Manager and then rerun the patch.
8. Now you can open up the 443 port on the firewall and allow the server to access the internet again.
—
Here are some other notes that need to go with this.
It’s known that the Hafnium exploits include a common export of the LSASS hashes, and that means that some passwords could be compromised. So change the local machine password, and the password of any account you use to log on to the Exchange server, at a minimum.
Doing this process should not end up triggering a “Code Green” in your mind. You are still at Code Yellow, at a minimum. It’s possible that the hack got further than your Exchange server. The longer you waited before patching, the more likely that is.
The Microsoft patch release was 9 days ago, and I’m under the impression that there is a window between the initial exploit explosion and the availability of time and attention for the hackers to apply human ingenuity against the particulars of your network. Ideally you’ll use that window to shore up your defenses, and this should go quite a ways towards that. But that window may be closed already.
Lastly, I’d recommend putting a next-gen antivirus (SentinelOne, CrowdStrike, Cylance, CarbonBlack, etc.) on your Exchange server. In the past, I tended to put AV everywhere except there, since I’d seen more catastrophic problems caused by AV gone amok on Exchange than I had seen problems prevented, but I’ve changed my mind about that. Since we know that it’s much more in the realm of possibility that there may be an undiscovered foothold in our networks, I’d really like to have some behavior-focused intrusion and detection happening within the network, and not primarily definition-file based security.
Elsewhere in this blog you’ll find a summary of the whole Exchange situation from patching, to clean-up to restore. Exchange server vulnerability summary – Ultimate Support for IT Pros – ThirdTier
All we do is support IT professionals. Help for IT Pros, Super Secret News, Security community, MSP Legislation community, Kits, papers, MSP training and more. https://www.thirdtier.net
One thought on “Hafnium Mitigation: Surgical Restore”