Happy Friday! First off, I want to let you know that as a Madrone Client your managed infrastructure is unaffected by the recent CrowdStrike issues. We don’t use CrowdStrike, and while we did evaluate their product for use in the last year, it didn’t meet our expectations. CrowdStrike does provide a great security product, despite their recent issues with an update that was pushed out. If you’re short on time - please know that we’re monitoring the outages this has caused, we’ll keep you updated on anything critical, and that while the coming days might be filled with annoyances as national companies experience disruptions, this will all resolve soon.
So, let's dive into what happened. Overnight, at least for us in the US, CrowdStrike pushed out an update to their software. This happens regularly, but this update set off a series of failures across the globe. CrowdStrike, like many security software solutions, integrates with Windows on a deeper level than most software. While this might seem like a vulnerability, it is necessary with the current architecture of Windows, and it allows security vendors to have deeper access to help protect against threats on a computer or server. CrowdStrike accomplishes this by loading a kernel-level driver, loading itself onto a computer at a similar level as a graphics driver, the software component of the hardware that allows you to see anything on your computer screen. This gives CrowdStrike all the access it needs to protect a computer, but also means that malfunctions in CrowdStrike can look just as bad as if a computer or server physically failed.
After this update was applied, computers began restarting as a result of this faulty driver, attempting
to recover from what essentially emulated bad hardware.
Since there was a fundamental flaw in the software CrowdStrike deployed, no amount of restarting will correct this issue, and the computer or server will continue to reboot indefinitely. This has led to many computers landing at a Blue Screen of Death or BSOD repeatedly. Fortunately, there is now a fix for this issue, but unfortunately it is a manual one. In order to repair this issue a computer must be rebooted into Safe Mode which skips loading most, if not all, drivers. After booting into Safe Mode, files matching C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys must be removed manually. Once this is done, the computer will reboot normally without the offending update and should be able to work again. CrowdStrike has removed the offending update, so provided they test updates in the future to catch similar faults, this shouldn’t happen again.
Where does this leave us all?
Well, unfortunately it leaves global IT in a bit of a mess. CrowdStrike controls nearly a quarter of the CyberSecurity EDR market, so they are deployed far and wide. Every computer or server with this faulty update needs to be fixed manually or replaced. So far, disruptions have hit financial institutions, airlines, hospitals and even Microsoft’s Cloud. Some of these organizations will recover faster than others, but we expect to see issues with those experiencing outages through at least this weekend.
The biggest take away for those of us not in the middle of this disruption is what is our plan if we were?
Do we all have a documented disaster recovery plan?
Is it on paper or only digital?
Do we know what order we need to recover our servers to get the most users working again as fast as possible?
Do we know which users to prioritize over others?
Have we planned this all out before we’re in the middle of a crisis?
If you don’t have a solid plan for all of these questions, please reach out to us today -
let's get a solid plan together.
Thank you,
Jake Campbell
Co-Founder & CEO