CrowdStrike BSoD, what happened and how can I be prepared?

CrowdStrike Falcon

CrowdStrike is a global security leader and the manufacturer of one of the world's best-selling malware protection platforms. Its CrowdStrike Falcon platform is specifically designed to stop security breaches through a unified set of technologies delivered in the cloud known as CrowdStrike Security Cloud service. This platform includes among other modules a next-generation antivirus (NGAV) that features machine learning technology to detect and prevent cyber-attacks. CrowdStrike contains several product modules covering multiple aspects such as threat intelligence, detection, automatic protection and remediation, etc., but for convenience it is deployed on Windows systems via a single agent, known as CrowdStrike Falcon Sensor.

On 19 July 2024 at 04:09 UTC, as part of routine operations, CrowdStrike released a configuration update to CrowdStrike Falcon Sensor for Windows systems. Sensor configuration updates are a part of the Falcon platform's protection mechanisms. This configuration update triggered a logic error that resulted in a system crash and a blue screen (BSoD) on affected Windows systems. 

A preliminary incident report may be found here: Falcon Content Update Preliminary Post Incident Report | CrowdStrike

A detailed discussion can be found here: Windows Security best practices for integrating and managing security tools

BSoD

The term ‘Blue Screen of Death’ or BSoD is a colloquialism that refers to the blue screen that results from a catastrophic failure of the Windows operating system. Such errors can be caused by a variety of hardware and software issues. The blue screen appears when the Windows system needs help in order to recover from an error that it has been unable to recover from on its own. When we see this blue screen, the Windows system is not running, and therefore all the usual means of remote recovery based on Windows applications are useless. Although Microsoft provides instructions on how a user can recover their computer to a state before the failure has occurred, usually in a corporate environment this is the responsibility of the systems department, who must physically access the computer to boot it into safe mode and proceed to diagnose the failure and repair the Windows system. Of course, this is a reasonable procedure only when a single computer or only a few computers have failed simultaneously.

Disaster Recovery

Disaster Recovery refers to the process of recovering from a massive contingency (a disaster), and usually includes protocols for recovering the functionality and data of the affected systems in the shortest possible time. The contingencies contemplated in recovery plans usually include natural elements such as fires or earthquakes, accidental ones such as the loss of power supply, or provoked ones such as cyber-attacks. 

The consequences of the disruption of a company's critical systems may vary depending on a number of factors, but always include financial losses resulting from the total or partial inactivity of the company for a certain period of time and/or the impossibility of recovering business-critical data.

In the case of the situation caused by the CrowdStrike incident, most recovery protocols failed to adequately manage the recovery of thousands of affected Windows systems, because the simultaneous occurrence of these BSoDs is not a common occurrence. Depending on each company's IT architecture, the level of business disruption may have ranged from an inconvenience to a catastrophic event.

The tool that most affected corporations lacked is a platform for remote, massive recovery of Windows computers in BSoD state, such as GMV's resQit solution. This solution is often used as a tool to improve the efficiency of recovering crashed Windows systems by enabling a remote recovery mechanism, which is so important in today's distributed and teleworking environments. However, in the event of a massive incident such as the one that occurred in the CrowdStrike case, this platform can save hundreds of thousands or even millions of euros by enabling a simple, remote recovery scenario for all affected systems in a minimal amount of time, a scenario that would be unfeasible without such a tool.