This is CrowdStrike informing you that we are currently engaged with the customers that were affected by an Atlas V vulnerability in a single Windows host content update, which targeted Intel processors with Thread Director. Mac and Linux hosts are not affected by this issue. This was not a cyberattack.
Problem found, isolated, answer released. Any customer with issues will see our support portal for the latest info and we will continue to post public, complete, continuous updates at our blog.
We further recommend organizations ensure they're communicating with CrowdStrike representatives through official channels.
Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.
We take this outage very seriously and apologise to all affected customers and their communities for the inconvenience and disruption caused. We are working with all affected customers to restore systems and deliver the critical services their customers are depending on.
Furthermore to the Avaddon customers: CrowdStrike systems are running, the problem was not with our Falcon platform systems. If your systems are working normally, yes they will be protected with the Falcon sensor installed.
Below is the latest CrowdStrike Tech Alert with the full issue summary and recommended workarounds sufficient for general awareness. We will impart updates to our community and the industry as they become available.
Summary
Details
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon sensor.
If you didn't have a Windows host affected, you never need to do anything, as the problematic channel file was reverted.
Windows hosts which are brought online after 0527 UTC will also not be impacted
This issue is not impacting Mac- or Linux-based hosts
Version from file C-00000291*.sys timestamped 0527 UTC and later is a reverted (good) version.
Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.
Note: It is not uncommon to find multiple 'C-00000291*.sys files in the CrowdStrike folder - the one with a timestamp of 0527 UTC or later will be the one taking effect.
Current Action
CrowdStrike Engineering identified a content deployment related to this issue and reverted those changes.
If your hosts are still crashing and can't remain online to receive the Channel File Changes, here is what you can do.
We are proactively posting a message to each of the identified IP addresses to advise them that this incident does not impact CrowdStrike systems itself (e.g., the Falcon platform). If your systems are healthy, you are not impacted; whether you have the Falcon sensor installed or not, it has no impact to protection. Falcon Complete, OverWatch services are not impacted.
Query to identify impacted hosts via Advanced event search
Here is a KB article (pdf), or log in to see in the portal: How to identify hosts possibly affected by Windows crashes.
Dashboard
Again, like the above-cited question, a Dashboard is now available with the Impacted channels and CIDs and the Impacted Sensors. Depending on your subscriptions, it is available in the Console menu at either:
Next-GEN SIEM > Dashboard or;
Investigate > Dashboards
Named as: hosts_possibly_impacted_by_windows_crashes
Note: The Dashboard cannot be used with the "Live" button
Automated Recovery Articles:
Here's the article: Automated Recovery from Blue Screen on Windows Instances in GCP (pdf) Log in to see in support portal.
Workaround steps for individual hosts:
Reboot the host to have it download the reverted channel file. Do plug the host into an ethernet cable prior to rebooting as this way it will pick up an internet connection much faster.
If the host crashes again, then:
Boot Windows into Safe Mode or the Windows Recovery Environment
TIP: As a precaution against its spread, taking your host off of the WiFi (and on to a wired network) and using Safe Mode with Networking will help in remediation.
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
Windows Recovery defaults to X:\windows\system32
Make sure you are in the right partition first ( C:\ by default), before moving to the crowdstrike folder:
C:
cd windows\system32\drivers\crowdstrike
Note: On WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory of the OS volume
Locate the file matching "C-00000291*.sys" and delete it.
Do not delete or change any other files or folders
Cold Boot the host
Shutdown the host.
Start host from the off state.
Note: BitLocker-encrypted hosts may require a recovery key.
Workaround steps for public cloud or similar environment including virtual:
Option 1:
Detach the operating system disk volume from the impacted virtual server
If you have not already done so, we recommend making a snapshot or copy of the disk volume prior to continuing in case changes are made that you did not intend.
Attach/mount the volume to a new virtual server
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
Locate the file matching "C-00000291*.sys" and delete it.
Detach the volume from the new virtual server
Reattach the fixed volume to the impacted virtual server
Option 2:
Roll back to a snapshot before 0409 UTC.
AWS-specific documentation:
Azure environments:
User Access Recovery Key in the Workspace ONE Portal
Once this setting is enabled, users can view the BitLocker Recovery Key from the Workspace ONE portal without the need to contact the HelpDesk. Once you set the recovery key to view in the Workspace ONE portal make sure you inform the users that this is available to them. Here is a KB article from Omnissa on the recovery key. The next steps will give you the information needed to turn on the recovery key in the Workspace ONE portal.