How Microsoft helped clean up CrowdStrike’s mess

As CrowdStrike scrambled to respond to an influx of crash reports early Friday morning, engineers inside Microsoft also noticed that something was majorly wrong. Millions of Windows machines were being sent into a perpetual Blue Screen of Death, taking down important servers and PCs around the world.

Microsoft quickly declared the incident a “severity zero,” or what’s known internally as sev0, according to sources familiar with the situation. This is the highest, most urgent level of an incident impacting Microsoft products or services. A sev0 incident is rare at Microsoft, and it means people get woken up in the middle of the night and on-call engineers immediately start figuring out what happened and how to respond.

That work was complicated by the fact that a third party — CrowdStrike — was at the center of the issue. On Friday, July 19th, a 12:09AM ET update from CrowdStrike ultimately knocked 8.5 million PCs offline. The error may not have been Microsoft’s fault, but it was certainly Microsoft’s problem. The outage impacted some of what Microsoft calls its “pri0 customers,” large organizations with critical infrastructure. That meant Microsoft had to be in constant communication with CrowdStrike engineers and even cloud rivals like Amazon and Google.

Microsoft has a real-time monitoring team internally that can identify critical outages, but the company’s support lines were also lighting up as more and more machines dropped offline. Microsoft had to fix its own Azure virtual machines to assist customers that were impacted by the faulty update.

CrowdStrike was quick to publish a workaround for IT admins to get machines back up and running, but it required a lot of manual steps; they had to boot into Safe Mode and delete a faulty file, requiring physical access to the machines in most cases. Microsoft engineers were determined to make it easier for IT admins to recover machines, so the company’s Intune support team started working on a recovery tool.

I understand Microsoft’s internal incident management tool had logged around 60 people, initially contributing to the creation of an easy fix for IT admins to bring machines back to life. The recovery tool was eventually released a day later on Saturday, and Microsoft engineers worked all weekend on improving the tool and releasing revisions that included improvements to logging, error handling, and the ability to recover in multiple ways for different system types. The tool was last updated on Monday to version 3.1, demonstrating how much work went into this. It’s impressive how quickly it was created given it supports Windows clients, servers, and operating systems hosted on Hyper-V.

While some engineers dedicated time to creating this tool, Microsoft also had to mobilize a team over the weekend to respond to commercial customers, speak to the press, and continue the recovery efforts. Hundreds of Microsoft engineers were called on over the weekend to help customers restore PCs. “We’re working around the clock and providing ongoing updates and support,” said David Weston, vice president of enterprise and OS security at Microsoft, in a blog post on Saturday.

Nearly a week after the incident, Microsoft is certainly at the tail end of this disaster. Once the dust has settled, the conversations around how this happened and how to avoid it in the future will begin. There is also widespread frustration internally at Microsoft over how a CrowdStrike update was able to take down millions of Windows machines.

Some employees were upset that Microsoft took the brunt of the blame in initial media reports. Frank Shaw, Microsoft’s head of communications, complained about “vibe-based journalism” in reference to a Wall Street Journal headline reading, “Blue Screens Everywhere Are Latest Tech Woe for Microsoft.” Shaw called the headline “pathetic” in a scathing post on X. (The Verge also called it a “Windows BSOD issue.”)

Internally, Microsoft employees joked about putting the logo of the company that caused a crash on the BSOD, with employees spreading plenty of memes as the week went on and sympathizing with IT admins around the world.

CrowdStrike certainly has to answer for itself. It didn’t follow the very basics of deploying updates in stages to customers with what looks like a blind trust in validation checks that failed due to a bug. It’s a huge and costly mistake made even more embarrassing because CrowdStrike is supposed to protect Windows from the types of threats that can take down machines. CrowdStrike also regularly bashes Microsoft’s own security efforts. “Microsoft’s security products can’t even protect Microsoft. How can they protect you?” read a marketing line on CrowdStrike’s website earlier this week. CrowdStrike has now removed that line after several media reports pointed it out.

Hopefully that’s a small sign that CrowdStrike and other security vendors won’t be so quick to point fingers at others. Microsoft has been through several high-profile security incidents in recent years, and I’ve noticed the company has gradually been calling for cooperation across the cybersecurity industry over the past year instead of perpetuating a culture of blame.

I’m sure there will be calls for Microsoft to improve how it handles security vendors’ deep integration into Windows, too. After all, CrowdStrike isn’t the first security vendor to make a mistake, and it certainly won’t be the last. “As we’ve seen over the last two days, we learn, recover and move forward most effectively when we collaborate and work together,” says Weston.

Microsoft hasn’t laid off its DEI team

In last week’s Notepad issue, I mentioned that Business Insider was reporting that Microsoft was closing one of its diversity, equity, and inclusion (DEI) teams. That’s not actually the case. GeekWire reports that Microsoft has laid off two diversity and inclusion roles, not its entire diversity and inclusion team.

To back this up, I’ve obtained a memo from Microsoft’s chief diversity officer, Lindsay-Rae McIntyre, about the reports. “Last week there was media coverage that inaccurately implied that Microsoft’s corporate diversity & inclusion team was eliminated,” said McIntyre. The memo says the situation has been clarified, but it has generated concern about Microsoft’s diversity and inclusion efforts.

McIntyre says Microsoft will continue to invest in a variety of diversity and inclusion initiatives over the next financial year, including self-expression in Microsoft 365 and accessibility in Copilot, Azure AI, and Microsoft 365.

“Each of us doing our best work depends on us collectively creating the conditions for all employees at Microsoft to thrive,” says McIntyre. “The world is counting on Microsoft to apply all we know about diversity and inclusion to realize an Al-enabled future that includes everyone. The work is not done, and we remain committed to push ahead for progress together.”

Thanks for subscribing and reading to the very end. I’d like to know if you have any thoughts on Microsoft Teams. Do you love it? Do you hate it? What would you like to see change? You can reach me via email at [email protected].

If you’ve heard about any of Microsoft’s secret projects, you can also speak to me confidentially on the Signal messaging app, where I’m tomwarren.01. I’m also tomwarren on Telegram if you’d prefer to chat there.