On July 19, 2024, the tech world experienced an unprecedented disruption when CrowdStrike’s faulty Falcon Sensor security software update triggered the largest IT outage in history. Affecting approximately 8.5 million systems worldwide, this incident challenged MSPs like Vital Services to demonstrate resilience and innovative problem-solving.

1. Third-Party Dependencies: Despite not using CrowdStrike directly, we faced significant indirect impacts. Clients’ critical third-party services and applications were offline, creating a complex operational landscape.
2. Platform Disruptions: Acrisure, a critical platform for many clients, went down simultaneously. This prevented essential business operations and decision-making. Our ticketing system captured critical issues:

• Persistent access failures
• Widespread system interruptions
• Inability to use essential platforms like IMS and NetRate

3. Client Anxieties: Businesses were gravely concerned about prolonged downtime, potential data loss, and significant financial implications, especially during critical business periods. 

Strategic Response to the CrowdStrike Outage

Our pre-established disaster recovery plans and consistent client data backups became our primary defense mechanism. These strategic investments proved invaluable during the crisis.

Upon identifying the issue, our team executed a comprehensive response:

• Deployed alternative security solutions

• Rerouted network traffic to bypass affected systems

• Utilized manual reporting methods

• Generated client reports through alternative data pulls to ensure continuous operational visibility

   

Communication Approach

Transparency and immediacy defined our communication strategy. Our advanced ticketing system enabled:

• Continuous client updates

• Rapid issue tracking

• Constant dialogue and reassurance

Remarkable Outcomes

Performance Metrics 

• Downtime Mitigation: Manually backing up and inputting data when Power BI reports failed to refresh, including:

  • Continuous troubleshooting of gateway issues

  • Extensive testing and error code investigation

  • Proactive outreach to team members for support

  • Preparing for potential system reset

     

• Average Ticket Closure: 2.01 hours per CrowdStrike-related issue

Financial Impact

Our intervention helped clients avoid an estimated $500,000 in potential revenue losses.

Client Satisfaction

Clients appreciated our innovative solutions and commitment to maintaining business continuity.

Forward-Looking Strategy 

The 2024 CrowdStrike outage reinforced our commitment to:

• Continuous improvement of disaster recovery protocols

• Ongoing investment in advanced backup and security technologies

• Maintaining agile, client-centric service delivery

   

While the global IT outage presented extraordinary challenges, Vital Services demonstrated our core strength: transforming potential technological disruptions into opportunities for client success.