How Lowe's SRE Team is Driving Reliability Through Hyper Automation and Unified Platforms

Online retail moves fast—so keeping systems reliable and running smoothly is more important than ever to provide a frictionless user experience.

At Lowe's, our Site Reliability Engineering (SRE) team has embraced the philosophy of hyper-automation to meet these demands. By integrating innovative tools and solutions, we've streamlined our processes, reduced manual effort, and enhanced our ability to respond swiftly to incidents.

This approach significantly contributes to Lowe's Total Home Strategy, ensuring we protect and improve the customer experience at every touchpoint. If you're interested in learning exactly how we did it (and you can too), then keep reading  to explore the transformative impact of our SRE initiatives, such as leveraging automation to drive excellence in system reliability and operational performance. 

 Unifying SRE Practices Through Hyper Automation 

 #1 Digital Resiliency Hub (DRH): Central Command for SRE Excellence 

In our journey to optimize operational efficiency, we identified the need for a unified platform to streamline Site Reliability Engineering (SRE) tools and resources.

The Digital Resiliency Hub (DRH) is our answer to this need, offering a centralized portal that integrates all SRE products, reducing manual effort and cognitive load. DRH ensures that SRE, business teams, and leadership have quick access to essential tools, fostering a culture of hyper-automation and efficiency. 

Shown Above: Snapshot of Digital Resiliency Hub (DRH).

#2 Comprehensive Reliability with SRE Scorecard 

Maintaining a unified and comprehensive view of performance and reliability across all product teams is crucial for ensuring the smooth operation of our online selling channels.

The SRE Scorecard was developed to meet this need, providing an automated tool that tracks and evaluates critical metrics across 40+ domain teams, all of which contribute to our online retail platform's success. 

Shown Above: Snapshot of SRE Score.

Key Features: 

1. Real-Time Data Integration: The SRE Scorecard taps into real-time data from all products within the Digital Resiliency Hub (DRH).

By consolidating this information, the scorecard offers a holistic view of our system's health, tracking key performance indicators (KPIs) such as release quality, error budgets (covering availability and latency), site speed deviations (both week-over-week and against competitor benchmarks), major incident trends, certificate expiry management, on-call efficiency, and operations ticket resolution. 

2. Automated Governance: This tool is an automatic governance mechanism for the 40+ domain teams that stitch together our online selling channels.

The SRE Scorecard keeps a tight tab on critical areas like release quality and error budgets, ensuring that domains adhere to the highest standards of reliability. 

3. Scoring and Grading System: The scorecard assigns grades (A, B, C, D) to each domain based on their performance: 

• A (Gold Standard): Represents the highest level of compliance and performance. Continuous adherence to this standard is expected, with regular audits conducted to ensure sustained excellence. Significant deviations may trigger a detailed review. 
• B: Domains can continue releasing features but must promptly address any compliance gaps. Failure to improve consistently may lead to a temporary freeze on new feature releases until compliance is restored. 
  
• C: Domains face a temporary restriction on releasing features. A detailed analysis of compliance gaps is required, along with a clear plan to achieve at least a 'B' grade. The release freeze is lifted only upon satisfactory improvement. 
• D: Domains are prohibited from releasing features until immediate compliance gaps are rectified. A comprehensive action plan to maintain at least a 'B' grade is mandatory. Continuous non-compliance could result in an extended feature freeze. 

4. Incentivized Performance: The SRE Scorecard incentivizes domains to perform better by recognizing those who consistently achieve high grades in quarterly town halls and other forums. This recognition encourages continuous improvement and adherence to the highest standards of operational excellence. 

Shown Above: Snapshot of SRE Score.

#3 Enhancing Performance with SRE Site Speed Budget 

In the fast-paced environment of online retail, maintaining optimal site speed across various funnels is crucial for delivering a frictionless user experience. But manually tracking performance? That was slow, tedious, and often inaccurate.

To address this, SREs have developed the Site Speed Budget framework in partnership with Blue Triangle, the business-outcomes platform that quantifies the cost of friction in your digital experience so you can fix what matters most.

Shown Above: Screenshot from the Blue Triangle platform using sample data for demonstration purposes only. Not affiliated with or based on data from Lowe’s.

This data is critical in benchmarking performance against competitors and setting annual and monthly targets for Core Web Vitals (CWV). By utilizing these insights, the framework continuously monitors key metrics such as Largest Contentful Paint (LCP), Cumulative Layout Shift (CLS), and Interaction to Next Paint (INP), ensuring sustained performance improvements and an enhanced customer experience.

Key Features: 

1. Automated Performance Monitoring: The tool automates the tracking of site speed, eliminating the need for manual processes and reducing the risk of errors. 

2. Industry Standard Benchmarking: By comparing our site's performance with our competitors, we ensure that we remain at the forefront of the industry, providing a competitive edge. Competitor models have been provided by the Blue Triangle Team. 

3. Annual and Monthly Target Setting: The tool allows for setting annual performance targets, which are then broken down into more manageable monthly goals. This structure ensures continuous assessment and improvement. 

4. Week Over Week Performance Tracking: To ensure that product release cycles do not inadvertently affect site speed, the tool tracks performance week over week. This allows for the swift identification of any issues introduced by new releases, ensuring they are promptly addressed. 

5. Swift Identification and Resolution of Issues: With the ability to monitor performance continuously, the tool enables rapid identification of any issues that arise, allowing for quick corrective action to maintain optimal site speed. 

#4 SRE Incident Management Framework 

Online retail demands impeccable site reliability, but our previous incident management system was fragmented, leading to delayed responses and poor communication.

The SRE Incident Management Framework addresses these issues by centralizing incident reporting and resolution. This platform provides a comprehensive view of incidents, enabling quicker, more informed decision-making and continuous improvement, enhancing our ability to maintain site reliability and customer trust. 

Key Features: 

1. Theater Tile View: Visualize the entire fiscal year's health with a theater tile view highlighting the days with incidents versus those without. This overview helps quickly assess overall site stability. 

2. Site Availability Metrics: Track site availability year-to-date (FYTD), ensuring we maintain the highest levels of uptime. 

3. MTTD & MTTRs: Monitor Mean Time to Detect, Mean Time to Recover, and Mean Time to Resolve, allowing for a detailed analysis of our response efficiency. 

4. Incident Trend Analysis: Evaluate incident trends by severity, enabling focused efforts on areas needing the most attention. 

5. Root Cause & Product Impact: Access detailed insights on the root cause themes, responsible products, and impacted products, helping drive continuous improvement. 

6. Real-Time Incident Updates: Receive real-time updates with snapshots of the site experience during incidents, providing a clear understanding of the impact on the customer journey. 

7. Embedded Postmortem Links: Easily access postmortems through embedded links, streamlining the process of reviewing and learning from past incidents. 

This comprehensive incident management approach not only ensures a more reliable site but also fosters a culture of continuous improvement, ultimately enhancing customer trust and satisfaction. 

#5 Proactive Stability with SRE Cert Watch 

In the complex landscape of online retail, ensuring the continuous validity of certificates across various systems is vital to maintaining site stability and security.

Previously, managing certificate expiries was a significant challenge due to fragmented reporting. SRE Cert Watch addresses these challenges by consolidating expiry data from various sources, providing comprehensive visibility, and enabling proactive management. 

Key Features: 

1. Comprehensive Certificate Mapping: SRE Cert Watch meticulously maps certificates related to External/Internal/Third-party DNS, Applications, and Databases. This mapping is not generic but is intricately tied to each domain in the purchase path of the site. This level of detail ensures that any potential certificate expiry can be pinpointed to its exact impact on the purchase path flow, allowing for targeted and efficient resolution. 

2. Proactive Alerting System: The tool includes an advanced alerting capability that automatically triggers alerts to the SRE team and other concerned teams when a certificate's expiry date is approaching—specifically within 30 days. This proactive approach ensures that teams have ample time to address any potential issues before they can impact the site's functionality or security. 

3. Consolidated Reporting: By bringing together data from various sources, SRE Cert Watch eliminates the fragmentation that previously hindered effective certificate management. The tool provides a unified view of all certificates across the site, simplifying the tracking process and ensuring no certificates are overlooked. 

4. Impact Analysis: With its detailed mapping to the purchase path, SRE Cert Watch allows teams to quickly assess the potential impact of a certificate expiry on critical site functions. This capability is crucial for maintaining a seamless and secure user experience, particularly during high-traffic periods. 

Driving the Future of Digital Retail Through Innovation and Automation

At Lowe's, our commitment to site reliability and operational excellence is driven by a culture of continuous innovation and automation. Through our hyper-automation initiatives, we've established a resilient and efficient SRE framework that enhances system reliability, improves incident response, and optimizes performance.

'From the Digital Resiliency Hub to the SRE Scorecard and Incident Management Framework, every tool and strategy we've implemented ensures a seamless customer experience and aligns with our Total Home Strategy.

A key aspect of our performance optimization efforts is the Site Speed Budget framework, developed in collaboration with Blue Triangle. By leveraging Blue Triangle's competitor benchmarking model, we can continuously track, analyze, and enhance site speed, ensuring our online retail platforms remain agile and responsive. This partnership provides us with data-driven insights that guide our performance goals, allowing us to stay ahead of the competition while delivering a frictionless shopping experience.

As we continue refining our SRE practices, our focus remains on driving proactive solutions that empower both our internal teams and our customers. Through strategic partnerships, advanced automation, and a relentless pursuit of operational excellence, we are shaping the future of digital retail—one innovation at a time.

Blog Authors:

• Raghuprasanth Ravichandran, Senior Engineering Manager, Site Reliability Engineering—Lowe’s Companies, INC
• Joe Praveena A, Senior Engineering Manager, Site Reliability Engineering—Lowe’s India
• Pavan Palagiri, Lead Software Engineer, Site Reliability Engineering—Lowe’s Companies, INC

Leadership Credit:

• Deepankar Singh, Senior Director, Site Reliability Engineering—Lowe’s India
• Shyam Palani , Director, Site Reliability Engineering—Lowe’s Companies, INC