Anupam Krishnamurthy

Elevate quality. Eliminate waste. Accelerate delivery.

Regression Test Pipeline Optimization

Global pumps and valves manufacturer
Nov 2021 – Mar 2022

Challenge

The nightly regression test pipeline on the client project (comprised of API and E2E tests) had a runtime of 3+ hours. Frequent instability and flaky tests meant that developers spent more than 2 hours per day on debugging and maintenance.

This pipeline became a significant bottleneck during release cycles and negatively impacted team morale.

Solution

  • Test Suite Optimization
    • Replaced numerous E2E tests with faster unit and integration tests.
    • Eliminated duplicate E2E tests.
  • Parallelization
    • Segregated tests into parallel-safe and sequential-only groups.
    • Minimized sequential tests to maximize concurrency.
  • Tooling Enhancement
    • Used Aerokube Moon to execute 18 browser tests in parallel on Kubernetes.
  • Team Enablement
    • Trained the development team to diagnose and resolve pipeline issues.
    • Enabled shared ownership of test quality.

Results

  • Pipeline Runtime: Reduced from 3+ hours to 10 minutes
  • Debugging Time: Dropped from 2+ hours/day to under 30 minutes
  • Team Impact: Improved team satisfaction and confidence in the regression pipeline

Contract Testing

Global pumps and valves manufacturer
Oct 2021 – Oct 2022

Challenge

The project’s test data was chaotic. Both the internal team and an external partner modified test data arbitrarily, with minimal communication and no established process.

Data often went missing or was altered without notice, making test failure analysis slow and unreliable. This became especially critical when the external partner’s API underwent a major upgrade, causing widespread regression test failures due to corrupted or missing test data.

Key problems

  • Arbitrary creation and modification of test data without traceability
  • Test data owned by multiple parties, with no shared visibility or governance
  • No separation of concerns across environments
  • It took the team 172-hours to recover from a major upstream API upgrade

Solution

  • Centralized Documentation: All test data was clearly documented, centrally organized, and regularly reviewed.

  • Automated Contract Testing: Contract tests validated test data regularly, enabling earlier detection of mismatches.

  • Structured Test User Management: Test users were named systematically and assigned scoped permissions based on environment and domain.

  • Process Implementation: Established a process for introducing and updating test data, including onboarding flows for new users.

Results

  • Recovery Time: Reduced from 172 hours to 4 hours after major upstream changes.
  • Debugging Efficiency: Test failures could be quickly traced to test-data issues.
  • System Stability: Significantly fewer test data-related issues, leading to higher reliability and team confidence in automated tests

Enabling Continuous Delivery

Global Pumps and Valves Manufacturer
Dec 2023 - May 2024

Challenge

Despite having an efficient regression test pipeline, the client faced persistent instability due to dependencies on an external API. The test pipeline failed frequently due to test data issues, which could only be resolved by external partners, often resulting in significant delays. This lack of control and reliability hindered the move towards a faster, more agile release process.

Solution

  • Mock Server Implementation: Introduced a Wiremock-based mock server as a proxy in the development environment, reducing reliance on the external API.

  • Targeted Endpoint Stubbing: Identified and prioritized problematic API endpoints, and replicated them with stubs in the mock server.

  • Contract Test Integration: Implemented contract tests to keep the mock server in sync with the real API.

  • Autonomy in Testing: Enabled the team to run tests against stable, predictable stubs, eliminating the bottleneck caused by external dependencies.

Results

Increased Pipeline Stability: Test pipeline stability improved 10x, providing the confidence needed to transition to continuous delivery.

Accelerated Release Cadence: Shifted from deploying a release every three weeks to generating multiple deployable artifacts per day.

Increased Team Autonomy: The team gained greater control over their testing environment, reducing delays and dependency on external partners.

Agile Transformation

Bitgrip GmbH
Apr 2024 - Jul 2024

Challenge

The quality engineering team operated as a traditional QA function, with certain testing responsibilities delegated exclusively to this group. This structure discouraged delivery teams from taking complete ownership of product quality, and resulted in quality becoming an afterthought rather than an integral part of the development process. Quality engineers were primarily limited to feature and release testing, underutilizing their potential for strategic impact. Additionally, there was a lack of standardization across projects — different teams solved similar problems in disparate ways, leading to inefficiencies and inconsistencies. The team lacked a clear strategic vision and well-defined responsibilities.

Solution

Strategic Vision and Systems Thinking

  • Redefined the quality engineering function in line with DevOps and Team Topologies principles, focusing on enabling high-performing delivery teams to maintain quality while shipping rapidly.
  • Adopted a systems thinking approach, aiming to improve quality holistically across the organization rather than in isolated silos.

Early Engagement and Risk Management

  • Involved quality engineers early in the development lifecycle to understand project-specific requirements and collaboratively formulate test strategies.
  • Identified potential risks and quality issues at the outset, enabling proactive mitigation.

Standardization and Enablement

  • Established standardized quality engineering processes and templates, ensuring consistency across projects.
  • Trained “quality officers” within each delivery team, fostering a whole-team approach to quality ownership.

Core Focus Areas

  • Shift-Left Testing: Integrated testing practices earlier in the development process to catch issues sooner.
  • Delivery Team Enablement: Empowered delivery teams to take responsibility for quality outcomes, reducing reliance on a centralized QA function.
  • Continuous Improvement: Implemented regular reviews and retrospectives to refine processes and drive ongoing enhancements.

Platform Modernization

  • Migrated the test platform from Jenkins to GitLab, unifying CI/CD practices and templates across the organization.

Results

Strategic Impact: Quality engineering was elevated from a support function to an organization-wide strategic enabler, ensuring quality was embedded in every stage of product development.

Early Risk Mitigation: New projects benefited from early involvement in defining quality strategies and conducting “pre-mortem” workshops to anticipate and mitigate risks.

Continuous Improvement: We helped projects elevate quality bottlenecks and continually improve. E.g. Stabilizing the test environment with a mock server and enabling continuous delivery.

Team Ownership: Training quality officers within delivery teams increased distributed ownership of quality and promoted a whole-team mindset.

Standardized Practices: All projects converged on consistent, mature quality engineering practices.