Scaling Quality: How We Built Resilient QA Practices and Migrated from Heroku to AWS at Roadie
Jeff Persson
6/30/20253 min read
Scaling Quality: How We Built Resilient QA Practices and Migrated from Heroku to AWS at Roadie
At Roadie, we faced the classic scaling challenge: how do you preserve product quality, speed of delivery, and operational resilience in the midst of startup chaos, tight deadlines, and evolving priorities?
As the QA and Performance Engineering leader, I had the privilege of tackling this challenge head-on. Our journey wasn't just about improving testing—it was about building a culture of quality, architecting scalable systems, and aligning QA with the speed and flexibility required for modern DevOps and product innovation.
QA That Scales With the Business
One of my first goals was to establish QA processes that weren’t just checklists, but strategic assets—designed to scale, evolve, and add value throughout the software development lifecycle.
We implemented foundational practices that included:
Diversified Testing: We expanded QA beyond functional validation. Integration testing, exploratory testing, backend validation, and performance modeling all became part of our DNA.
Reusable Test Artifacts: We simplified and modularized test case creation, allowing teams to build on a growing library of shared assets.
CI/CD-Integrated Automation: Our test automation suite was deeply integrated with our CI/CD pipelines. This enabled safe, frequent deployments where many commits could go straight to production without regressions.
The result? Faster releases, fewer escaped defects, and QA that was viewed not as a bottleneck, but as an accelerator.
Infrastructure Evolution: Heroku to AWS
As Roadie matured, so did our infrastructure needs. Heroku gave us great speed early on, but we began hitting limits—particularly with performance testing, observability, and system flexibility.
We migrated our services to AWS Cloud, and with that came the opportunity (and responsibility) to rethink how we tested and tuned our systems.
Here’s how we approached it:
Multi-Class Server Testing: We tested services across a variety of RDS, ELB, EKS, and EC2 instance classes (T3, C5, M5, etc.) to identify the best performance-to-cost ratio for each workload.
Containerization & Infrastructure as Code: With ECS and Terraform, we could replicate environments for test, stage, and production—ensuring environment parity and repeatable performance benchmarks.
Configuration Optimization: We fine-tuned EKS settings, auto-scaling thresholds, and thread pools based on real-world load simulations. These weren’t theoretical improvements—they reduced latency and stabilized resource consumption under peak load.
Performance Testing with Purpose
Performance testing wasn’t an afterthought—it was a design principle. We built full-stack load tests to simulate real-world user behavior at increasing scale and durations, focusing especially on backend services, data pipelines, and AI-driven logistics components.
Bottleneck Identification: We proactively uncovered issues long before they hit production, from slow endpoints to inefficient DB queries and memory leaks.
Production-Like Load Scenarios: Rather than synthetic loads, we used traffic patterns that closely mirrored production usage (e.g., peak-time delivery scheduling and routing).
Strategic Cost Reduction: One particularly rewarding project involved reducing spend on high-frequency, high-cost API calls. Through intelligent caching and direct vendor approval, we significantly lowered our usage of the Google Maps API—without impacting functionality.
The Business Outcome: High-Quality Releases & Successful Acquisition
All of this culminated in more than just great test reports. The systems and culture we built enabled:
Rapid release cycles without compromising stability
Audit-friendly QA practices that passed due diligence with flying colors
A measurable decrease in defect escape rate and improved MTTR (mean time to resolution)
A strong technical foundation that contributed directly to Roadie’s successful acquisition at a premium valuation
Final Thoughts: QA as a Strategic Enabler
Too often, QA is reactive—focused on catching bugs after the fact. At Roadie, we made QA proactive, embedded in the architecture, infrastructure, and culture of the company.
By building scalable practices, embracing DevOps, investing in performance testing, and aligning closely with infrastructure decisions like our AWS migration, we delivered not just quality software, but a quality company.
About the Author:
Jeff Persson is a QA and Performance Engineering leader with over 20 years of experience helping startups and enterprise teams build faster, more reliable, and more scalable software systems. He specializes in full-stack performance testing, test automation architecture, and cloud-native quality engineering.