What Should My Pilot Include — One Service or a Whole Fleet?

When planning a cloud migration or a major optimization exercise, one of the toughest decisions is defining your pilot scope. Should you start small, picking a single service with a seemingly easy workload? Or should you go broad, tackling the entire fleet at once to get a complete picture? The answer isn’t trivial and often depends on understanding the behavioral nuances of your services, the cloud provider’s CPU sharing model, and the data you collect from your monitoring tools.

Over my 12 years of cloud infrastructure and Site Reliability Engineering practice across AWS, Azure, and Google Cloud, I’ve seen countless pilot designs stumble on vague assumptions. In this post, I’ll walk you through what a robust pilot should really include, grounded in concrete data points, realistic workload characterizations, and smart use of cloud-native tools like AWS Compute Optimizer and Azure Advisor.

Why Pilot Scope Matters: One Service vs. Whole Fleet

The instinct to choose a single service with a low average CPU or limited traffic is understandable—it feels safe and controlled. Unfortunately, this approach often hides significant cloud waste and underutilization because it ignores the critical reality that many always-on small services pack hidden peaks and bursts. Conversely, trying to optimize your entire fleet at once can be unwieldy and increase risk without a clear hypothesis.

Here are the core trade-offs in pilot scope design:

One service pilot: Easy to control and rollback; less noise; but may not represent fleet-wide variability or peak usage.
Whole fleet pilot: Provides a comprehensive view; captures variability across service types; but harder to monitor and manage risk.

Instead of an either/or, I recommend a representative workload sampling approach that balances the two extremes by focusing on services that cover the breadth of usage patterns and resource profiles.

Understanding Your Workload: The Key to a Safe and Effective Pilot

Before you design any pilot, you must ask, “What do the P95 and P99 utilization percentiles look like?” And, critically, “What is the duration of spikes?” These questions are more meaningful than a blunt average CPU of 10% that lulls many teams into a false sense of security.

Why Averages Fail

Cloud teams often rely on average CPU utilization metrics to justify downsizing instances or switching to burstable/shared CPU offerings. This is a mistake for two reasons:

Peaks matter: Your application performance, latency, and error rates flare up during CPU spikes, not during quiet average states.
Duration counts: Even if the P99 CPU spikes to 90%, but those spikes last seconds versus minutes, your sizing decisions and migration plans should differ.

Using Percentiles and Spike Durations Effectively

Effective pilot planning requires monitoring tools that let you inspect the percentiles (P95, P99) of CPU, memory, and I/O utilization over windows of 5, computingforgeeks 15, 30 minutes, or longer, depending on your application’s typical spike duration.

For example, if you observe that your service’s CPU hits 80% utilization at P99 but only for 10 seconds at a time, a burstable instance with shared CPU may suffice. But if those bursts last several minutes or cause downstream bottlenecks, you need a bigger instance or isolated CPU (dedicated cores) to maintain SLAs.

Shared CPU Definitions Differ by Provider: Don’t Assume Equivalence

A common misstep is equating “shared CPU” offerings across AWS, Azure, and GCP without appreciating different implementation subtleties. For instance:

AWS T-series instances offer CPU credits to burst above baseline, but when credits run out, performance throttles unpredictably.
Azure B-series VMs operate similarly but have different credit earning and spending behaviors.
Google’s E2 shared-core VMs prioritize steady baseline CPU but may allow short bursts, often differing in CPU scheduling fairness.

Additionally, the underlying hypervisor’s scheduling and noisy neighbor effects vary among clouds, affecting availability and latency during bursts.

This means before choosing shared CPU, you must measure the real-world performance and outage patterns—and never blindly trust vCPU counts as performance guarantees.

Leverage AWS Compute Optimizer and Azure Advisor to Size Your Pilot Properly

Both AWS and Azure offer native tools to guide pilot planning, but using them correctly takes finesse.

AWS Compute Optimizer

AWS Compute Optimizer analyzes your account’s historical utilization data and recommends right-sizing actions for EC2 instances, Auto Scaling groups, and EBS volumes.

Recommendation confidence: Prioritize recommendations with high confidence, which are based on at least two weeks of stable usage data.
Granularity: Use it to identify candidacy services where P95/P99 CPU and network egress indicate potential savings without sacrificing performance.
Storage and Egress: Remember to factor in storage costs and network egress—Compute Optimizer focuses on instance CPU/memory footprint but ignores egress bills, a common source of cost surprises.

Azure Advisor

Azure Advisor provides similar optimization insights tailored to Azure VM types, Azure SQL databases, and other services.

Prioritize operational best practices: Look for recommendations addressing underutilized VMs, idle disks, and overprovisioned networking.
Use Azure’s cost analysis tools: Complement Advisor’s recommendations with Azure Cost Management reports to capture hidden cloud waste like orphaned resources or backup snapshots.
Validate the performance impact: Cross-reference Advisor’s suggested VM sizes with percentile CPU load and latency metrics to prevent disruptions post-migration.

Defining Rollback Criteria for Your Pilot: A Non-Negotiable Step

Before launching your pilot, define clear rollback criteria with quantitative triggers based on your monitoring data. For example:

You may specify that if the P99 latency spikes above your SLA by more than 15% for over 5 minutes during the pilot, the migration is paused and instances are reverted.
If error rates increase by more than 0.5% sustained for over 10 minutes, revert to baseline instances.

Rollback criteria make your pilot a controlled migration experiment rather than a stealth risk multiplier.

Putting It All Together: A Checklist for Pilot Scope Design

Identify representative workloads: Include services that cover typical, peak, and burst usage patterns across your fleet.
Analyze P95 and P99 CPU and memory percentiles: Use monitoring tools with configurable observation windows matching spike duration.
Understand shared CPU semantics: Benchmark burstable VMs on your cloud platform and understand their throttling models.
Leverage cloud optimization tools: Use AWS Compute Optimizer and Azure Advisor to cross-check your instance size choices and reveal hidden wastage.
Factor in storage and egress: Never base cost decisions solely on compute — check your storage and network bills too.
Define rollback criteria: Set quantitative failure thresholds before starting your pilot.
Incrementally widen your scope: Start with a small but representative sample, then scale up to fleet-wide once confident.

Conclusion

Whether you pilot one small service or the whole fleet depends less on convenience and more on rigorously understanding your workload, cloud provider quirks, and performance dynamics. Avoid the trap of focusing on average CPU or blindly adopting burstable instances just because they seem cheaper. Use percentile metrics, measure duration of spikes, and incorporate cloud-native optimization tools to design a pilot with a scope that balances risk and representativeness.

By applying these principles and embedding rollback criteria, you transform your pilot into a controlled migration experiment—one that uncovers cloud waste, balances performance, and builds confidence for scaling optimization efforts fleet-wide.

Ready to design your next pilot? Start with the percentiles, prioritize peak-aware metrics, and make sure your pilot scope mirrors your overall fleet characteristics.