The dev environment cost 11x production, and Terraform could not fix it

At a financial-reporting SaaS company, I pulled the Azure spend for a set of App Service Plans running on an App Service Environment v3 and found something that is more common than anyone likes to admit.

The development subscription cost eleven times more than production. Dev was running at roughly $22,000 a month against production's $1,970, and accounted for 85% of the total bill across the three environments.

Production had been sized carefully, because someone was watching it. Dev had been sized once, generously, by somebody who correctly judged that being slow was worse than being expensive, and then nobody looked again.

What the metrics actually said#

Before touching anything I went to Datadog and pulled resource utilisation for every plan. The pattern was consistent: CPU sat low across almost all of them, while memory was the binding constraint. One service, an XBRL processing workload, was genuinely memory hungry and running close to its ceiling.

So the plans were not simply oversized. They were oversized along the wrong axis, and undersized along the right one.

The reason is visible as soon as you lay out what Azure offers on Isolated v2:

There are two ladders here, not one. The plain v2 tiers double both vCPU and memory at each step. The memory-optimised mv2 tiers hold the vCPU count and double the memory instead.

If you only know about the first ladder, the only way to get 32 GB of memory is I3v2, which also hands you 8 vCPUs. The dev plans had climbed that ladder to reach the memory they needed, and were paying for CPU they never touched as the price of admission.

I2mv2 offers the same 32 GB on 4 vCPUs. Identical memory, half the compute.

The provider did not have those SKUs#

The change should have been one attribute in a Terraform file. It was not, because azurerm_service_plan validates the SKU name against a hardcoded list in the provider, and the memory-optimised tiers were not in it:

var isolatedSkus = []string{
	"I1", "I2", "I3", // Isolated V1 - ASEV2
	"I1v2", "I2v2", "I3v2", "I4v2", "I5v2", "I6v2", // Isolated v2 - ASEv3
}

The SKUs existed in Azure. They were documented and purchasable. They were simply absent from a slice in Go, so terraform plan failed validation before the request ever reached the API.

This is a specific category of problem worth naming: not "the platform cannot do this", but "the provider has not caught up with the platform". Providers lag. Azure ships SKUs, regions and properties continuously, and a hand-maintained list will always trail that by some amount.

Four ways out, and the one people forget#

Use azapi. The escape-hatch provider that talks to the Azure REST API directly. Correct for genuinely new or preview resources. The cost is that the resource now sits outside the main provider's schema, so you lose its validation and anyone reading the module has to context-switch.

Drop to ARM or Bicep for that piece. Same trade, heavier. Now you run two infrastructure tools and have to decide which owns what.

Work around it in place. Create the plan at a supported SKU, change it by hand, then add lifecycle { ignore_changes = [sku_name] } so Terraform stops reverting it. This is the one I see most often and it is the worst of the four. You have deliberately blinded your state file to the exact attribute the whole exercise is about. Six months later nobody remembers why the plan does not match the code.

Fix the provider. Add the SKUs, write a test, open a pull request.

The fourth option is the one people skip, usually because it feels like somebody else's job. It mostly is not. The AzureRM provider is developed in the open, and a missing entry in a validation list is close to the most tractable change anyone can make to it.

What contributing actually involves#

The code change was trivial:

var isolatedSkus = []string{
	"I1", "I2", "I3", // Isolated V1 - ASEV2
	"I1v2", "I2v2", "I3v2", "I4v2", "I5v2", "I6v2", // Isolated v2 - ASEv3
	"I1mv2", "I2mv2", "I3mv2", "I4mv2", "I5mv2", // Isolated v2 - ASEv3 memory optimized
}

One line. If that had been the whole contribution it would not have merged, and it should not have.

The real work is the acceptance test. The provider expects a test that stands up real infrastructure and asserts the resource exists, which for an App Service Environment means a resource group, a virtual network, a delegated subnet, the environment itself, and then the plan on top. That fixture is most of the diff. The merged PR was three files and about eighty lines added, the majority of it test setup and documentation rather than the fix.

It also takes time. I opened the PR in December and it merged at the end of February, shipping in provider v4.21.0. Roughly ten weeks on a repository handling an enormous volume of contributions.

That argues for doing both at once. Unblock the immediate work however you need to, and open the pull request in parallel so the workaround has an expiry date. What makes a workaround dangerous is not the workaround, it is the absence of any plan to remove it.

What changed, and what it cost#

Once the provider supported the SKUs, the moves were straightforward and different per environment.

In dev, six plans went from I3v2 to I2mv2. Same 32 GB of memory, 8 vCPUs down to 4. Nothing about the applications changed; they had never been using the CPU.

In production and sandbox, the plans went from I1v2 to I1mv2. Same 2 vCPUs, memory doubled from 8 GB to 16 GB. That costs more, not less.

Roughly $9,300 a month, about $112,000 a year, a 36% reduction.

The part that matters is the two lines that went up#

It would be easy to present this as a cost-cutting exercise. It was not, and the sandbox and production rows are the proof.

Production spend went up. So did sandbox. The memory-hungry service had been running near its ceiling the entire time, and the honest reading of the metrics was that it needed more headroom, not less. Fixing that cost an extra $462 a month across the two environments, and it was the right call independent of the saving.

Anyone can make a cloud bill smaller by starving workloads. The number means something only when you also spent more in the places that needed it, and the total still came down by a third.

The other thing worth sitting with is where the money actually was. Production was fine. Production is almost always fine, because production is what people watch. All of the waste was in a dev environment that had been sized once on a reasonable instinct and then never revisited, while quietly consuming 85% of the budget for the service.

If you are looking for cloud savings and starting with production, you are probably starting in the wrong place.

When to contribute upstream, and when not to#

Fixing the provider is not always right. A rough test:

Contribute when the gap is declarative and bounded: a missing SKU, region, or enum value, or a validation rule stricter than the API itself. These are small, testable, and obviously correct, which is what a maintainer can review quickly.

Do not contribute when the gap is architectural: a resource that does not exist, or a behaviour change that would affect other users. Those are genuine design discussions, and a production deadline is the wrong context for having one. Reach for azapi and move on.

And check the issue tracker first. Someone has often filed it already, and occasionally fixed it in a release you have not picked up yet.

The contribution is PR #28316 in hashicorp/terraform-provider-azurerm, merged into v4.21.0.

One footnote: Azure has since introduced an Isolated v4 generation, so this particular list is already partly historical. That is rather the point. The provider's list will be out of date again, probably by the time you read this, and the way it gets fixed is that somebody who needed the missing entry went and added it.