Keeping Secrets Out of Terraform State: Ephemeral Resources and Write-Only Arguments

The Terraform state file is the most sensitive file that most teams do not treat as sensitive. It sits in a bucket or a storage account that the whole platform team and every pipeline can read, and it holds, in plain text, every database password, API token and private key that was ever passed through a resource argument. Marking a value sensitive changes what is printed on the console. It changes nothing about what is written to disk.

That has been true for as long as the tool has existed, and the usual advice has been to encrypt the backend and restrict access to it. Both are still necessary. What has changed is that a secret can now pass through a run without being recorded at all. Terraform introduced ephemeral resources in 1.10 and write-only arguments in 1.11, and OpenTofu added both in 1.11.0 last month, so the approach is available whichever tool a team runs. This article explains the two mechanisms, then applies them against a local Secrets Manager endpoint and searches every file each tool wrote for the secret.

The pattern being replaced

The long-standing way to create a secret is to generate it as a managed resource and pass its result to whatever stores or consumes it.

# The long-standing pattern: generate a password as a managed resource and hand it to the secret store.
# It works, and it writes the password into the state file twice.

variable "name" {
  description = "Name of the secret."
  type        = string
}

resource "random_password" "db" {
  length           = 32
  special          = true
  override_special = "!#%*-_=+"
}

resource "aws_secretsmanager_secret" "db" {
  name                    = var.name
  recovery_window_in_days = 0
}

resource "aws_secretsmanager_secret_version" "db" {
  secret_id     = aws_secretsmanager_secret.db.id
  secret_string = random_password.db.result
}

output "secret_arn" {
  value = aws_secretsmanager_secret.db.arn
}

Applied with Terraform 1.14.3 and with OpenTofu 1.11.0, this writes the password into the state file twice: once as the result of random_password, and once as the secret_string of the secret version. The JSON rendering of the next plan contains it as well, because a plan carries the prior state. Anyone who can read the state bucket, download a plan artifact from a pipeline, or run terraform state pull has the password, whatever the IAM policy on the secret itself says.

Comparison of two ways to create a secret. With a managed random_password and secret_string, the password is in the state file twice in plain text, in the JSON of the next plan and in the state backup. With an ephemeral resource and a write-only argument it is in none of them; the state holds null and a version number. In both cases the secret store holds a 32-character password.
The same 32-character secret, created two ways, then searched for in every file the tool wrote.

Two mechanisms that work together

Ephemeral values exist only for the duration of one plan or apply. An ephemeral block is declared like a resource and has no address in state; a variable or a child-module output can be marked ephemeral = true. Because such a value is never persisted, it may only flow to places that do not persist it:

  • a write-only argument of a managed resource
  • another ephemeral block
  • a locals value, which then becomes ephemeral too
  • a variable or child-module output declared with ephemeral = true
  • a provider block
  • a provisioner or its connection block

Anywhere else, the run stops with an error.

Write-only arguments are the other half. A provider marks an argument as write-only, by convention with a _wo suffix, and Terraform sends its value to the provider without storing it in the plan or the state. Since nothing is stored, nothing can be compared, so a change to the value alone produces no plan. Each write-only argument therefore comes with a version argument, an ordinary number that is stored. Changing the number is what tells the provider to write the value again.

Flow diagram. An ephemeral random_password exists in memory for one run and feeds the write-only argument secret_string_wo, which the provider sends to Secrets Manager and then discards. State and plan hold null for the value and the version number. Rotation is done by raising the version. Old state versions, other routes for secrets and access to the secret store are not addressed.
The value passes through. The version number is the only thing Terraform keeps.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
    E["ephemeral random_password
exists in memory for one run"] --> W["secret_string_wo
write-only argument"] V["password_version
an ordinary number"] --> WV["secret_string_wo_version"] W --> P["AWS provider
uses the value, keeps nothing"] WV --> P P --> S["Secrets Manager
the only place the value lives"] WV --> ST["State and plan
hold the version, and null for the value"] S --> APP["Application
reads the secret at run time"] classDef compute fill:#FFF3E0,stroke:#FF9900,stroke-width:2px,color:#232F3E classDef db fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74 classDef edge fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95 classDef store fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202 class E,W edge class V,WV,P compute class ST db class S,APP store

The same secret with nothing recorded

# The same outcome with nothing secret in state or plan. The password exists only while the run is in
# progress, and reaches the secret store through a write-only argument.

variable "name" {
  description = "Name of the secret."
  type        = string
}

variable "password_version" {
  description = "Increase by one to generate and store a new password. Nothing else triggers a rotation."
  type        = number
  default     = 1

  validation {
    condition     = var.password_version >= 1 && floor(var.password_version) == var.password_version
    error_message = "password_version must be a whole number, starting at 1."
  }
}

# Not a resource: it has no address in state. A new value is generated on every run that opens it.
ephemeral "random_password" "db" {
  length           = 32
  special          = true
  override_special = "!#%*-_=+"
}

resource "aws_secretsmanager_secret" "db" {
  name                    = var.name
  recovery_window_in_days = 0
}

resource "aws_secretsmanager_secret_version" "db" {
  secret_id = aws_secretsmanager_secret.db.id

  # Sent to the provider, never stored. The version number is what Terraform tracks and diffs.
  secret_string_wo         = ephemeral.random_password.db.result
  secret_string_wo_version = var.password_version
}

output "secret_arn" {
  description = "Consumers read the password from the secret store by this ARN. Terraform cannot give it to them."
  value       = aws_secretsmanager_secret.db.arn
}

Three things differ from the first version. resource "random_password" has become ephemeral "random_password", secret_string has become secret_string_wo, and a version number has been added. The results, identical on both tools:

Looked forResult
A 32-character password in the secret storePresent
That password in any file in the working directory after applyIn none of them
That password in the JSON of the next planAbsent
What the state holds for the pairsecret_string_wo is null; secret_string_wo_version is 1
An entry in state for the ephemeral resourceNone

The search is not a string match against a guess. The script reads the secret back from the store after the apply and looks for those exact bytes in every file under the working directory, the state, its backup and the saved plan included.

The output at the bottom of the listing makes the consequence explicit. Terraform cannot hand this password to anything, because it does not have it. Whatever needs the password reads it from the secret store, under the store’s own access policy, which is where that decision belonged all along.

Rotation is a number

An ephemeral random_password produces a new value every time a run opens it. That sounds alarming until the version argument is taken into account: the provider only writes when the version changes.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','signalColor':'#0073BB','signalTextColor':'#0F172A','actorBkg':'#FFF3E0','actorBorder':'#FF9900','actorTextColor':'#232F3E','noteBkgColor':'#FFF8E7','noteBorderColor':'#FF9900','noteTextColor':'#232F3E'}}}%%
sequenceDiagram
    participant C as Pipeline
    participant T as Terraform
    participant P as AWS provider
    participant S as Secrets Manager
    C->>T: apply, password_version 1
    T->>T: Open ephemeral random_password
    T->>P: secret_string_wo and version 1
    P->>S: Write the secret value
    Note over T: State records version 1 and null
    C->>T: apply again, password_version 1
    Note over T: Version unchanged, no changes planned
    C->>T: apply, password_version 2
    T->>T: Open ephemeral random_password, new value
    T->>P: Replace the secret version
    P->>S: Write a new secret value
ActionResult on both tools
Plan again with the same versionNo changes
Apply again with the same versionThe stored password is unchanged
Plan with the version raised to 21 to add, 0 to change, 1 to destroy: the secret version is replaced
Apply with the version raised to 2The stored password is different
Search the working directory for either passwordIn no file

The plan line that drives it reads secret_string_wo_version = 1 -> 2 # forces replacement, and the value itself appears nowhere in the plan. This makes rotation an explicit, reviewable change. A pull request that raises password_version from 1 to 2 is a rotation, and nothing else is. The opposite failure is worth knowing about too: editing a write-only value without raising its version does nothing, silently, because there is no stored value for the new one to differ from.

A secret that already exists

Not every secret is generated. A token issued by a third party arrives from outside, and the pipeline has to pass it in. An input variable marked ephemeral does that without the value being written to the plan or the state.

# A secret that already exists outside Terraform, such as a token issued by a third party, passed in by the
# pipeline. Marking the variable ephemeral keeps it out of state and plan; it can then only flow to places
# that do not persist it.

variable "name" {
  description = "Name of the secret."
  type        = string
}

variable "api_token" {
  description = "Supplied by the pipeline as TF_VAR_api_token on every run."
  type        = string
  ephemeral   = true
  sensitive   = true
}

variable "api_token_version" {
  description = "Increase by one when the token has been reissued."
  type        = number
  default     = 1
}

resource "aws_secretsmanager_secret" "token" {
  name                    = var.name
  recovery_window_in_days = 0
}

resource "aws_secretsmanager_secret_version" "token" {
  secret_id                = aws_secretsmanager_secret.token.id
  secret_string_wo         = var.api_token
  secret_string_wo_version = var.api_token_version
}

The test supplies a known token through TF_VAR_api_token and looks for it afterwards.

Looked forResult on both tools
The token in the saved plan file, or in its JSON renderingAbsent
Applying that saved plan without supplying the token againRefused: “No value for required variable”
Applying the saved plan with the token supplied againApplies
The token in the secret storePresent
The token in any file in the working directoryIn none of them

The second row changes how a two-stage pipeline has to be built. With ordinary variables, the plan stage needs the inputs and the apply stage needs only the plan file. An ephemeral variable is not in the plan file, so the apply stage must be given it again. A pipeline that plans in one job and applies in another has to make the secret available to both.

What the tools refuse

The protection is only as good as the rule that stops an ephemeral value reaching somewhere it would be stored. Sending one to an ordinary output is the obvious mistake, and both tools stop at validation.

ToolError
Terraform 1.14.3Error: Ephemeral value not allowed
OpenTofu 1.11.0Error: Output does not allow ephemeral value

The same rule applies to an argument that is not write-only, to count and for_each, and to a root-module output. If a design needs Terraform to output the secret, the design still has the original problem.

Migrating an existing secret

Most teams are not starting clean. The test applies the first configuration, then swaps in the write-only version of the same resources and applies again.

ObservationResult on both tools
Plan for the change1 to add, 0 to change, 2 to destroy: the managed random_password is removed and the secret version replaced
The stored secret afterwardsA new password, generated by the ephemeral resource
The old password in the new state fileAbsent
The old password in terraform.tfstate.backupPresent

Two consequences follow. The migration is a rotation, so whatever uses the secret must be ready to pick up a new value when it is applied. And the old password does not leave the state’s history. With local state that is one backup file. With a remote backend it is every earlier version the bucket or the service has kept, for as long as the retention policy keeps them. Since the migration rotates the secret anyway, the old value is dead on arrival, which is the right way to deal with it: treat every secret that has ever been in state as disclosed to everyone who could read the state, and rotate it as part of the move.

Testing a module that uses them

A write-only argument gives a test very little to hold. The module’s tests run against a mocked AWS provider, with the random provider running for real since it needs no credentials, and they assert on the version and on input validation.

# The AWS provider is mocked; the random provider runs for real and needs no credentials.
# A test cannot usefully assert on the write-only value itself, so it asserts on the version that travels with it.

mock_provider "aws" {}

variables {
  name = "sample/db-password"
}

run "version_is_tracked_and_value_is_not" {
  command = plan

  assert {
    condition     = aws_secretsmanager_secret_version.db.secret_string_wo_version == 1
    error_message = "The version argument should carry the default of 1."
  }
}

run "rejects_a_fractional_version" {
  command = plan

  variables {
    password_version = 1.5
  }

  expect_failures = [var.password_version]
}

Both tests pass on both tools. One difference between the tools turned up along the way. An assertion that secret_string_wo == null in a mocked plan holds on Terraform 1.14.3 and fails on OpenTofu 1.11.0, which reports a sensitive value there. After a real apply, both tools hold null in state. The portable test asserts on the version argument and leaves the value alone.

What this does not solve

Keeping a secret out of state closes one exposure. A design review should be clear about the ones that remain.

ExposureAddressed?What still has to be done
Secret readable in the current state fileYesNothing, for arguments that have a write-only form
Secret readable in saved plans and plan artifactsYesNothing
Secret in earlier state versions and backupsNoRotate on migration; set a retention period on state versions
Arguments with no write-only formNoCheck each provider’s documentation. Until it has one, the value is stored as before
Other sensitive data in stateNoConnection strings, generated keys and resource attributes are still there. Keep the backend encrypted and access-controlled
Secret visible to the pipelineNoThe runner holds an ephemeral variable in its environment. Scope it to the job and mask it in logs
Who can read the secret storeNoThat is the store’s own access policy, and now the only one that matters for this secret
⚠️
PROVIDER SUPPORT DECIDES

Both mechanisms depend on the provider. An ephemeral resource exists only if the provider implements one, and an argument is write-only only if the provider declares it so. The examples here use aws_secretsmanager_secret_version from the AWS provider and random_password from the random provider. For anything else, the resource’s documentation is the place to check, and a missing _wo argument means the pattern is not yet available for that resource.

What adoption costs

There is no licence or runtime cost. The cost is in versions, habits and a little lost convenience.

CostDetail
Minimum versionsTerraform 1.11 for write-only arguments, 1.10 for ephemeral resources alone; OpenTofu 1.11. Set required_version so an older binary fails clearly
Provider versionsRecent enough to offer the ephemeral resource and the write-only argument in question
No drift detection on the valueTerraform cannot tell that somebody changed the secret out of band. Rotation is something you do on purpose, by number
No outputs carrying secretsModules that exported a password have to export an ARN or an identifier, and consumers have to read the store
Pipeline changesAn ephemeral variable must be present at apply as well as at plan
Migration is a rotationPlan it with the owners of whatever uses the secret

Decision framework

SituationUse
Terraform generates a secret and a store keeps itAn ephemeral resource into a write-only argument, with a version variable
A secret arrives from outside and Terraform must place itAn ephemeral input variable into a write-only argument
A provider needs a credential to configure itselfAn ephemeral resource or variable in the provider block
The resource has no write-only argument yetThe classic pattern, an encrypted and restricted backend, and a note to revisit when the provider adds one
Something downstream wants the secret as a Terraform outputChange the consumer to read the secret store. The output is the leak
The platform can avoid the secret entirelyPrefer that: a managed identity, an IAM role, or a service that generates and manages its own master password

The last row is the one to reach for first. A secret that is never created needs no protection, and write-only arguments are for the cases where one has to exist.

Key takeaways

  • sensitive masks console output. The classic pattern still wrote the password to state twice and into the next plan.
  • An ephemeral value into a write-only argument left the password in no file on disk, on Terraform 1.14.3 and on OpenTofu 1.11.0 alike.
  • The version argument is the only thing that triggers a write. Raising it is a rotation, and changing the value without raising it does nothing.
  • An ephemeral variable is not in a saved plan, so the apply stage of a pipeline must be given it again.
  • Migration rotates the secret and leaves the old one in state history. Treat anything that was ever in state as disclosed.

If you have moved a large estate to write-only arguments, I would be interested to hear in the comments which resources still lack them and how you handled the state history.

ℹ️
HOW THIS WAS CHECKED

Every result in this article comes from applying the configurations with Terraform 1.14.3 and OpenTofu 1.11.0, AWS provider 6.15.0 and random provider 3.7.2, against moto 5.2.5 as a local Secrets Manager endpoint, in GitHub Actions. After each apply a script reads the secret back from the endpoint and searches every file in the working directory for it. Forty-eight observations were recorded, twenty-four for each tool, and the two tools agree on all but the mocked-plan probe described above. No AWS account was used, so the behaviour of the real service, IAM and KMS was not exercised, and only the Secrets Manager secret version resource was tried. The workflow run has the logs.

Glossary

  • State— The file in which Terraform or OpenTofu records every managed resource and its attributes, in plain text.
  • Ephemeral resource— A block whose result exists only during a run and is never written to state or plan.
  • Ephemeral variable— An input variable marked ephemeral = true, usable only where its value will not be persisted.
  • Write-only argument— A resource argument, conventionally ending in _wo, that is sent to the provider and not stored.
  • Version argument— The stored number paired with a write-only argument. Changing it makes the provider write the value again.
  • Saved plan— The file written by plan -out, applied later. It contains prior state and planned values.
  • moto— An open-source emulator of AWS service APIs, used here as a local Secrets Manager endpoint.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.