The Terraform state file captures the last known state of your infrastructure. Simply put, it's the single most critical file in your configuration, making it crucial to understand its role and significance.

In this blog post, we’ll explore the Terraform state file, discuss how to tailor it to your setup, and cover best security practices.

Disclaimer

All use cases of the Terraform state file discussed here work similarly in OpenTofu, the open-source Terraform alternative. However, to keep it simple and familiar for DevOps engineers, we will use “Terraform state file” as a catch-all term throughout this blog post.

What is the Terraform state file?

Terraform records its known state of the world in the Terraform state file.

In essence, the state file contains a list of the resources that have been created by Terraform. Each resource in the state file is recorded with all its metadata and attributes, including any secret values.

The content of the state file represents a snapshot of what your infrastructure looked like after the last [.code]terraform apply[.code]. In other words, it is not a live representation of your infrastructure.

Purpose of the Terraform state file

New Terraform users often wonder what the purpose of the Terraform state file is.

Terraform is a provider-agnostic tool for Infrastructure as Code. It works equally well for creating resources on AWS, Azure, GCP and even your local filesystem. For this reason, Terraform requires a provider-agnostic way to keep track of the resources it creates through the different providers. The state file is this provider-agnostic record.

The state file has some advantages. Two important benefits are:

• You can use state file content to detect resource drifts in your environment. Resource drift is when a resource is modified in some way outside of Terraform. Drift shows up as a discrepancy between your state file and what is actually deployed in your environment.
• You can utilize state file locking to prevent two processes from simultaneously making changes to the same Terraform state and the same underlying provider resources. If multiple processes would update the state file at the same time, chances are you would end up with a corrupt state file.

Syntax of the Terraform state file

The Terraform state is stored in JSON format. You generally do not read the state file, but it can be valuable to understand the structure and syntax of the state file.

Consider the following simple Terraform configuration that creates a random password using the random provider and outputs the value of the password:

terraform {
    required_providers {
        random = {
            source  = "hashicorp/random"
            version = "3.6.3"
        }
    }
}

variable "password_length" {
    type = number
}

resource "random_password" "account" {
    length = var.password_length
}

output "password" {
    value     = random_password.account.result
    sensitive = true
}
A simplified version of the corresponding Terraform state file after a [.code]terraform apply[.code] is shown below:
{
    "version": 4,
    "terraform_version": "1.10.4",
    "outputs": {
        "password": {
            "value": "p2r)_2BG?w"
        }
    },
    "resources": [
        {
            "mode": "managed",
            "type": "random_password",
            "name": "account",
            "provider": "provider[\"registry.terraform.io/hashicorp/random\"]",
            "instances": [
                {
                    "schema_version": 3,
                    "attributes": {
                    "length": 10,
                    "result": "p2r)_2BG?w"
                }
            ]
        }
    ]
}

Notice the following details:

• The root level [.code]default[.code] property specifies the version of the Terraform state document format. The current version is 4. The state file is in general backwards-compatible with older versions, but care must be taken if a new version is introduced. However, the state file format is rarely updated.
• The [.code]terraform_version[.code] property shows the version of Terraform that was used when the state file was last updated.
• The [.code]outputs[.code] property contains all the outputs and their values. Note how the password is shown in plain text.
• The [.code]resources[.code] array contains all the resources that are part of the state. In this example, the array contains a single resource. For each resource you can see details of what type it is, the provider that it was created with, and all of its attributes including secret values.

Note that variables are not an explicit part of the state file. The values of variables only appear implicitly (e.g., in the [.code]length[.code] property of the resource shown in the state file above).

The full state file includes some additional metadata, most importantly a [.code]serial[.code] property which is a monotonically increasing serial number that increases each time the state file is updated.

Working with the Terraform state file

In this section, we will cover the basics of working with the Terraform state file, covering some of the fundamental related concepts.

Terraform state file lifecycle

For a new configuration, the state file is initially created when you run your first [.code]terraform apply[.code] command.

Updates to your infrastructure follows these high-level steps:

1. Terraform reaches out to the remote systems (e.g., AWS or Azure) to read the current state of your resources.
2. Terraform compares your current Terraform configuration (i.e., .tf files) with your prior state (i.e., your state file).
3. Terraform computes the required changes to bring the state of the remote objects to match your Terraform configuration.
4. If you apply the proposed changes, Terraform uses its providers to update the remote objects.
5. The new state of the world is recorded in the state file.

This process repeats each time you make changes to your infrastructure and run through [.code]terraform plan[.code] and [.code]terraform apply[.code].

Remote and local backend options 

Terraform stores its state files in a state backend. You can configure what state backend you want to use in the [.code]terraform[.code] block of your configuration.

If you do not specify an explicit state backend, Terraform uses the local backend. By default, the local backend stores the Terraform state in a local state file named terraform.tfstate. The local state file is stored in the same directory where you run [.code]terraform apply[.code].

You can explicitly configure the local backend by adding this to your Terraform code:

terraform {
    backend "local" {
        path = "terraform.tfstate"
    }
}

If you collaborate with others on the same Terraform configuration you need a way to share the state file between collaborators. This is what remote state backends are for. A remote state backend is an external system that is used to store the state file.

Examples of popular remote state backends are AWS S3 buckets, Azure storage, and Google Cloud Storage (GCS).

An example of configuring an AWS S3 bucket as a remote state backend:

terraform {
    backend "s3" {
        bucket = "mystoragebucket"
        key = "path/to/state/terraform.tfstate"
        region = "eu-west-1"
    }
}

Remote state storage backends normally require credentials to properly authenticate and authorize access. These credentials can often be configured directly in the [.code]backend[.code] block, or more commonly as environment variables.

The benefit of using a remote state backend is the ability to share the state file with others. It allows the use of platform-specific features for protecting the state file.

You could also use a platform-specific remote state backend. One example is env0. In your environment settings in env0, confirm that you want to use env0 for remote state:

You can also control what other environments in your env0 organization can access your state.
Read this article to learn more about Terraform's local and remote backend options.

Sharing state between Terraform configurations

Technically speaking, you could keep all your infrastructure in a single Terraform configuration “monolith”. This has several benefits, one of which is simplifying dependency management between different resources and data sources. 

However, this approach could prove to be a bad idea for large infrastructures due to the potential blast radius in case of mistakes and errors.

Therefore, it is often a better option to split your infrastructure into different Terraform state files, each holding its own configuration and interacting with the other files via three main methods:

1. Using a data source to read attributes of resources that are provisioned via another Terraform configuration
2. Reading outputs from a different Terraform configuration using the [.code]terraform_remote_state[.code] data source
3. Use an Infrastructure as Code management platform (e.g., env0) to share data between multiple Terraform configurations

The first and third options are usually preferred. The second option, in general, is less recommended, as it requires you to provide read access to the state file, which introduces an additional element of risk.

Advanced state file operations

The common way to make updates to your Terraform state file is by updating the resources in your Terraform configuration files and running [.code]terraform apply[.code] to apply the changes.

However, in some situations you will need to manipulate your Terraform state outside of the normal workflow. 

This includes scenarios in which you want to import resources that were created through some other means or when you want to refactor your Terraform configuration (renaming resources, moving resources into modules, etc).

Manipulating your state with the Terraform CLI

You can use the Terraform CLI to manipulate the state file configuration, but only as a last resort if no other option is available. The downside of this approach is that you will update the state file without having the option to preview the changes.

Listing resources in your state

List all your state resources with the [.code]terraform state list[.code] command. This command is safe to run, and the output is a list of resource addresses. 

Below is an example output for a Terraform state containing an AWS VPC with three subnets:

$ terraform state list
aws_vpc.default
aws_subnet.all[0]
aws_subnet.all[1]
aws_subnet.all[2]

Moving a resource in your state

Use the [.code]terraform state mv[.code] command to move a resource from one resource address to another. You need to do this if you rename a resource or refactor parts of your Terraform configuration into modules. You pass the source address and destination address to this command.

Below is an example of moving a resource at the source address:

 [.code]aws_subnet.all[1][.code] to the destination address [.code]aws_subnet.europe["eu-west-1"][.code]:

$ terraform state mv aws_subnet.all[1] aws_subnet.europe["eu-west-1"]

Importing resources into your state

Use the [.code]terraform import[.code] command to import resources into your Terraform state.

Resources that you have created through some other means than the current Terraform configuration can be imported. This allows you to start managing the resource using Terraform.

To import a resource you must add a corresponding resource block in your Terraform configuration. You must know the ID of the resource in the target platform (e.g., AWS) to import it successfully.

Below is an example of importing an AWS VPC resource with ID [.code]vpc-034315[.code] to the destination address [.code]aws_vpc.europe_network[.code]:

$ terraform import aws_vpc.europe_network vpc-034315

Removing a resource from your state

Use the [.code]terraform state rm[.code] command to remove a resource from your state. This is required if you no longer want to manage the resource using Terraform, but you want to keep the deployed resources. Pass the resource address of the resource you want to remove to this command.

Below is an example of removing an AWS subnet resource from your state:

$ terraform state rm aws_subnet.all[2]

Using HCL to update your state file 

You can manipulate the configuration inside your Terraform state using a declarative approach with HashiCorp Configuration Language (HCL).

The main benefit of using a declarative approach to manipulating your state is that you can preview what will happen using [.code]terraform plan[.code]. This makes it a better alternative to the CLI adjustments discussed above.

The general steps for manipulating your state file through HCL are:

1. Add one or more [.code]import[.code], [.code]moved[.code], and [.code]removed[.code] blocks that describe what you want to do
2. Run [.code]terraform plan[.code] to verify that the correct changes will take place
3. Run [.code]terraform apply[.code] to apply the changes
4. Remove the blocks you defined in step 1

Moving a resource in your state

To move a resource from one resource address to another you can use the [.code]moved[.code] block:

moved {
    from = aws_instance.old
    to = aws_instance.new
}

The [.code]from[.code] argument refers to the source resource address. This address should currently be listed in your state (i.e., you should see it listed by [.code]terraform state list[.code]) but should no longer be defined in your Terraform configuration.

The [.code]to[.code] argument refers to the destination resource address. This address should match a new resource block in your Terraform configuration.

Importing a resource into your state

You can import a resource into your Terraform state with the [.code]import[.code] block:

import {
    to = aws_instance.imported
    id = "i-xxxxxxx"
}

The [.code]id[.code] argument corresponds to the resource ID in the target platform (e.g., AWS). 

The [.code]to[.code] argument is the destination resource address to a resource block in your Terraform configuration. You can let Terraform generate the destination resource block for you. To do this, run [.code]terraform plan -generate-config-out=path[.code] with a path to a new file where Terraform outputs the resource block.

Removing a resource from your state

With Terraform 1.7 and later you can remove a resource from your Terraform state using the new [.code]removed[.code] block:

removed {
    from = aws_instance.old
    lifecycle {
        destroy = false
    }
}

The [.code]lifecycle[.code] block allows you to control whether Terraform should destroy the corresponding resource or not.

This block is useful if you have imported the resource to a different Terraform configuration or if you no longer want to manage the resource with Terraform.

Migrating your state file between different backends

You can migrate a state file between two state storage backends. You may want to do this for two reasons:

• You are moving all your storage, including Terraform state, to a different platform (e.g., your organization is switching from AWS to Azure).
• You are restructuring how you store state files in your current state storage backend and need to move the file to a new virtual directory or change its filename.

The steps to migrate the state file consists of:

1. Define the new backend block in your Terraform configuration
2. Run [.code]terraform init -migrate-state[.code] and follow the prompts

To see a simplified example of a state migration, consider the following backend definition:

terraform {
    backend "local" {
        path = "terraform.tfstate"
    }
}

We can migrate this backend to use another local path. Update the [.code]path[.code] argument accordingly:

terraform {
    backend "local" {
        path = "state/terraform.tfstate"
    }
}

Next, run [.code]terraform init -migrate-state[.code] to perform the migration:

$ terraform init -migrate-state
Initializing the backend…
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for the existing state in the backends.

Do you want to copy existing state to the new backend?
Enter a value: yes
…
Terraform has been successfully initialized!

The process of migrating between any types of backends is the same as the simple example outlined above. One difference will be that most remote backends require credentials to be configured.

Locking and unlocking the state file

State locking is a feature supported by some state backends. Read the documentation for the state backend you want to use to learn if it supports state locking.

Terraform will automatically lock the state file if the operation performed by Terraform could potentially write changes to the file. By locking the file, Terraform makes sure no other process will perform simultaneous writes to your state.

The process of locking the state file can't be performed manually. This is automatically handled by Terraform.

Terraform could fail during a [.code]terraform apply[.code]. If this happens, the state file lock could remain in place. In such a case, you need to force unlock the state file using the [.code]terraform force-unlock[.code] command.

You should use the unlock command carefully. If you unlock the state file while Terraform is writing changes to the file you could end up with a corrupt state file.

Handling state drift

If you update your resources outside of Terraform, your infrastructure will drift from its desired state. 

One common example of drift is when you change a resource attribute outside Terraform, e.g., by clicking a graphical interface.

When your infrastructure experiences drift, your Terraform state no longer matches the real world. There are a few ways you can handle this situation:

1. Reconcile your state with reality by running [.code]terraform plan -refresh-only -out=tfplan[.code] followed by [.code]terraform apply "tfplan"[.code]
2. Update your Terraform configuration to reflect the drifted state. You  can only do this if the drift represents a change you want to make.

If you detect unexpected drift in your infrastructure, you should investigate the root cause.

You can enable drift detection for your environments in your env0 organization. This feature automates drift detection across all infrastructure.

To set this up, go to the settings for the environment where you want to enable drift detection. Under ‘Drift Detection’ you can enable the feature and provide a cron expression to specify when you want drift detection to run:

The minimum drift detection interval is one hour.

If drift is detected, there will be a notification informing you of this on the environment overview page:

Instead of relying on someone noticing this information in the env0 portal, you can set up notifications for Slack, Microsoft Teams, email or webhooks for when a drift event occurs.

You can further drill down to learn more about what caused the drift. Follow the ‘Latest Drifts’ tab to see more:

In this example, one resource is reporting drifts. Click on the eye icon to analyze the drift cause for the S3 bucket:

All cloud events related to this resource are displayed in the list. In our case, the cause of the drift is clear—the resource was initially created through IaC, followed by a ClickOps update about an hour later. 

By viewing this sequence of events, you also gain insights into who made the change, which provides additional important context and can help prevent recurring issues and ensure proper drift reconciliation. After all, some ClickOps or script-induced changes might be necessary and should be added to the main configuration, instead of being reverted.

Best practices for the Terraform state file

Use remote state storage

The local backend is fine for testing and experimentation when you are not collaborating with others. For all other use-cases you should use a remote state storage backend.

The main benefit of remote state storage is that you can collaborate with others on the same Terraform configuration.

Use state encryption

Your Terraform state should be encrypted at rest and in-transit.

Ensure your state storage backend supports encryption. For the cloud provider backends (AWS S3, Azure storage, GCS) this is true, and you can even configure and use your own custom encryption keys.

Protect the file storage backend

You should ensure that you use any available protection mechanism for the storage backend you are using. For example, if you use an AWS S3 bucket as your storage backend then you should make sure to turn off public access for the S3 bucket. You should also require correct IAM permissions to access the S3 bucket.

Avoid imperative commands

Only use imperative Terraform CLI commands to modify your state file content as a last resort. These commands update the state file in-place, and they do not offer any undo capabilities.

Do not share the entire state file

If you need to share data from your Terraform configuration to other Terraform configurations, you should avoid sharing access to your entire state file.

The [.code]terraform_remote_state[.code] data source can be used to access a state file where you want to access some output data. However, by using this data source you are giving away access to your whole state file.

A better option for sharing data between Terraform configurations is the use of specific data sources for the data you need to share. An example is the [.code]aws_vpc[.code] data source, which you can use to read all attributes of an AWS VPC resource.

Take backups of your state file

It is crucial that you regularly backup your Terraform state in case you encounter an error and end up with a corrupt state file. Manually fixing a broken state file is difficult, if not downright impossible.

Ideally, you should backup before each [.code]terraform apply[.code] or any other command that manipulates the state.

Your backup files should be stored in a different storage location than the Terraform state file itself. 

If you are using an AWS S3 bucket as the storage backend, make sure to use a different S3 bucket for your backups. The other S3 bucket should be in another AWS region. This ensures that your state file backups are available even if your primary AWS region is unavailable.

Final Thoughts

The Terraform state file contains the known state of the world according to Terraform. It records all attributes and secrets related to all the resources of your Terraform configuration.

You can make changes to the state file by updating your Terraform configuration and applying the change. If you need to refactor your state you can use the Terraform CLI or specific HCL blocks.

It's important to protect the state file, given what it contains. This includes encrypting the state file in-transit and at rest, restricting who can read and write to the file, and to regularly backup the state file.

Frequently asked questions

Q. What is a Terraform state file?

The Terraform state file is a JSON file where Terraform records a snapshot of all the resources that are part of the current Terraform configuration. All attributes are recorded for each resource in the state. This includes secret values.

Q. What if my Terraform state file is deleted?

If your Terraform state file is deleted and you do not have any backups of it available, then there is still hope to get back to a working state.

Follow these steps:

1. Define [.code]import[.code] blocks for all the resources in your Terraform configuration
2. For each [.code]import[.code] block, configure the [.code]to[.code] argument with the destination resource address of the imported resource and the [.code]id[.code] argument with the resource ID for the resource from the target platform
3. Run [.code]terraform plan[.code] to verify that the import blocks are correctly configured
4. Run [.code]terraform apply[.code] to perform the resource imports
5. Delete the [.code]import[.code] blocks from your code; they are no longer needed

Q. My Terraform state file is locked, how do I unlock it?

Some state backends support state file locking. If Terraform encounters an error during the plan or apply phases, it is possible that the state file remains in a locked state. If this is the case, Terraform won't be able to use the state file until it is unlocked.

You can force unlock the Terraform state file using the following command:

$ terraform force-unlock LOCK_ID

How you find the lock ID for the command depends on what backend you are using. Read the corresponding documentation for the backend to learn where to find the lock ID.

Q. How can I enable encryption for my Terraform state file?

You should select a remote state backend that supports:

1. Transport Layer Security (TLS) encryption in-transit
2. Encryption at rest

Examples of backends that have these features are AWS S3, Azure storage, and Google Cloud Storage.

With OpenTofu, you can enable encryption at rest for your plan and state files. This is in addition to any encryption that the state backend uses. You configure encryption settings in the [.code]terraform[.code] block. A minimal example of configuring encryption with AWS KMS is shown below:

terraform {
  encryption {
    key_provider "aws_kms" "example" {
      kms_key_id = "131313b8-21ed-49b7-ac15-77ab9dd4addf"
      region = "eu-west-1"
      key_spec = "AWS_256"
    }
}

See the OpenTofu documentation for more options to configure state and plan encryption.