Terraform Modules and Remote State for Real Projects

What are Terraform modules?

Terraform modules are reusable collections of configuration files that create related infrastructure resources. They help teams avoid duplicated code, apply consistent standards and keep large projects easier to test and maintain.

Every Terraform configuration is technically a module. The directory where you run Terraform is the root module, while modules called from that directory are child modules.

For example, an AWS project might contain separate modules for:

• VPC networking
• Security groups
• Application Load Balancers
• EC2 Auto Scaling groups
• RDS databases
• CloudWatch alarms

A useful diagram-in-words is:

Root module for production
  |
  |-- calls VPC module
  |-- calls security group module
  |-- passes subnet IDs to application module
  |-- passes VPC ID to database module
  |
  `-- records all managed resources in remote state

A module should normally represent a logical infrastructure component, not one file or one resource. A vpc module may create the VPC, subnets, route tables and gateways because those resources form one networking unit.

Why should real projects use modules?

Modules provide a controlled interface between reusable infrastructure code and environment-specific values. A team can use the same tested module for development, staging and production while supplying different CIDR blocks, instance types or scaling limits.

The main benefits are:

Do not create a module only to reduce the number of lines in the root directory. Create one when the component has a clear purpose, useful inputs, meaningful outputs and a reasonable chance of reuse.

Learners building these patterns in CI/CD labs can explore the practical syllabus of our AWS DevOps course. For broader AWS architecture decisions, see the AWS Solutions Architect course.

How should a Terraform project be structured?

A practical project separates reusable modules from environment-specific root modules. Each environment should have its own backend state and variable values so that a development deployment cannot accidentally overwrite production state.

Here is a suitable structure for a small AWS platform:

terraform-project/
|-- modules/
|   |-- vpc/
|   |   |-- main.tf
|   |   |-- variables.tf
|   |   |-- outputs.tf
|   |   `-- versions.tf
|   `-- web-security-group/
|       |-- main.tf
|       |-- variables.tf
|       `-- outputs.tf
|-- environments/
|   |-- dev/
|   |   |-- backend.tf
|   |   |-- main.tf
|   |   |-- providers.tf
|   |   |-- terraform.tfvars
|   |   `-- variables.tf
|   `-- prod/
|       |-- backend.tf
|       |-- main.tf
|       |-- providers.tf
|       |-- terraform.tfvars
|       `-- variables.tf
`-- README.md

In this model, environments/dev and environments/prod are independent root modules. They can call the same child modules but use different remote state keys and input values.

Example child VPC module

The module declares its AWS provider requirement in modules/vpc/versions.tf:

terraform {
  required_version = ">= 1.10.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

Its variables define the module interface:

variable "name" {
  description = "Name prefix for VPC resources"
  type        = string
}

variable "vpc_cidr" {
  description = "IPv4 CIDR block for the VPC"
  type        = string
}

variable "public_subnets" {
  description = "Map of Availability Zones to public subnet CIDRs"
  type        = map(string)
}

variable "tags" {
  description = "Common resource tags"
  type        = map(string)
  default     = {}
}

The module creates resources in main.tf:

resource "aws_vpc" "this" {
  cidr_block           = var.vpc_cidr
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = merge(var.tags, {
    Name = "${var.name}-vpc"
  })
}

resource "aws_subnet" "public" {
  for_each = var.public_subnets

  vpc_id                  = aws_vpc.this.id
  availability_zone       = each.key
  cidr_block              = each.value
  map_public_ip_on_launch = true

  tags = merge(var.tags, {
    Name = "${var.name}-public-${each.key}"
  })
}

Useful values are exposed through outputs.tf:

output "vpc_id" {
  description = "ID of the VPC"
  value       = aws_vpc.this.id
}

output "public_subnet_ids" {
  description = "Public subnet IDs indexed by Availability Zone"
  value       = { for az, subnet in aws_subnet.public : az => subnet.id }
}

Notice that the child module does not contain an AWS provider block with a region or credentials. The root module configures the provider and passes that configuration to child modules automatically.

How does a root module call a child module?

A root module calls a local child module with a module block and passes values through its declared variables. Outputs from one module can then become inputs to another module, creating explicit and readable dependencies.

An environment root module could contain:

module "vpc" {
  source = "../../modules/vpc"

  name     = "shop-dev"
  vpc_cidr = "10.20.0.0/16"

  public_subnets = {
    "ap-south-1a" = "10.20.10.0/24"
    "ap-south-1b" = "10.20.20.0/24"
  }

  tags = {
    Environment = "dev"
    ManagedBy   = "Terraform"
  }
}

module "web_security_group" {
  source = "../../modules/web-security-group"

  name   = "shop-dev-web"
  vpc_id = module.vpc.vpc_id
}

After adding or changing a module source, initialise the working directory:

cd environments/dev
terraform init
terraform fmt -recursive
terraform validate
terraform plan

terraform init downloads providers and installs modules. terraform validate checks configuration syntax and internal references, while terraform plan compares the configuration with the current state and provider APIs.

For shared modules stored in Git, pin a tag or commit instead of always using a changing branch:

module "vpc" {
  source = "git::https://example.com/platform/terraform-aws-vpc.git?ref=v1.2.0"

  name     = "shop-prod"
  vpc_cidr = "10.30.0.0/16"
}

What is Terraform remote state?

Terraform state maps configuration addresses, such as aws_vpc.this, to real infrastructure objects and their attributes. Remote state stores that mapping in a shared backend rather than leaving terraform.tfstate on one engineer's computer.

Local state creates operational problems. It can be lost, copied inconsistently or changed by two users without coordination. State files may also contain sensitive values returned by providers, even when an output is marked sensitive.

For AWS projects, the S3 backend provides central storage. Modern Terraform versions can use an S3 lock file to prevent concurrent state changes, while S3 versioning helps recover an earlier object version after an operational mistake.

A diagram-in-words looks like this:

Engineer or CI runner
  |
  |-- authenticates to AWS
  |-- acquires state lock
  |-- reads state from S3
  |-- compares state, code and AWS resources
  |-- applies approved changes
  `-- writes updated state and releases lock

Remote state is not a replacement for reviewing plans or controlling AWS permissions. Backend access should be limited to the engineers and CI roles that manage the relevant environment.

How do you configure an S3 backend safely?

Create the backend infrastructure before configuring Terraform to use it. Enable S3 versioning and encryption, block public access, restrict access with IAM and use a different state key for each independently managed root module.

A production backend declaration can be placed in backend.tf:

terraform {
  backend "s3" {
    bucket       = "company-terraform-state-prod"
    key          = "shop/prod/network/terraform.tfstate"
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true
  }
}

Backend blocks cannot use normal input variables or references such as var.environment. This is because Terraform must initialise the backend before evaluating the rest of the configuration.

Do not hard-code AWS access keys in this file. Use an IAM role, AWS IAM Identity Center, environment variables or an approved CI/CD identity mechanism.

The S3 bucket itself must exist first. A common bootstrap process is:

1. Create a separate bootstrap root module using local state.
2. Create the S3 state bucket with versioning, encryption and public access blocking.
3. Apply the bootstrap configuration.
4. Add the S3 backend to the main root module.
5. Run terraform init -migrate-state if existing local state must be moved.

terraform init -migrate-state
terraform state list
terraform plan

Always inspect the state list and plan after migration. A plan proposing unexpected recreation may indicate that the wrong state file, backend key, AWS account or region is being used.

For related design controls around access, reliability and cost, read AWS Well-Architected basics.

How should environments and state files be separated?

Use a separate state file for each environment and for infrastructure components that have different ownership or change cycles. This limits the effect of mistakes and prevents every deployment from locking one large state file.

A useful state layout is:

S3 bucket
|-- shop/dev/network/terraform.tfstate
|-- shop/dev/application/terraform.tfstate
|-- shop/prod/network/terraform.tfstate
`-- shop/prod/application/terraform.tfstate

Terraform CLI workspaces change which state instance a configuration uses, but they do not provide IAM isolation. For long-lived production environments, separate root configurations and AWS accounts are usually clearer operational boundaries.

Avoid reading many values through terraform_remote_state. It gives the consumer access to state snapshots, which can contain more data than the exposed outputs suggest. Prefer stable interfaces such as AWS Systems Manager Parameter Store, DNS records or service discovery when independently managed systems need to exchange values.

How do you troubleshoot module and remote state errors?

Start by identifying whether the failure concerns module installation, configuration validation, backend access, locking or a mismatch between state and AWS. Do not delete state or lock files until you understand who owns them and whether another apply is active.

Module not installed

Typical message:

Error: Module not installed

Run:

terraform init

If a Git module changed versions, run terraform init -upgrade. Confirm that the source path is correct relative to the root module, not relative to the current shell location.

Unsupported argument

This means the caller supplied an input that the child module has not declared, or the installed module version does not support it.

terraform providers
grep -R 'variable "vpc_cidr"' ../../modules/vpc

Check the module's variables.tf, spelling and selected version. Do not add unused variables merely to silence the error.

Access denied for the S3 backend

Confirm the active AWS identity and bucket region:

aws sts get-caller-identity
aws s3api get-bucket-location --bucket company-terraform-state-prod

The identity needs appropriate S3 permissions for the state object and locking operations. Also inspect bucket policies, permission boundaries, service control policies and KMS permissions if customer-managed encryption keys are used.

State lock cannot be acquired

First check whether another pipeline or engineer is running terraform apply. If the original operation has ended and the lock is genuinely stale, record the lock ID and use force unlock carefully:

terraform force-unlock LOCK_ID

Never force-unlock an active operation. Two writers can produce conflicting infrastructure changes and an unreliable state snapshot.

Resources exist but Terraform wants to create them

Verify the backend bucket, key, AWS account, region and workspace:

terraform workspace show
terraform state list
aws sts get-caller-identity

If a resource exists outside the selected state, import it only after confirming the correct resource address:

terraform import 'module.vpc.aws_vpc.this' vpc-0123456789abcdef0
terraform plan

Import connects an existing object to state; it does not automatically create a complete maintainable configuration.

Practical summary

Terraform modules give infrastructure components clear inputs, resources and outputs. Real projects should combine reusable child modules with environment-specific root modules, pinned dependencies, separate state boundaries and controlled S3 backend access.

Before applying a project, verify these points:

• Module responsibilities are clear and not excessively small.
• Providers are configured in the root module.
• Shared module versions are pinned.
• Development and production use different state keys.
• S3 versioning, encryption, locking and access controls are enabled.
• Credentials and secret values are not committed to Git.
• CI runs formatting, validation and plan checks before apply.
• State migrations and imports are followed by a reviewed plan.

Reviewed by Network Rhinos DevOps trainers.

To practise Terraform modules, remote state, AWS automation and CI/CD workflows in guided labs, view the AWS DevOps course batch and enquiry details.

Related reading: AWS DevOps Project Ideas That Impress Interviewers