Part II - Remote Terraform — Learn Terraform The Hard Way

Part II - Remote Terraform

In this section you will use Terraform to create resources external to your local host machine. Building on your knowledge from Part I, you’ll see how to create resources on a given provider. That might be a cloud-based provider (we’ll be using AWS in our examples) or any other kind of resource accessible via an API that’s supported by Terraform.

In this part you will cover:

Terraform providers
Implicit and explicit resource dependencies
Graphing Terraform module dependencies
Importing pre-existing Terraform resources

This will complete the ‘basics’ part of the course, before you move on to Part III, where you start to cover more advanced Terraform concepts.

Managing Remote Resources

In this first section of Part II you’re going to create your first piece of cloud infrastructure using Terraform. I’ve chosen AWS for my examples as that’s the most popular cloud platform today.

This won’t cost you a penny (nor will any other section in this book), as you are going to create resources on AWS that are (as of writing) completely free to create (even if you’re not on the AWS ‘Free Tier’). If this changes, let the author know as soon as possible!

In this section you’re going to cover:

Remote resources
Providers
Refreshing resources

How Important is this Section?

Terraform was created specifically to manage remote resources, so this section is essential.

Creating A Remote VPC

First, type this out and think about what’s new as you’re doing it. As you come across something you don’t understand you might want to quickly research it (but don’t worry if your researches don’t make too much sense, as we cover the ground as we go):

1 $ export AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY
2 $ export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID

 3 $ mkdir -p ltthw_aws_vpc
 4 $ cd ltthw_aws_vpc
 5 $ cat > hello_remote.tf << EOF
 6 > provider "aws" {
 7 >   region     = "us-east-1"
 8 > }
 9 > resource "aws_vpc" "main" {
10 >   cidr_block = "10.0.0.0/16"
11 > }
12 > EOF

You should have noticed that you created another Terraform module similar to previous ones. This time, in addition to the usual resource stanza, you created a provider stanza. Also, we introduced a new resource type: aws_vpc.

aws_vpc is an example of a ‘remote resource’. It’s a resource managed remotely instead of locally on your machine. In this case, aws_vpc is a ‘virtual private cloud’ on AWS. This provides you a network space that you have control over, and in which you can add other AWS resources (such as virtual machines with AWS’s EC2 service, or databases with AWS’s RDS service). The cidr_block allocates a set of private IP addresses to this VPC.

A provider is the part of Terraform that interacts with any external APIs to make resources available to you. There are many mainstream cloud providers including:

AWS
Azure
OpenStack

In addition, there are many providers you might not expect to see, such as:

GitHub
Kubernetes
MySQL
Mailgun

These providers provision resources on those services. For example, the GitHub provider allows you to add membership of users to teams, configure branches, set up teams, and many other resources on GitHub.

Again, you may want to spend a little time researching these providers and what they do at this point, especially if you have a use case in mind for Terraform.

Now run the Terraform module to get your VPC created:

13 $ terraform init
14 $ terraform apply

Running terraform apply will give you output describing the resource creation that looks like this:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_vpc.main will be created
  + resource "aws_vpc" "main" {
      + arn                              = (known after apply)
      + assign_generated_ipv6_cidr_block = false
      + cidr_block                       = "10.0.0.0/16"
      + default_network_acl_id           = (known after apply)
      + default_route_table_id           = (known after apply)
      + default_security_group_id        = (known after apply)
      + dhcp_options_id                  = (known after apply)
      + enable_classiclink               = (known after apply)
      + enable_classiclink_dns_support   = (known after apply)
      + enable_dns_hostnames             = (known after apply)
      + enable_dns_support               = true
      + id                               = (known after apply)
      + instance_tenancy                 = "default"
      + ipv6_association_id              = (known after apply)
      + ipv6_cidr_block                  = (known after apply)
      + main_route_table_id              = (known after apply)
      + owner_id                         = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_vpc.main: Creating...
aws_vpc.main: Still creating... [10s elapsed]
aws_vpc.main: Creation complete after 13s [id=vpc-07053dc65c34e248f]

The ‘(known after apply)’ items are a useful placeholder that indicate that the value will be given to you by the provider of the resource once the resource is created. cidr_block was given by us, and the Terraform provider code given to us for AWS sets a default for assign_generated_ipv6_cidr_block and enable_dns_support of false and true respectively.

At this point, you can look at what Terraform thinks the state is with a terraform show command:

15 $ terraform show terraform.tfstate

# aws_vpc.main:
resource "aws_vpc" "main" {
    arn = "arn:aws:ec2:us-east-1:701780912049:vpc/vpc-07053dc65c34e248f"
    assign_generated_ipv6_cidr_block = false
    cidr_block                       = "10.0.0.0/16"
    default_network_acl_id           = "acl-0190de7297f62c4ae"
    default_route_table_id           = "rtb-0f6085dd75274c332"
    default_security_group_id        = "sg-01822fa63ddffc7ff"
    dhcp_options_id                  = "dopt-5d671426"
    enable_classiclink               = false
    enable_classiclink_dns_support   = false
    enable_dns_hostnames             = false
    enable_dns_support               = true
    id                               = "vpc-07053dc65c34e248f"
    instance_tenancy                 = "default"
    main_route_table_id              = "rtb-0f6085dd75274c332"
    owner_id                         = "701780912049"
}

Notice how all the items previously marked as (known after apply) are now filled out.

Changing A Resource Outside Terraform

Now you’re going to change the resource you just created without using Terraform. To do this you’ll need to log into your AWS web console, and find the just-created VPC resource.

Make sure you are in the correct region (which will be us-east-1, unless you changed it above). If you can’t find it, you may want to try visiting

https://console.aws.amazon.com/vpc/

and picking the VPC with the ID you saw in the terraform show output.

Once you’ve found your VPC, give it a name by hand in the console.

To do this, go to the ‘Tags’ tab at the bottom of the VPC page and add a tag called ‘Name’ with a name of your choice (eg ltthwvpc).

Now you are in a state where Terraform has one view of the resource’s state (it has no name as far as Terraform is concerned), while the provider of that resource (AWS) has another (the name you just gave it in the console).

Terraform is therefore in an inconsistent state with the resource. We saw with local files before that this was a problem for Terraform. However, there is a command that can help:

16 $ terraform refresh

Read the output carefully. What did that do?

Now if you look again at your state file, you will see it has the name you gave it:

17 $ terraform show terraform.tfstate

Finally, you’re going to destroy the resource you just created.

18 $ terraform destroy -auto-approve

Now go back to the AWS console to confirm that it has gone.

Cleanup

To clean up what you just did, run:

19 $ cd ..
20 $ rm -rf ltthw_aws_vpc

What You Learned

In this section you created your first cloud provider resource using Terraform, updated it by hand, and then used terraform refresh to make the state consistent again.

What Next?

In the next section you’re going to look at how Terraform manages dependencies for you, and how you can also specify dependencies for it to manage for you.

Exercises

1) Go back to Part I and use ‘refresh’ against the files created to embed the knowledge of how the state file and refreshing works.

2) Go through all the providers documented on the Terraform docs and find out what each one does. This will give you a good idea of the extent of Terraform’s current capabilities.

Dependencies

In this section you’re going to get to grips with dependencies in Terraform. One of Terraform’s key selling points is that it can manage dependencies between different parts of your infrastructure.

You’ll cover how Terraform:

Can create implicit dependencies for you
Can be used to define explicit dependencies
Manages targeted destruction of resources

These dependencies are the bane of any infrastructure manager’s life. If you switch off that VM who will complain? What about that Load Balancer?

Stories of ‘just switching off servers to see who complains’ as a way of managing unknown dependencies are incredibly common in the IT industry. Terraform can help manage that problem by codifying and tracking resource dependencies, but it’s important to remember that it does not do so by magic, and that your application dependencies may still need to be mapped out by whoever’s in charge of them.

How Important is this Section?

Terraform scripts that are in any way sophisticated will use dependencies, both explicit and implicit, so this section is important if you plan to use scripts that go beyond the basics.

Implicit Dependencies

First, set up your credentials as you did the previous section:

1 $ export AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY
2 $ export AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID

Create another folder with a new Terraform module:

3 $ mkdir -p ltthw_dependencies
4 $ cd ltthw_dependencies
5 $ cat > dependencies.tf << EOF
6 > provider "aws" {
7 >   region     = "us-east-1"
8 > }
9 > EOF

So far, so familiar. You’ve created a Terraform module with an AWS provider defined.

Now you’re going to add two resources to that file. The first is an AWS VPC, similar to the one you created before.

This time you’re going to give it a name directly in the Terraform module (ltthw-vpc) rather than doing so manually in the AWS console:

10 $ cat >> dependencies.tf << EOF
11 > resource "aws_vpc" "ltthw-vpc" {
12 >   cidr_block = "10.0.0.0/16"
13 >   tags = {
14 >     Name = "ltthw-vpc"
15 >   }
16 > }
17 > EOF

Now the second resource, which is of the type aws_subnet. This subnet is attached to the VPC, and is linked by the VPC id.

That raises a problem: before you create the VPC, you don’t know its ID (it’s given to you by AWS on creation). Yet you want to refer to it so that you can link the subnet to the VPC via the vpc_id.

18 $ cat >> dependencies.tf << 'EOF'
19 > resource "aws_subnet" "ltthw-vpc-subnet" {
20 >   vpc_id = aws_vpc.ltthw-vpc.id
21 >   cidr_block = aws_vpc.ltthw-vpc.cidr_block
22 > }
23 > EOF

In Terraform, every resource has attributes that you can reference with the syntax TYPE.NAME.ATTRIBUTE. Remembering the three-letter acronym ‘TNA’ might help you remember this pattern of referencing.

24 $ terraform init
25 $ terraform plan
26 $ terraform apply -auto-approve

Next you’re going to run a terraform plan -destroy command with an extra flag to specify that you only want to destroy the VPC.

27 $ terraform plan -destroy -target=aws_vpc.ltthw-vpc

Refreshing Terraform state in-memory prior to plan… The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage.

Read the full output on your terminal carefully.

You didn’t destroy anything there, but you did see that the subnet and the VPC were both planned for destruction. This is despite the fact that you specified -target=aws_vpc.ltthw-vpc.

This is because there is an implicit dependency between the subnet and the VPC. This dependency is understood and managed by Terraform for you. It’s part of the logic of the Terraform AWS provider.

If you were to run terraform apply at this point, it would delete both items. To cancel the effect of this plan to destroy, rerun a plan without the -destroy flag. This puts the terraform plan back to a state where nothing needs changing.

28 $ terraform plan

Try the destruction above again, but this time use the flag: -target=aws_vpc.ltthw-vpc-subnet to target only the subnet, not the vpc:

29 $ terraform plan -destroy -target=aws_subnet.ltthw-vpc-subnet

Now you can see that this time only the subnet is marked for destruction. It was targeted, and no other resource was dependent on it.

Now destroy everything you have created so far:

30 $ terraform destroy -auto-approve

Order is Unimportant

Now you’re going to do the same thing, but this time you’re going to reverse the order of declaration. The subnet will be declared first, and then the VPC. What do you think will happen?

31 $ cat > dependencies.tf << 'EOF'
32 > provider "aws" {
33 >   access_key = "YOURACCESSKEY"
34 >   secret_key = "YOURSECRETKEY"
35 >   region     = "us-east-1"
36 > }
37 > resource "aws_subnet" "ltthw-vpc-subnet" {
38 >   vpc_id = aws_vpc.ltthw-vpc.id
39 >   cidr_block = aws_vpc.ltthw-vpc.cidr_block
40 > }
41 > resource "aws_vpc" "ltthw-vpc" {
42 >   cidr_block = "10.0.0.0/16"
43 >   tags = {
44 >     Name = "ltthw-vpc"
45 >   }
46 > }
47 > EOF
48 $ terraform init
49 $ terraform plan
50 $ terraform apply -auto-approve

What happened? Was it what you expected?

This should show you that the ordering of resources within a module is unimportant. Terraform takes the resources declared and creates a dependency graph that allows it to know what needs to be done on each run. In the next section you’ll look at how to visualize this graph.

If you’ve written a query in SQL that queries a database, Terraform works a bit like that: with a SQL query you specify the data you want returned and its constraints. How it gets that data out from the database and returned to you is up to the query parser, optimiser and database management system.

Or, if you’ve compiled a program you might already know that a program is translated to machine code, which might itself process things in a different order to the order you told it to run in. Even the chip the code runs on might order its internal instructions differently to the way you declared it.

The point is that Terraform allows you to specify the infrastructure you want, and it takes care of the detail of what needs to happen for that to be provisioned, updated, or destroyed. Dependencies are one aspect to this; ‘(known after apply)’ data, and other defaults, are another.

Destroy everything again:

51 $ terraform destroy -auto-approve

Explicit Dependencies

You just created a resource that Terraform knew was dependent on another (the VPC subnet). It knew it was dependent because the logic within the provider code ‘knows’ that.

But what about dependencies that are not part of the structure of resources themselves, but part of the logic of your application?

For example, you might have a dependency in your application between an EC2 virtual machine on AWS and a load balancer on Azure. This would not be captured automatically by Terraform, but fortunately you can specify it.

Type this scenario out and try and figure out what’s different this time:

52 $ cat > dependencies.tf << 'EOF'
53 > provider "aws" {
54 >   access_key = "YOURACCESSKEY"
55 >   secret_key = "YOURSECRETKEY"
56 >   region     = "us-east-1"
57 > }
58 > resource "aws_vpc" "ltthw-vpc" {
59 >   cidr_block = "10.0.0.0/16"
60 >   tags = {
61 >     Name = "ltthw-vpc"
62 >   }
63 > }
64 > resource "local_file" "hello_local_file" {
65 >     content    = "Hello terraform local!"
66 >     filename   = "${path.module}/hello_local.txt"
67 >     depends_on = [aws_vpc.ltthw-vpc]
68 > }
69 > EOF
70 $ terraform init
71 $ terraform plan
72 $ terraform apply -auto-approve

Now if you plan to destroy the VPC, what do you think will happen?

73 $ terraform plan -destroy -target=aws_vpc.ltthw-vpc

Did that do what you expected? Can you explain why both were planned for destruction?

What about this one? Try and predict what it will plan to do:

74 $ terraform plan -destroy -target=local_file.hello_local_file

Did you get it right? If so, you now know how to set explicit dependencies.

Now destroy what you created:

75 $ terraform destroy -auto-approve

In recent versions of Terraform, it warns you that -target is not for routine use, and should only be used in when recovering from a problem, or when directed to by Terraform in an error message.