from Hacker News

Terraform Gotchas and How We Work Around Them

by kalmar on 6/12/17, 6:43 PM with 80 comments

  • by luhn on 6/12/17, 8:47 PM

    > Always write your plan -out, and apply that plan

    I have in my dotfiles:

        alias tfplan='terraform plan -out=.tfplan -refresh=false'
    alias tffreshplan='terraform plan -out=.tfplan'
    alias tfapply='terraform apply .tfplan; rm .tfplan'
    That way I never accidentally `terraform apply` without creating a plan first. I also have it not refresh the state by default, which is mostly unnecessary and speeds up the planning significantly.

  • by 7ewis on 6/12/17, 10:26 PM

    Terraform has interested me for a while, and I've been meaning to give it a try, but haven't had a chance just yet.

    From what I have seen so far though, there isn't really that much difference/benefit over CloudFormation. We currently have 95% of our resources in AWS with about 4% in Azure, and 1% in Google Cloud. It's great that Terraform is 'mulit-cloud' but it still seems like you have to write .tf's catered to each cloud, you can't just lift and shift to another cloud by copying and pasting a file?

    People say the 'plan' feature is one of the advantages over CFN, but as far as I can tell, CFN now offers the same feature... it tells you what's going to change when you upload a new stack.

    I sound like a CFN advocate now, but I genuinely don't have that much experience with it, and really do want to give Terraform a chance. Convince me?

    Oh, and since CFN started supporing YAML it looks easier to write too

  • by lobster_johnson on 6/13/17, 1:50 AM

    I've wanted, and tried and failed, to adopt Terraform several times now. What always gets in my way is that we already have all our infrastructure in place, and Terraform's import capabilities are too limited.

    For example, the last time I used it, a few months ago, it was not able to import almost any of our Google Cloud stuff, and I discovered that import support is only provided for some resources. There's a third-party tool called Terraforming, but it apparently only works with AWS.

    I'm quite disheartened that the world is lagging this far behind. The only competitor I've found is Salt, and I found its orchestration support to be a bit of a mess. And just as with Terraform, the code is constantly lagging behind the providers.

    The one provider I'd have expected to be on the forefront of orchestration is Google, and in a different multiverse their engineers are swarming around Terraform to make sure it has top-notch, official, first-class support, but alas, not in this one.

    Are there any competitors that provide a smoother experience?

  • by ian_d on 6/13/17, 1:27 AM

    I've been using terraform for a couple of months now (love it), but honestly our biggest pain was just project organization. It looks like a lot of people make a file per-resource type (elb.tf, ec2.tf, rds.tf) but we thought that would be a lot of bloat. We opted to have a file per system (dev_db.tf, dev_ecs_asg.tf, dev_haproxy.tf, etc) and everything related to that particular system is contained in a single file (security groups, dns entries, roles/profiles, etc). But it's still in one flat directory per environment. (I know tf has introduced environments, but we haven't switched over yet.)

    I know you can hack this together with modules, but it seems like environment/project organization would be easier if terraform just recursed subdirectories. Right? I've seen a couple of issues for it, but I don't believe I've seen a concrete reason why it's a no-go.

  • by philsnow on 6/12/17, 10:44 PM

    There's another similar issue with how EC2 security group rules are encoded: you can encode them either as ingress/egress stanzas on an "aws_security_group" resource, or you can attach rules to a security group resource with separate "aws_security_group_rule". You can't mix the two approaches on a single security group resource.

    We adopted the ingress/egress stanza on security group resource approach.

    If we ever wanted to change to the other approach (as described in the article), I don't think I would do state surgery by hand or even use "terraform state mv". I would:

        1. change terraforming to generate .tf files and tfstates the way I want
    2. remove the security groups from my config and my state
    3. use terraforming to regenerate the .tf files and tfstate

  • by pavement on 6/12/17, 9:01 PM

    Oh, geeze. This is about: terraform.io

  • by johnmarcus on 6/12/17, 10:15 PM

    I absolutely can't stand how destructive terraform is by nature. We have switched to Ansible, which has an excellent AWS module, and never looked back.

  • by iofiiiiiiiii on 6/13/17, 10:08 AM

    I am just now implementing Packer and Vagrant in our devops workflows. Terraform is next on the list.

    So far, it leaves me rather anxious - Packer and Vagrant appear to offer the bare minimum of usable functionality, with any advanced scenario bumping into (sometimes intentional) walls.

    For example, it takes me 15-20 minutes to transfer a 50 MB file to a Windows VM being created by Packer. The GitHub issue, filed nearly 2 years ago, is closed with a comment that this is by design: https://github.com/hashicorp/packer/issues/2648#issuecomment...

    Yet there is a PowerShell command that uses the same communication mechanism that can somehow do it in a matter of seconds. Of course, I cannot use this PowerShell command because Packer does not give me a variable with a machine's IP address because... it is improper somehow? https://github.com/hashicorp/packer/issues/4993

    What the hell, Hashicorp...

    I have a list of 10+ issues I have found so far and I am only starting to use these tools. From the activity in GitHub, they seem to be abandonware.

    Maybe if I submitted PRs they might be accepted (then again, maybe not: https://github.com/hashicorp/packer/pulls) but I expect more from software than just accepting PRs - I expect its authors to actually develop it and to show an interest in improving it.

    There is unfortunately nothing better out there. I admit, I am forced to use these products even though I do not find them satisfactory and the authors do not seem helpful.

    If I had to start all over again with my current knowledge, I might perhaps just write my own scripting and skip Packer/Vagrant altogether. The value they offer with VM management comes with the downside of being left in the mud and having the system work against you when you try something nontrivial.

    I am scared of what I will find when I touch Terraform. As I write this, I think I will first see whether I can just script it manually.

  • by kalmar on 6/12/17, 7:56 PM

    Hey author here! Happy to answer any questions etc :-)

  • by nunez on 6/13/17, 12:55 AM

    here's something i got bit by more recently re: terraform plan -out and tooling using Terraform's Golang API.

    Handling package dependencies with Go is not straighforward. There are several ways of doing it, and none are native to Golang.

    Additionally, Go doesn't support getting versions of packages by tag or branch.

    This bit me hard when I tried to update Palantir's TFJSON utility (turns tfplan binaries into json) so I could do unit testing of my Terraform plans with rspec.

    The utility depended on v0.7.4 of terraform, but Terraform maintains a plan format constant that defines which plans can be used by what versions. They changed the plan format between 0.7.4 and 0.9.8 without bumping that constant, so when I tried running tfjson against plans created by the latter version, I got a weird non-matching datatype error that took a while to figure out. (I eventually had to vimdiff the hex outputs of plans created by both versions to figure that out.)

    Additionally, HashiCorp made a significant change to the way they handled providers between 0.9.8 and 0.10.0 that justified them to bump the plab format version AGAIN. The catch: 0.10.0 isn't released yet, despite that being the code in their master branch.

    I figured that updating tfjson's vendored terraform library to 0.9.8 would solve it. I first did a go get to fetch the latest TF codebase and used gvt to vendor it. That's when I discovered that plans generated by 0.9.8 are no longer compatible. After discovering that go get can't fetch packages by tag (Hashicorp tags their release commita) because Google believes in stable HEADs, I had to find a tool that could support fetching packages by tags. Govendor did that, so I used that.

    It takes FOREVER to fetch all of the subpackages used by terraform. I couldn't do it during a three hour flight. Rubygems has its problems, but fetching deps isn't one of them. And even when I thought I fetched the entire source tree at v0.9.8, I would still get errors about missing types or missing packages.

    I'm hopeful that I'll eventually find a solution, but it's a dog compared to using Gemfile.lock.

  • by mental_ on 6/12/17, 10:19 PM

    I thought terragrunt was a must have for that kind of deployment.

  • by Artemis2 on 6/13/17, 7:30 AM

    This sums up our experience with Terraform perfectly:

    > Most outages are caused by human error and configuration changes, and applying Terraform changes is a terrifying mix of the two.

    Terraform is a great tool nonetheless. Just like Heap, we have code reviews for the configuration itself, and a CI pipeline for validating it. This pipeline is quite superficial (`terraform validate` mostly does syntax checking), so we are too working on using centralized state to `terraform plan` for reviews.

  • by sevagh on 6/12/17, 8:38 PM

    >Terraform state surgery

    Did you try to use `terraform state mv`? I've found that command useful (albeit for much less than thousands of resources).

  • by nategri on 6/12/17, 8:16 PM

    So... ahem what other idiots came here expecting a post about troubleshooting Martian habitability?

    ::Sulks off dejectedly::