Writing and playing with custom Terraform Providers

i’ve been digging deeper on terraform. it’s something i’ve tinkered with in the past, but i’ve not really sat down to really use it in anger and try and tie a large project together.

so, i picked something that i recently was doing manually: the configuration of a demo of vault with the gcp backend. right now i was doing most of the steps for that manually, and i wanted to automate the entire process, and have a fully reproducible demo environment created in gcp. that’s a larger project i’m going to blog about later, but for now i’m going to concentrate on one thing that came up that led me down a rabbit whole of creating a provider.

one of the things that i realised was that as part of my setup for the demo, i had a vault instance available on the public internet, before it got initialized. the initialization step is extremely quick, but i always feel a little paranoid that someone might mess with it whilst it’s getting ready.

no problem: i can write some terraform code to configure a gcp firewall:

right now, this is accepting 0.0.0.0, so anyone on the internet can access it. how can i make it use my current ip address?

but how do i do get my current ip to load in in the terraform code? i went digging and found a thread that suggested using a datasource resource:

which worked like a charm, as long as you check the response from the url and make sure it’s a valid string, no newlines and such.

but, i thought this would give me the perfect chance to play with some golang and try writing a provider myself: a datasource that returns the current ip of the system. getting started

as someone who’s mainly dabbled in ruby and java, the go dependency system is super weird. eventually it seemed to boil down to adding this to my dotfiles:

then my repo i’m working on being inside the gopath folder:

now here i cheated a bit. i’d already played with the writing custom providers doc earlier this year, and got the demo setup working fine. but i realised i could get setup a lot faster if i re-used a lot of the existing http provider work. ultimately the data source would look very similar, and it would already have a lot of the more difficult stuff pre-done, such as mocking out the calls to a web address. so i used the original http provider as a base and added new things into it. getting an external address.

i toyed with a few different ways of doing this. first i found an existing golang library for fetching external ips - https://github.com/glendc/go-external-ip

it had a lot of cool features, such as a built in logger, a pre-built list of external ip resolving websites, as well as a cool feature of a quorum of multiple websites with weighted voting.

ultimately i got the code working using the library, but it was a little overkill. adding a library for something i could do in way less lines?

so i ended up pulling it out and switching to a much more simpler option using the example code from the net/http library https://golang.org/pkg/net/http/:

then, we wrap that in a function getexternalipfrom() function, that takes a string of a url to get the external ip from:

from there, we just need to write a schema for a parameter specifying what resolver url you want to use:

then get that value:

then in our datasourceread, run that function with the resolver string, and bam working data source!

and since the behaviour we were looking to test was very similar to the http mocking of the original http provider, we could reuse a lot of the tests:

i ended up using vsc over sublime when writing this, since i’d read the golang and terraform support is miles ahead of sublime, and they weren’t wrong.

it was great at setting up autosuggestions for code, highlighting compilation mistakes and helping run tests by adding little clickable “run test” links for test cases:

i think this will be the main push i have to transfer fully from sublime to visual studio code.

oh man, vendoring.

as i mentioned, i’m a ruby person. so i’m used to some sort of gemfile or something, but golang has a vendoring model, which downloads everything and puts it into the repo.

it seems most of the terraform providers out there are using govendor, so i thought i’d stick to conventions. i asked around internally, and hashicorp standardized around govendor sometime in 2015. <code>govendor</code> was probably one of the better early-day managers (ie: out of the ones released not so long after go 1.5, which is when dependency vendoring was initially introduced), but has been showing its age for quite a while. after that, <code>dep</code> was going to be go’s native upstream package manager, but is now being superseded by <code>vgo</code>. go’s got a bit of a problem right now with not being able to hold all of its dependency managers.

for background i suggest reading these posts which seem to summarize the situation well:

https://sdboyer.io/blog/vgo-and-dep/ https://blog.golang.org/versioning-proposalhttps://github.com/golang/go/wiki/vgo

anyways, we’ve got all the dependencies inside, and it’s compiling. so we built it: how do i actually use it?

so, we need to have the binary built and in the gobin path. so the easiest way i found was to do this:

then we write some terraform to get the external ip:

we need to initialize the repo otherwise we get this:

we initialize the folder so it knows to look for the extip provider:

then we apply it:

great, all working and good!

https://github.com/petems/terraform-provider-extip

i’ll be blogging further about how i used this in my google demo setup.