Create an External Resource

Some of what your infrastructure consists of has no Terraform provider. The shop registers each new queue in an internal service catalog, over HTTP, against a system somebody wrote years ago. There is no resource type for that, and there never will be.

An external resource is Octo's answer. It is a resource whose create, update, and delete are code you write, and Octo still puts it in the Terraform graph so that ordering, dependencies, and reports all keep working. Terraform calls back into Octo mid-apply to run it.

This is an escape hatch. One resource in @quadnix/octo-aws-cdk is external, the published container image, because no provider puts an image into a registry; everything else in the library is a Terraform resource, and yours should be too unless the provider genuinely cannot express what you need.

octo create-external-resource -n shop-catalog-entry --package @shop

The resource

src/resources/shop-catalog-entry/shop-catalog-entry.resource.ts

@Resource<ShopCatalogEntry>('@shop', 'shop-catalog-entry', ShopCatalogEntrySchema)
export class ShopCatalogEntry extends AResource<ShopCatalogEntrySchema, ShopCatalogEntry> {
  declare properties: ShopCatalogEntrySchema['properties'];
  declare response: ShopCatalogEntrySchema['response'];

  constructor (
    resourceId: string,
    properties: ShopCatalogEntrySchema['properties'],
    parents: [MatchingResource<ShopQueueSchema>],
  ) {
    super(resourceId, properties, parents);
  }
}

The only structural difference from a Terraform resource is the base class: AResource instead of ATerraformResource, and no toHCL(). The lifecycle lives in actions instead.

The parent matters more here than usual. The catalog entry is a child of the queue, so the queue is created first, and the queue's outputs are handed to the action as inputs.

The actions

Three of them, one per lifecycle step, in actions/.

actions/add-shop-catalog-entry.resource.action.ts

@Action(ShopCatalogEntry)
export class AddShopCatalogEntryResourceAction extends ANodeAction implements IResourceAction<ShopCatalogEntry> {
  constructor () {
    super();
  }

  filter (diff: Diff): boolean {
    return (
      diff.action === DiffAction.ADD &&
      diff.node instanceof ShopCatalogEntry &&
      hasNodeName(diff.node, 'shop-catalog-entry') &&
      diff.field === 'resourceId'
    );
  }

  async handle (diff: Diff<ShopCatalogEntry>): Promise<ShopCatalogEntrySchema['response']> {
    const { catalogUrl, name } = diff.node.properties;

    const created = await fetch(`${catalogUrl}/entries`, {
      body: JSON.stringify({ name }),
      headers: { 'content-type': 'application/json' },
      method: 'POST',
    }).then((r) => r.json());

    return { entryId: created.id };
  }
}

What handle() returns becomes the resource's response, and Octo records it at commit. An action with nothing to report, such as a delete, returns nothing. Between the action and that commit the response is held under the resource's ID, so a run that stops in between leaves a record that octo validate reports until a commit picks it up.

An action gets 90 seconds by default. Set actionTimeoutInMs on the class if the thing you are calling is slower than that.

Keep the explicit constructor() { super(); }. ANodeAction's constructor is protected, and without a public one the @Factory registration below does not typecheck.

An action that reads a file resolves the path with ProjectUtility.resolveFromProjectRoot(). Terraform runs the add and update actions from a Terragrunt cache directory rather than from your project, and octo commit runs the delete action from the project root, so a relative path means two different things depending on which one is running. Keep the path relative in the resource's properties and resolve it in the action: an absolute one puts your directory layout into Octo state and into the properties hash the generated Terraform commits.

What Octo generates

An external resource still appears in the Terraform folder, wrapped:

.octo-tf/orders-queue-module/main.tf

data "external" "catalog-entry-shop-orders" {
  program = ["octo", "get-response", "--resourceId=catalog-entry-shop-orders"]
  depends_on = [null_resource.catalog-entry-shop-orders]
}

resource "null_resource" "catalog-entry-shop-orders" {
  triggers = {
    octo_properties_hash = "8b33c9058ec4675b"
    input_queue_shop_orders_Arn = "${aws_sqs_queue.queue-shop-orders.arn}"
    input_queue_shop_orders_Url = "${aws_sqs_queue.queue-shop-orders.url}"
  }
  provisioner "local-exec" {
    command = "octo run-action --resourceId=catalog-entry-shop-orders --input queue-shop-orders.Arn=${aws_sqs_queue.queue-shop-orders.arn} --input queue-shop-orders.Url=${aws_sqs_queue.queue-shop-orders.url}"
  }
  provisioner "local-exec" {
    when = destroy
    command = "octo run-action --resourceId=catalog-entry-shop-orders --input queue-shop-orders.Arn=${self.triggers.input_queue_shop_orders_Arn} --input queue-shop-orders.Url=${self.triggers.input_queue_shop_orders_Url}"
  }

  depends_on = [aws_sqs_queue.queue-shop-orders]
}

Read the triggers block, because it is what decides when your action runs again. There is a hash of the resource's properties, and one entry per output of each parent. Change a property, or change the queue's ARN, and Terraform re-runs the provisioner. Change nothing, and it does not.

The hash covers every property, which is right until a resource decides that one of its properties is not a change. Whatever diffProperties() ignores has to be left out of the hash as well, by overriding getExternalTriggerProperties() to return only the properties that count. The published image in our AWS library is that case: a tag is written once and nothing about it can be edited afterwards, so it returns {} and re-triggers on a parent's output alone. A property only the hash covers is one Terraform plans a replacement for and Octo reports no change for.

A null_resource is never updated in place. A changed trigger destroys and recreates it, which means Terraform runs the provisioner twice for one change, once on the way out and once on the way in. Your action still runs once. Octo records what an action made of a resource, and a repeated call for the same inputs and the same properties does nothing. You do not need to make your action idempotent for this case, though making it idempotent anyway is never wrong.

The data "external" block reads the response back and makes it available to other resources, which is how an external resource's response participates in the graph like anything else. Terraform reads it on every plan and every apply, and your action runs on neither of those schedules, so it reads from Octo state rather than from the action. octo run-action records what it published the moment it publishes, octo commit folds that into the resource graph, and octo get-response reads whichever of the two holds it. Note that it is one output for the whole resource rather than one per response key:

.octo-tf/orders-queue-module/outputs.tf

output "catalog-entry-shop-orders" {
  value = data.external.catalog-entry-shop-orders.result
}

Octo also adds hashicorp/null and hashicorp/external to the folder's required providers, without you declaring them.

Two limits

Tags do not trigger it. They do reach it: Octo resolves a resource's tags before the actions run, so diff.node.tags holds whatever octo.yaml declared for it, and an action that wants to put them on the thing it creates can read them there. What tags cannot do is cause a run. The triggers are the properties hash and the parent inputs, and tags is not among them, so changing only a tag re-triggers nothing and no action is invoked. Apply tags from the add and update actions, on the runs those actions already have.

There is no validate action. octo validate compares Terraform's plan against Octo's diff, and does not run resource actions. What you see in a report for an external resource is that the null_resource will be triggered, not what your action will do.

Deletes run at commit

The delete action does not run during the apply. By the time Terraform would run it, the resource's configuration is already gone from the folder, so there is nothing to invoke. octo commit runs it instead.

Which means: if you skip the commit after removing an external resource, the cloud-side thing it represents is still there. That is the strongest reason commit is not an optional step.

It also means removing one destroys nothing during the apply, so octo apply lets it through without --auto-approve. A resource declaring destroyLosesData is the exception: apply refuses its removal like any other destroy that loses data.

Because these deletes run after Terraform has finished, and outside the order Terraform destroys its own resources in, they are only safe once the apply reached the desired state everywhere. An apply that stopped short defers them all. Octo cannot tell which live infrastructure a failed module left pointing at the resource, so it deletes nothing, says so in the commit's warnings, and keeps the resource recorded as present. The next successful run deletes it.

When it is worth it

Use one when a provider cannot express the thing at all: an internal system, a vendor with no provider, a step that is genuinely a script.

Do not use one to avoid learning a provider's resource type, and do not use one for something with a perfectly good provider that you find awkward. Everything Terraform manages gets you plan output, state reconciliation, and drift detection for free, and an external resource has none of that. It has your code, and your code runs in the middle of an apply where debugging is expensive.