Skip to main content

State Providers

Octo's state is three JSON files and a Terraform state file per module, and where they live decides who can run your infrastructure. On a laptop, one person can. In a repository, whoever has the repository can, as long as they take the lock.

A state provider is what makes that choice. It holds the files, it holds Terraform's own state, and it holds the lock that keeps two runs from writing at once. Three ship with Octo, and settings.stateProvider picks one.

The three that ship​

octo.yaml
settings:
  stateProvider:
    type: LocalStateProvider
    statePath: .octo

LocalStateProvider writes plain JSON into a directory. Readable, diffable, and the right choice while you are learning.

What those files hold decides how far that choice travels. Octo records every Terraform output it maps to a resource, whatever that output is, and resources-actual.json is a file a project commits: a module that creates a credential and returns it as part of a resource's response puts it there in plain text. Nothing in @quadnix/octo-aws-cdk returns one today, which is a fact about those modules rather than about the file. Terraform's own state, which sits in the same directory, has always worked this way and holds every attribute a provider returned rather than only the outputs Octo maps. Reports are the exception and need nothing done to them: the renderer prints (sensitive) in place of any value Terraform marked, and the scaffolded .gitignore excludes the directory they are written to.

TestStateProvider keeps everything in memory, and is what the test harness uses.

RemoteStateProvider keeps state in a backend a whole team reaches, and has a section of its own. It is the only provider for a team: LocalStateProvider is for one person.

It holds the lock​

Octo's state has one writer at a time, and the provider is what enforces it.

octo lock takes the lock and prints an ID. octo apply and octo commit each present that ID, and a commit that succeeds releases it. generate and validate never touch any of this, because neither writes shared state. A second run that tries to take a held lock fails immediately, naming the ID that holds it, before it reads state or plans anything.

LocalStateProvider keeps the lock as a file beside the state, local-state.lock, created exclusively so that of two runs racing for it exactly one wins. The ID is what proves ownership: every check compares against the ID that was handed out, so one run can never release another's lock, and a caller holding a stale ID is told it is stale rather than told the app is free. The file also records when the lock was taken, and a run that finds it held is told how long that has been. It is a file on one disk, so it keeps apart two runs on one machine and nothing more. RemoteStateProvider keeps the lock in its backend instead, where it keeps apart runs on any machines.

No lock has a timeout. A lock that expires mid-apply would let a second apply start while the first might still be running, and from outside you cannot tell whether a dead run applied nothing, everything, or half of it. A run that died is finished through its own lock rather than released: Recovering a Failed Run has the steps. How long the lock has been held is the first thing to look at: a lock a few seconds old belongs to a run still going, and one older than any run you have ever had belongs to a runner that died.

Releasing the lock seals the change. LocalStateProvider copies the state, and moves the journal, into archives/<commitId>/ before it deletes the lock file, and keeps the lock held when it cannot. The newest archive is the sealed state, which an app that extends this one, or reads it as a store, reads. Archives are never deleted for you. RemoteStateProvider seals the change's record in its backend, then writes the same archive on the machine.

It holds Terraform's state too​

Terraform keeps its own record of what exists, and by default a local backend leaves it in the generated module folder, where only the machine that ran the apply can see it. That is the one file Octo cannot rebuild: lose it and Terraform plans to create infrastructure that is already there.

So it goes through the provider as well. Before any Terraform command, Octo writes each module's state from the provider into its folder; after the command returns, whether it succeeded or failed, it writes what the folder now holds back, under <moduleId>.tfstate. A long apply is also checked in every ten seconds, so a runner killed halfway through loses at most ten seconds of record rather than all of it.

Three consequences worth knowing:

  • A module whose folder holds no state, or state recording no resources, is deleted from the provider rather than written. That is a module Terraform has destroyed.
  • A module the provider holds nothing for keeps whatever its folder already has. That is what adopts the state of a stack which has been running with a local backend until now: the first check-in hands it over, and from then on the provider is the one that answers.
  • A folder whose state is a later version of the provider's copy keeps it too. That is what a run killed before its last check-in leaves behind, and it is why the machine a run died on can finish it without losing anything the run built.

Remote state​

LocalStateProvider keeps the state and the lock on one machine, and a run that dies there takes both with it. RemoteStateProvider keeps them in a backend every machine reaches, and writes to it as the run goes. The GitHub Actions Pipeline runs on it.

octo.yaml
env:
  - key: STATE_SESSION_ID
    kind: string
    value: ""

settings:
  stateProvider:
    type: RemoteStateProvider
    stateManagementServiceUrl: https://state.example.com/api/v1/state   # the address you were given
    organizationId: shop-organization                                   # your organization's ID
    appId: shop                                                         # the app's ID in the backend
    sessionId: "${env.STATE_SESSION_ID}"
    statePath: .octo

stateManagementServiceUrl is where the backend answers, including the prefix its routes sit behind. organizationId and appId name the app the state belongs to. sessionId is the personal access token of the account every request is made as. It does not expire, and it is a credential, so declare it under env and let the shell supply it. lockContext is optional, and is what a blocked run is told about whoever holds the lock; it defaults to octo on <hostname>, and a pipeline can put its run URL there instead. checkInIntervalInSeconds is optional as well: how often, from 5 to 60 seconds, an apply uploads Terraform's state. It defaults to 10. Each upload is one request to the backend, and a check-in that finds nothing changed makes none, so a longer interval means fewer requests and more to lose when a run dies.

statePath is a working copy rather than the store. Each command reads and writes the directory exactly as LocalStateProvider would, and the provider keeps the backend in step:

  • octo generate first replaces what the directory holds with the newest state a finished run left in the backend, and plans against that.
  • octo lock opens a record for the run, and refuses with App state moved since it was last refreshed! when another run finished after your generate. Run generate again, then lock. So on this provider, generate comes before lock.
  • Every write the run makes reaches the backend before the command goes on, and Terraform's state reaches it on the check-in described above, every checkInIntervalInSeconds. A run that dies has lost at most that interval.
  • A commit that succeeds, or octo unlock, finishes the record. From then on it is the state the next generate reads. A write that failed to reach the backend is still in the directory, so both upload the directory once more before finishing, and when that fails too, the lock stays held and the record unfinished.

The backend keeps the finished record of every run, and never reads an unfinished one back into a plan. A run that died leaves its record unfinished and its lock held, so everybody else plans against the last finished run and cannot take the lock until somebody finishes that one. Any machine can, as the account that took the lock or as an organization administrator: Recovering a Failed Run has the steps.

Every file at the top of the directory is uploaded. The provider refuses a name the backend would refuse before the write lands on disk: names are made of letters, digits, _, -, and ., ending in .json or .tfstate. The backend holds your app to its own limits, 20 MB a file, 1000 files a state and 100 MB of state in all unless your app's limits were raised, and refuses a write past them before any file is uploaded. That includes the journal of the lock under way, so the record a lock seals holds every command that led to it. generate never replaces journal files when it refreshes the directory, resuming a lock on another machine keeps that machine's journal files and uploads them with the rest, and the archives/ they move to once the lock is released, beside a copy of the state, is never uploaded.

The directory is scratch space, and the backend is the copy that counts, so add .octo/ to your .gitignore. Two files of the provider's own live there beside the state, remote-state.lock and remote-state-upload.lock, and neither is uploaded.

To move a project on LocalStateProvider onto this provider, change settings.stateProvider and run the lifecycle as usual. While the backend holds nothing for the app, generate leaves the directory alone, and the lock uploads what it holds as the run's starting point.

What a provider does not do​

It does not give you a way back. Octo writes each file whole, every time, and has no command that restores an earlier version. The journal keeps a copy of each version a command wrote, and each archive a copy of the state a change sealed, but Octo reads back only the newest archive, for an app extending or reading this one, so going back has to come from wherever the files live: a repository's history for state you commit, or versioning on the storage the directory sits on. The backend behind RemoteStateProvider keeps the record of every finished run, but nothing in Octo reads an old one back. Restoring old state does not undo infrastructure either. It makes Terraform forget what exists, and the next apply tries to build it again.