What Octo Saves and When

One change to your infrastructure goes through five octo commands, and an apply in the middle of them can take a minute or an hour. Any of them can stop part way: an error from AWS, a closed terminal, a CI job somebody cancelled. Whether that costs you anything depends on what was already saved when it stopped, and on whether the next command can tell what happened.

This page follows one change to the shop through the commands, and says at each one what lands in the state directory, .octo/, and what a stop there leaves behind. The change is a new receipts bucket for checkout.

One change, five commands

Command	Needs the lock	Saves to .octo/
generate	No	models.json, your intent as the definition file now describes it, and imports.json, the other apps it read
validate	No	Terraform's state, after each plan
lock	Takes it	local-state.lock, or a lock in the backend with remote state
apply	Yes	Terraform's state, every ten seconds and at the end, and a record for each external resource it creates
commit	Releases it	models.json, resources-actual.json, resources-old.json, then unlock

Every command also writes its journal file, if your definition file lists the journal.

generate and validate plan and change nothing in your AWS account, so they need no lock and can be run as often as you like. They read the state directory and write intent and Terraform's record of what exists; neither is a promise that anything will be built.

lock is where a change starts for real. From the lock until the commit or unlock that releases it, nobody else can apply or commit, and every command passes the lock ID it printed:

LOCK_ID=$(npm run --silent octo:lock)
npm run octo:apply -- --lockId $LOCK_ID
npm run octo:commit -- --lockId $LOCK_ID

During the apply

octo apply hands your generated Terraform to Terragrunt, and Terraform records what it creates in its own state file, inside each module folder under .octo-tf/. That folder is a working copy. Octo copies each module's state into .octo/<moduleId>.tfstate before Terraform starts, every ten seconds while it runs, and once more when it stops, whether it succeeded or failed. With remote state, each of those copies is uploaded, and checkInIntervalInSeconds sets how often.

So an apply that stops for any reason has saved Terraform's record up to its last copy. One that failed with an error saved all of it. One that was killed outright lost at most its last ten seconds.

Octo's own record, resources-actual.json, is not written during an apply. Terraform knows the receipts bucket exists the moment it is created; Octo learns it at commit, which reads Terraform's outputs and records them against the module that asked for the bucket.

At the commit

octo commit writes Octo's record in one go: if a Terraform output it expects is missing, it throws before writing anything, and the lock stays held so you can run it again. Once the record is written, it releases the lock in three steps:

1. The journal writes the commit's own file. If that fails, the commit fails and the lock stays held, so running the commit again retries it.
2. The change is sealed. With local state, every state file in .octo/ is copied, and every journal file moved, into .octo/archives/<commitId>/. If that fails, the commit fails and the lock stays held. With remote state, the backend seals the change's record, journal included.
3. The lock is released. With remote state, the same archive is then written on this machine; if that fails, the commit prints Failed to archive the state of commit "..." and carries on.

octo unlock, which gives a change up rather than finishing it, takes the same three steps.

The journal

The journal records what each command did: every step it took, the inputs each module ran with, each error, everything Terraform and Terragrunt printed, and a copy of each version of Octo's state it wrote. Octo's state says what your infrastructure is now; the journal says how it got there, which is what you need when the terminal that ran a command is gone.

It is a listener, on when octo.yaml lists it, as a new project does:

settings:
  listeners:
    - type: JournalEventListener
    - type: LoggingEventListener

Each command writes its files into .octo/, named after the command and an ID that starts with the time the command started, so they sort in order. After generate, validate and lock:

journal.1791692243700-a690d91d.generate.f793c0d6cea01dba9aa521449489482f.json
journal.1791692243700-a690d91d.generate.json
journal.1791692244187-30dd727f.validate.json
journal.1791692244187-30dd727f.validate.terraform.json
journal.1791692253700-032b3e64.lock.json

File	Holds
journal.<id>.<command>.json	Every step of the command, in order, and how it ended
journal.<id>.<command>.terraform.json	What each Terraform and Terragrunt process printed, as it printed it
journal.<id>.<command>.<md5>.json	A copy of models.json or a resource graph it wrote

A command's files are saved when it ends, and also every time it saves state, so an apply's journal goes out with each ten-second copy of Terraform's state, Terraform's output so far included. An apply that is killed keeps its journal, and what Terraform printed, up to its last copy.

The Terraform file is a list of records in the order they arrived: a process starting, each piece of output it printed, and its exit code. Nothing is left out, the JSON Octo reads from a plan or from Terraform's outputs included. While a command runs, the file is written into a temporary file in .octo/ as each piece arrives, and that file is deleted when the command ends.

A run-action that Terraform starts in the middle of an apply writes files of its own, and names the apply that started it as parentCommandId.

The files of one lock stay together. generate and validate files wait in .octo/ until the next lock, and when that lock is released, they move into a folder named after the change's commit ID, which starts with the time the lock was taken, beside a copy of the state the change left:

.octo/archives/1791692253600-5bbc6995/
├── journal.1791692243700-a690d91d.generate.json
├── journal.1791692244187-30dd727f.validate.json
├── journal.1791692253700-032b3e64.lock.json
├── journal.1791692257972-3e0cd513.apply.json
├── journal.1791692267354-d74b1acd.commit.json
├── models.json
├── resources-actual.json
├── resources-old.json
└── ...

A change that needed three tries at apply before its commit has all three apply files in that folder.

Each file is JSON, so jq reads it. The steps the last sealed commit took:

jq -r '.entries[].type' "$(ls -d .octo/archives/*/ | tail -1)"journal.*.commit.json | uniq -c

   1 CommandStartedEvent
   4 ModuleEvent
   1 TerraformCommandStartedEvent
   1 TerraformProcessStartedEvent
   1 TerraformProcessExitedEvent
   1 TerraformCommandFinishedEvent
   1 StateFilesSavedEvent
   1 ModeStartedEvent
   1 ModelDiffsTransactionEvent
   5 ModelActionTransactionEvent
   1 ModelTransactionTransactionEvent
   1 ResourceDiffsTransactionEvent
   3 StateFilesSavedEvent
   1 CommitModeCompletedEvent
   1 ModeFinishedEvent
   1 AppUnlockingEvent

Events and Listeners says what each of those is. The error a command failed with:

jq -r '.entries[]? | select(.type == "CommandFailedEvent") | .payload.error.message' .octo/journal.*.json

Beside its message, an error of Octo's own carries its className and what it is about: an AppLockError names the reason and the holder of the lock, an error from an action names the action and the diff it ran for. Any other error carries only its name. When the command knows what to run next, as a failed octo apply does, the entry also holds it as nextStep:

jq -r '.entries[]? | select(.type == "CommandFailedEvent") | .payload.nextStep // empty' .octo/journal.*.json

To read these files from code, their shapes are TypeScript types: @quadnix/octo-event-listeners/types has JournalFile and TerraformLogFile, with each event's payload as the journal holds it, and @quadnix/octo/types has ModelStateFile and ResourceStateFile.

The journal is gitignored in a new project, both the files in .octo/ and archives/. The inputs each module ran with are recorded as given, so a secret passed as a module input is in the journal in plain text. Nothing in Octo reads the journal back: deleting it changes nothing about your infrastructure or what Octo knows of it.

The newest archive is the sealed state: an app that extends this one, or reads it as a store, reads it, and never the files a change still under way is writing. Archives are never deleted for you. Delete the old ones when you no longer need them; with none left, an app extending or reading this one has nothing to read until the next change is sealed.

With remote state, the journal files in .octo/ are uploaded with the rest of the state while the lock is held, so the record a lock seals holds every command that led to it. archives/ stays on the machine.

When it stops part way

Where it stopped	What is saved	What to do
generate or validate	Nothing is locked	Fix what the error names and run it again
apply, with an error	Terraform's record of everything it built	The apply failed
apply, killed	Terraform's record, but its last ten seconds	The run died
commit, with an error	Nothing new; the lock is still held	The commit failed
commit, after Failed to archive	Everything; the lock is released	Move or delete the journal files left in .octo/

The last row needs nothing but tidying, and octo lock points it out: it refuses to start a new change while .octo/ holds a journal file of a command that only runs under a lock, and no lock is held:

Error: State holds the journal of a run that was never archived: journal.1791692481268-593d89e5.commit.json! Move or delete these files, then lock again.

Move the files it names somewhere you want to keep them, or delete them, and lock again.

A command that only runs under a lock writes no journal file when no lock is held, so running octo apply before octo lock, or octo unlock on an app already unlocked, leaves nothing behind.