You Can’t Import a GSI by Itself

The incident started like this: someone thought the provisioned capacity on the prod DynamoDB table wasn’t enough, so they directly ran two aws dynamodb update-table commands — switched billing from PROVISIONED to on-demand, and added a GSI while they were at it. They tested it, it worked, done.

They did not go through CloudFormation.

A week later, during a completely unrelated release, CloudFormation got stuck hard in rollback, the whole stack entered UPDATE_ROLLBACK_COMPLETE, and nobody dared click deploy again. The error message was just one line: Attempting to create an index which already exists.

CFN Only Trusts Its Own Ledger, Not Reality

CloudFormation is essentially a ledger: you give it a template, it records “what this set of resources should look like,” then creates and updates according to that. The problem is that when it computes a diff, it compares the new template vs its own ledger — it never checks what the cloud resource actually looks like.

So when someone bypasses it and changes a resource directly, the ledger no longer matches reality. This is drift.

The nastier part is rollback itself: when CFN rolls back a failed “add GSI” operation, it will try to delete the GSI it thinks it just created — but that is actually the index your production traffic is already using. Rollback deletes the wrong thing. That is why nobody dared touch it: every deployment was a gamble.

The Counterintuitive Part: You Can’t Import a GSI

The natural thought is: then just let CFN “claim” the out-of-band GSI again, right? CloudFormation has resource import.

But there is a trap here: import works at the whole-resource level. You can import an entire table or an entire bucket, but a GSI is not an independent resource; it is a property of AWS::DynamoDB::Table. You cannot import “one child property inside a table that is already managed by the stack” on its own.

To make CFN accept drift in a child property, you can only remove the whole table from the stack first, then import the whole table back in.

The Fix: Remove → Retain → Reimport

The key is to use DeletionPolicy: Retain — when CFN “lets go of” a resource, it does not actually delete it.

When writing the import changeset in step 4, there are also two traps that are not placed anywhere obvious. You only learn them after stepping on them once:

aws cloudformation create-change-set \
  --change-set-type IMPORT \
  --template-body file://original.yaml \   # Must be Original YAML (before transform), not processed JSON
  --parameters "$PARAMS" \                 # Pass all stack parameters with UsePreviousValue
  --resources-to-import '[{"ResourceType":"AWS::DynamoDB::Table",...}]'

• You must pass UsePreviousValue for all parameters at once (not just the required ones). Any omitted parameter falls back to the template default, causing CFN to decide that a bunch of unrelated resources “changed” and reject the import with “N resources modified.”
• The template must be the Original YAML (before transform), not the processed JSON returned by get-template by default. If you submit the processed version back, CFN will still detect differences in resources expanded and generated by SAM.

A Few Notes for Myself

The really scary part was not this procedure — the detach step does not touch DynamoDB at all, the table keeps serving traffic, and you can roll back if you mess it up. The scary part was that it stayed buried for a week before exploding, and it exploded during a release that had absolutely nothing to do with it.

• Bypassing IaC and directly running update-table / changing things in the console is a slow poison. It works for the moment, but the cost is written into the ledger, waiting to be settled all at once when the next unlucky person ships a release.
• CFN’s diff is “template vs. ledger”; it never includes “reality.” So detect-stack-drift is worth running regularly — don’t wait for it to tell you during a rollback.
• import works at resource granularity. For drift in child attributes (GSI, one item in a parameter group…), accept your fate and reimport the whole resource.

That table is now back under CFN, the ledger matches reality, and detect-stack-drift is clean. For the next release, we finally don’t have to gamble.