GCP won’t admit the service account exists. So Terraform fails hard. Link to heading

Google Cloud service accounts and IAM grants are not usable the instant Terraform creates them. They are eventually consistent. While the API, and thus Terraform, indicates creation, your next dependent resource fails with a 404, a 403, or an “inconsistent result” error.

Why it matters: This is one of the most common flaky pipeline issues on GCP. The go-to fix would be a sleep, but determining how long to sleep is troublesome and it has the potential to hide real permission bugs.

Google says IAM policy changes usually propagate in about 2 minutes but can take 7 minutes or longer.

The big picture: There’s no single fix and I’d recommend a layered approach. This requires leaning on your own configuration, dependencies, the provider, and finally a wait or readiness check when GCP just keeps giving you problems.

First, figure out which thing is late Link to heading

Why it matters: There’s four main delays which producee similar errors, and these four main delays require different fixes.

What is delayed	What you see	The real dependency
Account becomes visible to a consuming API	“service account does not exist,” not found	Account creation → consumer
Account’s permissions take effect	403, cannot access target resource	Role grant → resource
Runner’s ability to attach or impersonate	`actAs` or `getAccessToken` denied	Grant on the account → attach or impersonation
Group-derived access takes effect	Intermittent auth failures, sometimes for hours	Group change → a later stage

Please note: Not every 403 is a propagation issue. If you’re hitting real permissions issues - like principals lacking roles/iam.serviceAccountUser or roles/iam.serviceAccountTokenCreator, you don’t need waits, you need to look back at your code and make the needed changes to permissioning.

1. Upgrade the provider and let it retry Link to heading

The bottom line: The Google provider already retries known transient errors for many resources.

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 6.0" # Pin a tested, current version in your lockfile.
    }
  }
}

Before adding sleeps, look at the actual failure:

TF_LOG=INFO terraform apply

If a second apply succeeds immediately, this is strong evidence of propagation lag.
However, a consistent 403 that won’t go away is more likely a missing permission or other configuration issue.

2. Reference attributes, never rebuild strings Link to heading

How it works: Pointing the consumer at google_service_account.runtime.email gives Terraform an implicit dependency. It’s tempting to recreate the email in a string but this doesn’t help the graph represent the actual dependencies at play.

resource "google_service_account" "runtime" {
  project      = var.project_id
  account_id   = "app-runtime"
  display_name = "Application runtime"
}

resource "google_cloud_run_v2_service" "app" {
  name     = "app"
  location = var.region

  template {
    service_account = google_service_account.runtime.email
    containers {
      image = var.image
    }
  }
}

Yes, but: This is necessary but not sufficient as it creates the account first but GCP may still not propagate it in time for the other resource to use it.

3. Add `depends_on` for IAM bindings Link to heading

Why it matters: An IAM grant usually shares no data with the resource that needs it, so an explicit dependency ties the resources together and helps order the graph.

resource "google_project_iam_member" "runtime_storage_access" {
  project = var.project_id
  role    = "roles/storage.objectViewer"
  member  = "serviceAccount:${google_service_account.runtime.email}"
}

resource "google_cloud_run_v2_service" "app" {
  # ...
  depends_on = [google_project_iam_member.runtime_storage_access]
}

Yes, but: Once again, this fixes graph order, not eventual consistency. Note that mixing the wrong types of IAM resources on the same scope causes policy churn that you might mistake for eventual consistency issues.

4. Put one bounded wait at the consistency boundary Link to heading

When to use it: A specific downstream service keeps racing the grant and causes issues, and you’ve tried the previous steps.

How it works: Use the hashicorp/time provider’s time_sleep after the last prerequisite. Then, make only the affected consumer depend on it.

resource "time_sleep" "wait_for_runtime_identity" {
  depends_on      = [google_project_iam_member.runtime_storage_access]
  create_duration = "90s"

  triggers = {
    service_account = google_service_account.runtime.unique_id
    role            = google_project_iam_member.runtime_storage_access.role
    member          = google_project_iam_member.runtime_storage_access.member
  }
}

resource "google_cloud_run_v2_service" "app" {
  # ...
  depends_on = [time_sleep.wait_for_runtime_identity]
}

Start at 60 to 120 seconds. Google’s documented typical lag is 2 minutes after all.
Tune from CI evidence. You might be able to get away with a short sleep, speeding up your pipeline!
Watch the gotcha: time_sleep only waits when it is created or replaced. If a binding changes in-place later, the sleep does not rerun. Using triggers forces replacement when the identity or grant changes.

The bottom line: A static sleep is a workaround, and it’s brittle.

5. Split bootstrap from workload deployment Link to heading

The big picture: For production CI/CD, the cleanest design is two applies with a gate(and delay) between them.

Bootstrap apply: service accounts, Workload Identity Federation, project and folder IAM, API enablement, shared foundations, etc. – things you might get caught up in eventual consistency issues with and related resources that don’t use those identities.
Wait for & verify propagation.
Workload apply: Anything that uses those identities.

Why it matters: One monolithic apply subjects everything to the whims of IAM lag. This allows a consumer apply to get everything that its resources depend on, with them being created independently and first.

Yes, but: Split them up in the code, do not use terraform apply -target=... as a crutch or crux of your pipeline architecture. You’ll hit nightmarish state issues inevitably.

6. Probe readiness instead of guessing in your `sleep` Link to heading

The bottom line: A poll that tests the real operation gets accuracy a sleep won’t.

What to check, by failure mode:

Account exists: gcloud iam service-accounts describe.
Runner can impersonate: attempt token creation or a short impersonated command.
Target access works: call the target API with impersonated credentials.
Runner can attach: confirm iam.serviceAccounts.actAs on the target account.

Example of a minimal CI gate:

set -euo pipefail

SA="app-runtime@${PROJECT_ID}.iam.gserviceaccount.com"

for attempt in $(seq 1 24); do
  if gcloud iam service-accounts describe "$SA" \
       --project="$PROJECT_ID" >/dev/null 2>&1; then
    echo "Service account is visible."
    exit 0
  fi
  sleep 10
done

echo "Service account did not become visible in four minutes." >&2
exit 1

Yes, but: The IAM API successfully returning the account doesn’t mean every consumer accepted its grants. Make sure to test not just presence but use with the same identity and API path the workload will use.

7. Retry the apply, narrowly Link to heading

When to use it: The flaky operation cannot be isolated inside Terraform.

A good retry policy has:

A small cap of maybe 2 to 4 attempts, but this depends on use case.
Exponential backoff with jitter(this can be applied to 6 above as well.)
An allowlist of transient errors. Post-create 404s and the specific 403s known to appear during IAM propagation and shouldn’t be blocking.
Fast failure on structural errors - if you hit bad resource IDs, unsupported regions, quota, policy violations, etc., it’s not about the waiting and you should fail fast.

for attempt in 1 2 3; do
  terraform apply -auto-approve && exit 0
  delay=$((30 * attempt))
  echo "Terraform apply failed; retrying in ${delay}s..." >&2
  sleep "$delay"
done
exit 1

Go deeper: In a mature pipeline, capture the error output and retry only on a reviewed match. Blind retries are asking for trouble.

Fix the authentication architecture Link to heading

The worst race case: Terraform creates a service account, then immediately tries to authenticate as it. Don’t do this if you can avoid it.

How to avoid it: Use a stable existing deployer identity to create accounts and grants, then let workloads run as the new account. Don’t have Terraform quickly auth as a new identity.

Go deeper: Google recommends Application Default Credentials for Terraform. Locally, that means user ADC or impersonation. In CI outside GCP, use Workload Identity Federation, not long-lived keys.

The bottom line Link to heading

Layer your solution by getting down correct IAM first, using implicit references and depends_on for order, provider retries for transient errors, and a (scoped, tuned) sleep wait or a real API readiness check only once you’ve isolated it as an eventual consistency error that’s sticky.

For stable production automation, provision identities and baseline IAM in a bootstrap stage using a pre-existing deployer and then deploy workloads after propagation. And definitely avoid authenticating Terraform as the account it created moments ago.

If you found this helpful, you can connect with me on LinkedIn here for more painfully gained knowledge.

Sources