After the terraform apply, getting issue with vault export root token and crt looks invalid

[saravana1992]

Hi

I have did cloning and exported the values of root token then looks token not available then when i run vault status looks ca.pem or crt is invalid and the errors are below

vault status
Error checking seal status: Get https://x.x.x.x/v1/sys/seal-status: x509: certificate signed by unknown authority

export VAULT_TOKEN="$(terraform output root_token)"
The output variable requested could not be found in the state
file. If you recently added this to your configuration, be
sure to run terraform apply, since the state won't be updated
with new output variables until that command is run.

Let me know the work around for this, looks like there was PR few days before if this related to it?

[dhduvall]

The "unknown authority" issue might be due to the fact that Vault is set up with a self-signed certificate, and anything that communicates with the service will need to know what the CA is that signed that cert. For the vault CLI, you can set VAULT_CAPATH="..../vault-on-gke/tls/ca.pem" in your environment, or use the -ca-path flag. Other tools will have other solutions.

The root_token is no longer kept in the state. Use root_token_decrypt_command instead, and run that command. It looks like that needs to be changed in the README.

[stale]

Hi there! This has been automatically marked as stale because it has not had activity in the past 14 days. It will be closed in 14 days if no further activity takes place.

[prankin272]

I'm running this through a terraform based azure devops pipeline so these 2 bits are obviously creating the ca.pem and vault.pem on the build agent:

provisioner "local-exec" { command = "echo '${self.cert_pem}' > /tmp/ca.pem && chmod 0600 /tmp/ca.pem" } }

provisioner "local-exec" { command = "echo '${self.cert_pem}' > /tmp/vault.pem && echo '${tls_self_signed_cert.vault-ca.cert_pem}' >> /tmp/vault.pem && chmod 0600 /tmp/vault.pem" } }

Would an extra provisioner which just copies the files over be ok?

[prankin272]

No matter even if I transfer the ca.pem and vault.pem across and do the export capath, tls still doesn't work. Any advice?

[dhduvall]

Maybe. What's the full command-line and output? Plus any $VAULT_* environment variables, sensitive parts redacted as appropriate.

[prankin272]

Thanks for the reply @dhduvall

In my tls.tf:

# Generate self-signed TLS certificates. Unlike @kelseyhightower's original
# demo, this does not use cfssl and uses Terraform's internals instead.
resource "tls_private_key" "vault-ca" {
  algorithm = "RSA"
  rsa_bits  = "2048"
}

resource "tls_self_signed_cert" "vault-ca" {
  key_algorithm   = tls_private_key.vault-ca.algorithm
  private_key_pem = tls_private_key.vault-ca.private_key_pem

  subject {
    common_name  = "vault-ca.local"
    organization = "HashiCorp Vault"
  }

  validity_period_hours = 8760
  is_ca_certificate     = true

  allowed_uses = [
    "cert_signing",
    "digital_signature",
    "key_encipherment",
  ]

  provisioner "local-exec" {
    command = "echo '${self.cert_pem}' > /tmp/ca.pem && chmod 0600 /tmp/ca.pem"
  }
}

# Create the Vault server certificates
resource "tls_private_key" "vault" {
  algorithm = "RSA"
  rsa_bits  = "2048"
}

# Create the request to sign the cert with our CA
resource "tls_cert_request" "vault" {
  key_algorithm   = tls_private_key.vault.algorithm
  private_key_pem = tls_private_key.vault.private_key_pem

  dns_names = [
    "vault",
    "vault.local",
    "vault.default.svc.cluster.local",
  ]

  ip_addresses = [
    google_compute_address.vault.address,
  ]

  subject {
    common_name  = "vault.local"
    organization = "HashiCorp Vault"
  }
}

# Now sign the cert
resource "tls_locally_signed_cert" "vault" {
  cert_request_pem = tls_cert_request.vault.cert_request_pem

  ca_key_algorithm   = tls_private_key.vault-ca.algorithm
  ca_private_key_pem = tls_private_key.vault-ca.private_key_pem
  ca_cert_pem        = tls_self_signed_cert.vault-ca.cert_pem

  validity_period_hours = 8760

  allowed_uses = [
    "cert_signing",
    "client_auth",
    "digital_signature",
    "key_encipherment",
    "server_auth",
  ]

  provisioner "local-exec" {
    command = "echo '${self.cert_pem}' > /tmp/vault.pem && echo '${tls_self_signed_cert.vault-ca.cert_pem}' >> /tmp/vault.pem && chmod 0600 /tmp/vault.pem"
  }
}

I did change the provisioner cmds to the below, which allows a successful terraform run but I'm unable to find the ca.pem and vault.pem on both the build agent and pod:

provisioner "local-exec" { command = "echo '${self.cert_pem}' > ${path.module}/ca.pem && chmod 0600 ${path.module}/ca.pem" } }

provisioner "local-exec" { command = "echo '${self.cert_pem}' > ${path.module}/vault.pem && echo '${tls_self_signed_cert.vault-ca.cert_pem}' >> ${path.module}/vault.pem && chmod 0600 ${path.module}/vault.pem" } }

Terraform apply stage output after using the above cmds instead: "> ../modules/gcp/vault-terraform/tls/ca.pem && chmod 0600 ../modules/gcp/vault-terraform/tls/ca.pem"]"

Would like to be able to have the ca.pem and vault.pem transfer over during the terraform run.

After a terraform apply, vault pods are all running and I kubectl exec into the vault and run these commands:

/ # export VAULT_ADDR="<api-address/load balancer endpoint ip>"
/ # export VAULT_TOKEN=""
/# export VAULT_CAPATH=""

Then when I proceed to do any kind of vault command, I get this:

Error authenticating: error looking up token: Get "https://<lb-endpoint>/v1/auth/token/lookup-self": x509: certificate signed by unknown authority

EDIT:

Logs

kubectl -n vault logs vault-0 vault
==> Vault server configuration:

             Api Address: https://<lb endpoint>
                     Cgo: disabled
         Cluster Address: https://<cluster>
              Go Version: go1.16.6
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
              Listener 2: tcp (addr: "10.212.0.7:8200", cluster address: "10.212.0.7:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: warn
                   Mlock: supported: true, enabled: true
           Recovery Mode: false
                 Storage: gcs (HA available)
                 Version: Vault v1.8.1
             Version Sha: <removed>

==> Vault server started! Log data will stream in below: