secrets: support expiring secrets/automatic through providers, deliver secrets to workloads as files #28709

The secret block provides a way to retrieve secrets from built-in and custom providers, but from what I understand that's pretty much it. I'm not sure if there's been any internal consideration for this, but it'd be great to see the functionality extended so that providers can return an expiration time where Nomad would then automatically fetch replacements and deliver updated values to workloads.. Trying to do this today, probably end up running Vault and using the template{} stanza, or roll your own business logic in services, wrappers, or sidecar containers.

For me specifically, my main use case is issuing short-lived PKI certificates through a custom secret provider, which I know has a lot of overlap with functionality Vault + Nomad can provide, but I currently don't run Vault in this environment. I want Nomad to obtain a certificate (in this case signed against Google's CAS product from workloads running outside GCP), write the certificate and private key to files accessible by the workload, and replace them before expiration.

Proposal

Extend the existing fetch response with an optional expires_at timestamp:

{
  "result": {
    "certificate": "PEM certificate contents\n",
    "private_key": "PEM private key contents\n"
  },
  "expires_at": "2030-01-02T03:04:05Z"
}

When present, Nomad would schedule another fetch before expiration. Responses without expires_at would retain their existing behavior.

Proposed job spec

Inside a task:

secret "tls" {
  provider = "example-pki"
  path     = "services/api"

  # optional, by default nomad would attempt to fetch a new secret at 80% of the secret's lifetime
  refresh {
    threshold = 0.8
  }

  # today `env{}` is supported, but at least in my case the ability to write a response out to a
  # file would also be a welcome addition (here we write out `result.certificate` to secrets/tls.crt)
  file {
    key         = "certificate"
    destination = "secrets/tls.crt"
    perms       = "0644"
  }

  # i wonder if it's worth allowing nomad to splat the entire result out to a path?
  # creates `result.certificate` in `secrets/tls/certificate`,
  # `result.private_key` in `secrets/tls/private_key`
  destination = "secrets/tls"

  # similar to `template{}`, what should nomad do when it fetches a new secret?
  change_mode   = "signal"
  change_signal = "SIGHUP"
}

I have a working PoC of this locally -- if there's interest in building something like this into Nomad, would be happy to iterate on a design and contribute it back.