Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions docs/backend-waiting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Waiting for sleeping backends

A tsjwt Gateway can hold authorized requests until the configured upstream is
connectable. The tailnet listener and WhoIs identity lookup remain inside tsjwt.
Waiting is disabled by default. Enable it only after the platform has a working
external demand metric and autoscaler for every backend of the Gateway:

```yaml
metadata:
annotations:
tsjwt.dev/backend-wait-timeout: 60s
```

The controller validates a positive duration of at most two minutes and passes
it to the data plane as `-backend-wait-timeout`. Standalone data planes accept
that flag and `-backend-wait-max-pending` (default 128 per upstream).

Requests are authorized before they register demand. Each request waits at most
the configured timeout, or until its caller disconnects. Concurrent waiters
share a readiness probe per upstream. The default probe opens a TCP connection
to the configured upstream; for Kubernetes Services this relies on normal
ready-endpoint publication. Do not use publishNotReadyAddresses or an upstream
proxy that accepts connections before the target app is ready. Core consumers
can inject a stronger readiness probe using WaitConfig.Ready.

On readiness, the proxy reauthorizes the caller and mints a fresh assertion,
then forwards the original request once. The tenant selected on arrival stays
pinned through the wait, even if the caller's default changes; loss of access
to that tenant returns 403 rather than falling back to another tenant. It does not retry application requests
or replay POST bodies after forwarding. Failure or queue saturation returns 503
with Retry-After. Waiting does not read/buffer the body; memory used for waiting
requests is bounded by request limits rather than body size. Active requests,
including upgraded connections, remain demand until proxying finishes.

The gate exports the following on `/metrics` on the existing internal operations
listener (the listener serving JWKS and health), respecting its TLS setting:

- `tsjwt_backend_pending_requests{upstream="..."}`
- `tsjwt_backend_active_requests{upstream="..."}`
- `tsjwt_backend_last_activity_seconds{upstream="..."}`

These are process-local metrics keyed by configured upstream URL, shared across
route reloads. The platform must map ExternalName bridges and alternate ingress
paths to a stable backend identity and sum pending/active demand across them.
Retain a nonzero recommendation for the chosen idle window using last activity.
Metrics exist after the first authorized request, remain explicit zero when idle,
and are absent after restart until demand is seen again. Missing/stale metrics
must fail safe and never be converted to idle zero. Backends are retained for
explicit zero publication; a process admits at most 256 unique upstreams over
its lifetime, after which new ones return 503 until restarted. Labels never
contain caller-supplied hosts or paths.

This feature does not scale applications, create HPAs, discover dependency
workloads, or install a metrics adapter. In particular it cannot wake an app
without the platform consuming its demand metric. Keep gates, identity services,
metrics infrastructure and databases running. Check every ingress path before
enabling suspension. Keep HPA as the only replica writer.

Release the library first, then bump the deployment repository's pinned source
and enable the annotation for a disposable pilot through its normal CI/CD path.
After deployment validate the controller/dataplane rollout, Gateway conditions,
JWKS and unauthorized responses, then the pilot's idle → zero → held request →
ready → successful response lifecycle. Inspect demand metrics and HPA events.
Rollback through CI/CD: restore the workload's autoscaler minimum to one first,
wait for readiness, then disable waiting or revert the release. Never disable
the waiting gate while workloads remain asleep.
62 changes: 41 additions & 21 deletions gateway/cmd/tsjwt-dataplane/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,28 +47,33 @@ func main() {

func run() error {
var (
gatewayRef = flag.String("gateway", "", "the Gateway this serves, as namespace/name (required)")
routesPath = flag.String("routes", "/etc/tsjwt/routes.json", "the rendered route table, normally a mounted ConfigMap")
routeCache = flag.String("route-cache", "", "where to keep the last table that parsed, so a restart during an outage still has routes; defaults to <state-dir>/routes.cache.json")
hostname = flag.String("hostname", "", "tailnet hostname; defaults to the Gateway name")
stateDir = flag.String("state-dir", "/var/lib/tsjwt", "directory for tailnet node state")
issuer = flag.String("issuer", "", "iss claim; defaults to https://<hostname>")
tenants = flag.String("tenants", "", "path to the tenant policy JSON file (required)")
listen = flag.String("listen", ":443", "tailnet address to listen on")
jwksListen = flag.String("jwks-listen", "0.0.0.0:9100", "ordinary address for the key set and health")
header = flag.String("header", tsjwt.DefaultHeader, "header the assertion is injected into")
bearer = flag.Bool("bearer", false, `write the assertion as "Bearer <token>"`)
capName = flag.String("cap", string(tsnetid.CapGroups), "tailnet capability carrying the caller's groups")
ttl = flag.Duration("ttl", signer.DefaultTTL, "token lifetime")
rotate = flag.Duration("rotate", 24*time.Hour, "signing key rotation interval")
overlap = flag.Duration("overlap", time.Hour, "how long a retired key still verifies")
resync = flag.Duration("resync", 5*time.Second, "how often to re-read routes")
keyTags = flag.String("key-tags", "", "comma-separated tags for an auth key minted at startup")
shareCM = flag.String("share-keys", "", "publish verification keys to this ConfigMap")
plaintext = flag.Bool("plaintext", false, "serve HTTP instead of HTTPS")
jwksTLS = flag.Bool("jwks-tls", false, "serve the key set over HTTPS, using this node's own certificate")
waitTimeout = flag.Duration("backend-wait-timeout", 0, "hold authorized requests for backend readiness; zero disables waiting")
waitPending = flag.Int("backend-wait-max-pending", 128, "maximum waiting requests per upstream")
gatewayRef = flag.String("gateway", "", "the Gateway this serves, as namespace/name (required)")
routesPath = flag.String("routes", "/etc/tsjwt/routes.json", "the rendered route table, normally a mounted ConfigMap")
routeCache = flag.String("route-cache", "", "where to keep the last table that parsed, so a restart during an outage still has routes; defaults to <state-dir>/routes.cache.json")
hostname = flag.String("hostname", "", "tailnet hostname; defaults to the Gateway name")
stateDir = flag.String("state-dir", "/var/lib/tsjwt", "directory for tailnet node state")
issuer = flag.String("issuer", "", "iss claim; defaults to https://<hostname>")
tenants = flag.String("tenants", "", "path to the tenant policy JSON file (required)")
listen = flag.String("listen", ":443", "tailnet address to listen on")
jwksListen = flag.String("jwks-listen", "0.0.0.0:9100", "ordinary address for the key set and health")
header = flag.String("header", tsjwt.DefaultHeader, "header the assertion is injected into")
bearer = flag.Bool("bearer", false, `write the assertion as "Bearer <token>"`)
capName = flag.String("cap", string(tsnetid.CapGroups), "tailnet capability carrying the caller's groups")
ttl = flag.Duration("ttl", signer.DefaultTTL, "token lifetime")
rotate = flag.Duration("rotate", 24*time.Hour, "signing key rotation interval")
overlap = flag.Duration("overlap", time.Hour, "how long a retired key still verifies")
resync = flag.Duration("resync", 5*time.Second, "how often to re-read routes")
keyTags = flag.String("key-tags", "", "comma-separated tags for an auth key minted at startup")
shareCM = flag.String("share-keys", "", "publish verification keys to this ConfigMap")
plaintext = flag.Bool("plaintext", false, "serve HTTP instead of HTTPS")
jwksTLS = flag.Bool("jwks-tls", false, "serve the key set over HTTPS, using this node's own certificate")
)
flag.Parse()
if *waitTimeout < 0 || *waitTimeout > 2*time.Minute || *waitPending <= 0 {
return errors.New("backend waiting requires a timeout between 0 and 2m and a positive pending limit")
}

ns, name, ok := strings.Cut(*gatewayRef, "/")
if !ok || ns == "" || name == "" {
Expand Down Expand Up @@ -184,8 +189,20 @@ func run() error {
// The route table starts empty. That is not an error state: a Gateway
// with no routes yet answers 503, which tells a caller to retry rather
// than that the path is wrong.
var gate *proxy.WaitGate
var requestGate proxy.RequestGate
if *waitTimeout < 0 {
return errors.New("-backend-wait-timeout must not be negative")
}
if *waitTimeout > 0 {
gate, err = proxy.NewWaitGate(proxy.WaitConfig{Timeout: *waitTimeout, Interval: 250 * time.Millisecond, MaxPending: *waitPending, MaxBackends: 256})
if err != nil {
return err
}
requestGate = gate
}
dyn, err := proxy.NewDynamic(proxy.Config{
Signer: sg, Header: *header, Bearer: *bearer,
Signer: sg, Header: *header, Bearer: *bearer, Gate: requestGate,
}, nil)
if err != nil {
return err
Expand Down Expand Up @@ -286,6 +303,9 @@ func run() error {
})
// Fixed, small payloads (JWKS and a status line): bound the whole
// request and response, not just the header, against a slow client.
if gate != nil {
ops.Handle("/metrics", gate)
}
opsSrv := &http.Server{
Addr: *jwksListen,
Handler: ops,
Expand Down
14 changes: 7 additions & 7 deletions gateway/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ require (
github.com/huin/goupnp v1.3.0 // indirect
github.com/jsimonetti/rtnetlink v1.4.1 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.19.1 // indirect
github.com/klauspost/compress v1.20.2 // indirect
github.com/mdlayher/netlink v1.7.3-0.20250113171957-fbb4dce95f42 // indirect
github.com/mdlayher/socket v0.5.0 // indirect
github.com/mitchellh/go-ps v1.0.0 // indirect
Expand Down Expand Up @@ -86,14 +86,14 @@ require (
go.yaml.in/yaml/v3 v3.0.4 // indirect
go4.org/mem v0.0.0-20240501181205-ae6ca9944745 // indirect
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
golang.org/x/crypto v0.54.0 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
golang.org/x/net v0.57.0 // indirect
golang.org/x/net v0.60.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/term v0.45.0 // indirect
golang.org/x/text v0.40.0 // indirect
golang.org/x/sync v0.23.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/term v0.46.0 // indirect
golang.org/x/text v0.42.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 // indirect
golang.zx2c4.com/wireguard/windows v0.5.3 // indirect
Expand Down
11 changes: 11 additions & 0 deletions gateway/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,8 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/compress v1.20.2 h1:v5tiZvm1KGLyg5d5oXdXe3lsSPrbEUiKoXGia//I7Qo=
github.com/klauspost/compress v1.20.2/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/kortschak/wol v0.0.0-20200729010619-da482cc4850a h1:+RR6SqnTkDLWyICxS1xpjCi/3dhyV+TgZwA6Ww3KncQ=
github.com/kortschak/wol v0.0.0-20200729010619-da482cc4850a/go.mod h1:YTtCCM3ryyfiu4F7t8HQ1mxvp1UBdWM2r6Xa+nGWvDk=
github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8=
Expand Down Expand Up @@ -295,6 +297,8 @@ go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBs
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f h1:W3F4c+6OLc6H2lb//N1q4WpJkhzJCK5J6kUi1NTVXfM=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f/go.mod h1:J1xhfL/vlindoeF/aINzNzt2Bket5bjo9sdOYzOsU80=
golang.org/x/exp/typeparams v0.0.0-20240314144324-c7f7c6466f7f h1:phY1HzDcf18Aq9A8KkmRtY9WvOFIxN8wgfvy6Zm1DV8=
Expand All @@ -305,18 +309,25 @@ golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
Expand Down
16 changes: 16 additions & 0 deletions gateway/provision.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package gateway
import (
"fmt"
"strings"
"time"

appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
Expand Down Expand Up @@ -191,6 +192,11 @@ func DataPlaneServiceAccount(gw *gwapi.Gateway) *corev1.ServiceAccount {
// So it is per Gateway, and defaults off.
const JWKSOverTLSAnnotation = "tsjwt.dev/jwks-tls"

// BackendWaitAnnotation opts one Gateway into bounded backend waiting. The
// value is a positive Go duration of at most two minutes. This enables demand
// metrics but never creates an autoscaler or changes application replicas.
const BackendWaitAnnotation = "tsjwt.dev/backend-wait-timeout"

// KeysRole lets one Gateway's data planes publish their verification keys,
// and nothing else.
//
Expand Down Expand Up @@ -257,6 +263,13 @@ func StatefulSet(gw *gwapi.Gateway, cfg Config, tenantsConfigMap string) (*appsv
if cfg.Tag == "" {
return nil, fmt.Errorf("gateway: Config.Tag is required")
}
wait := gw.Annotations[BackendWaitAnnotation]
if wait != "" {
d, err := time.ParseDuration(wait)
if err != nil || d <= 0 || d > 2*time.Minute {
return nil, fmt.Errorf("gateway: %s must be a positive duration no greater than 2m", BackendWaitAnnotation)
}
}
n := names{gw}
replicas := int32(1)

Expand Down Expand Up @@ -284,6 +297,9 @@ func StatefulSet(gw *gwapi.Gateway, cfg Config, tenantsConfigMap string) (*appsv
"-share-keys=" + n.keys(),
}

if wait != "" {
args = append(args, "-backend-wait-timeout="+wait)
}
env := []corev1.EnvVar{
{Name: "POD_NAME", ValueFrom: &corev1.EnvVarSource{
FieldRef: &corev1.ObjectFieldSelector{FieldPath: "metadata.name"}}},
Expand Down
37 changes: 37 additions & 0 deletions gateway/wait_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package gateway_test

import (
"github.com/bitcomplete/tsjwt/gateway"
"strings"
"testing"
)

func TestBackendWaitIsOptInAndValidated(t *testing.T) {
for _, tc := range []struct {
value string
valid bool
}{
{"", true}, {"60s", true}, {"2m", true}, {"0s", false}, {"-1s", false}, {"3m", false}, {"true", false},
} {
t.Run(tc.value, func(t *testing.T) {
gw := gatewayWithClass("edge", "infra", "tsjwt")
gw.Annotations = map[string]string{gateway.BackendWaitAnnotation: tc.value}
sts, err := gateway.StatefulSet(gw, gateway.Config{Image: "img", CredentialSecret: "creds", Tag: "tag:example"}, "tenants")
if (err == nil) != tc.valid {
t.Fatalf("error %v, valid %v", err, tc.valid)
}
if err != nil {
return
}
found := ""
for _, arg := range sts.Spec.Template.Spec.Containers[0].Args {
if strings.HasPrefix(arg, "-backend-wait-timeout=") {
found = strings.TrimPrefix(arg, "-backend-wait-timeout=")
}
}
if found != tc.value {
t.Fatalf("wait arg %q, want %q", found, tc.value)
}
})
}
}
Loading
Loading