Skip to content

feat: Add secret resolution system design for managed secrets - #750

Open
induwara-yaala wants to merge 36 commits into
developfrom
feature/749-api-resolve
Open

induwara-yaala wants to merge 36 commits into
developfrom
feature/749-api-resolve

Conversation

@induwara-yaala

@induwara-yaala induwara-yaala commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Description

Add secret resolution system design for managed secrets

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update
  • CI/CD update
  • Other (please describe):

Changes Made

Add secret resolution system design for managed secrets

Testing

  • Unit tests pass locally
  • Integration tests pass locally
  • Manual testing completed
  • New tests added for changes

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The design has unresolved scope, API, security, concurrency, error-contract, and deployment requirements.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity · 6 Medium severity · 1 Low severity

Open (8)
What changed in this PR

This PR adds a Stage 1 design for managed secret resolution with pluggable providers, SSM, environment fallback, caching, and AWS deployment wiring.

Changes:

  • Defines manager/provider APIs, configuration, injection, and error handling.
  • Specifies caching, IAM/Terraform integration, examples, and testing.
  • Documents scope, non-goals, and open questions.
File Description
docs/​specs/​749-secret-resolution/​design.md Stage 1 managed-secret resolution system design

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/specs/749-secret-resolution/design.md Outdated
Comment thread docs/specs/749-secret-resolution/design.md Outdated
Comment thread docs/specs/749-secret-resolution/design.md Outdated
Comment thread docs/specs/749-secret-resolution/design.md Outdated
Comment thread docs/specs/749-secret-resolution/design.md Outdated
Comment thread docs/specs/749-secret-resolution/design.md Outdated
Comment thread docs/specs/749-secret-resolution/design.md Outdated
Comment thread docs/specs/749-secret-resolution/design.md Outdated
…he TTL constraints, and concurrency safety

- Enforced key grammar (`^[a-z][a-z0-9_]*$`) to validate at the API boundary.
- Defined behavior for `cache_ttl`: `> 0` caches, `0` disables, `< 0` errors.
- Clarified synchronization with a resolution-wide lock for single-flight cold key fetch.
- Enhanced documentation on Terraform opt-in (`ssm_enabled`) with least-privilege policies for serverless and containerized deployments.
- Updated examples to demonstrate SSM integration for both deployment modes.
- Introduced `agentkernel/secret/` package with layered secret resolution: cache, provider, and environment-variable fallback.
- Documented core classes, factory patterns, and provider implementations (`noop`, `ssm`).
- Enhanced error handling with distinct errors for backend failures and missing secrets.
- Updated `StarburstManager` to consolidate environment variable reads at initialization.
- Elaborated on consumer compatibility, concurrency, and deployment configuration.
…ownership

- Updated secret key grammar to uppercase format (`^[A-Z][A-Z0-9_]*$`) and validated at API boundary.
- Clarified provider-specific addressing, removing path composition from manager.
- Adjusted `inject` behavior to write keys verbatim to `os.environ`.
- Documented updated `awssm` provider with explicit prefix handling and lodash-case transformation.
- Enhanced error handling clarity for empty prefixes and credential requirements.
…back

- Detailed iteration-based plan for introducing layered secret resolution.
- Covers configuration, capability core, provider support (`noop`, `in_memory`, `env`, `awssm`), testing, and integration.
- Documents Terraform changes and SSM integration for serverless and containerized deployments.
- Updates examples, CI matrix, and cross-component tests for functional validation.
- Includes steps to synchronize dev skills, bundled user skills, and docs for secret management.
- Updated implementation plan and spec to prioritize environment variables over cache and provider.
- Adjusted terminology from `awssm` to `aws_ssm` for consistency.
- Enhanced examples, CI, and test coverage to reflect clear environment-first behavior.
- Improved concurrency handling with distinct environment, cache, and provider locks.
…nd caching

- Introduced `_SecretConfig` and `_SecretProviderConfig` classes for secret resolution.
- Added configuration validation, default provider (`env`), and cache TTL support.
- Extended `AKConfig` to include `secret` configuration.
- Added tests to validate default behavior, environment overrides, and validation constraints.
- Implemented `SecretManager` for layered resolution: environment, cache, provider.
- Added `SecretProviderFactory`, defaulting to `EnvSecretProvider` backend.
- Introduced `SecretCache` with TTL configuration; supports eviction and invalidation.
- Added unit tests to cover concurrency, cache behavior, environment overrides, and provider specifics.
- Established error handling distinctions for provider failures (`SecretError`) and missing secrets (`SecretNotFoundError`).
- Introduced `SecretProviderContract` in a new `testing` module for reusable contract validation.
- Added tests for `EnvSecretProvider` and `_DictSecretProvider` to ensure compliance.
- Verified that `testing` module and pytest dependencies are excluded from main imports (`agentkernel.secret`).
…ctory/tests

- Introduced `AWSSMSecretProvider` supporting AWS SSM Parameter Store as a managed secret store.
- Updated `SecretProviderFactory` to support `aws_ssm` type with validation for prefix configuration.
- Ensured lazy and singleton initialization of SSM clients for efficient use.
- Added comprehensive tests for `AWSSMSecretProvider` contract compliance, error handling, and configuration validation.
- Included factory-level tests for building `AWSSMSecretProvider` and validating dependency requirements.
… modules

- Added `ssm_enabled` variable to enable read-only access to SSM Parameter Store secrets with scoped IAM permissions.
- Updated Lambda functions to inject `AK_SECRET__PREFIX` into environment variables when `ssm_enabled` is true.
- Applied changes across handler modules (request, response, agent, WebSocket connection) for consistent secret resolution.
- Enhanced documentation and state management to include `ssm_enabled` configuration and behavior.
…ules

- Added `ssm_enabled` variable to grant scoped IAM permissions for accessing SSM Parameter Store.
- Updated ECS task roles for REST service and agent runner to support `ssm_enabled`.
- Injected `AK_SECRET__PREFIX` into environments when `ssm_enabled` is enabled.
- Enhanced Terraform with IAM policy for parameter access and updated documentation to reflect changes.
…ocumentation with detailed instructions for secret setup, resolution order, and key rotation.
…ate documentation for SSM Parameter Store integration
- Introduced `config.yaml` to define secret provider and cache TTL configuration.
- Added `build.sh` for environment caching and syncing dependencies.
- Included `uv.lock` with detailed dependency list for reproducibility.
…able and wildcarding ARN

- Updated all resource ARNs to wildcard the account ID (`arn:aws:ssm:<region>:*:parameter/...`) for SSM access.
- Removed unnecessary `account_id` variables from all modules, streamlining configuration.
- Updated documentation to reflect the changes in SSM permissions and module setup.
@induwara-yaala
induwara-yaala marked this pull request as ready for review September 23, 2026 15:36
@amithad
amithad requested a lite review from Copilot September 23, 2026 15:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Critical IAM scope, deployment secret injection, lock reproducibility, and validation issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 9 High severity · 1 Medium severity · 4 Low severity

Open (14)
Resolved since last review (8)

Comment thread ak-deployment/ak-aws/containerized/modules/agent-runner/main.tf Outdated
Comment thread ak-deployment/ak-aws/containerized/modules/rest-service/main.tf Outdated
Comment thread ak-deployment/ak-aws/serverless/modules/agent-runner/main.tf Outdated
Comment thread ak-deployment/ak-aws/serverless/modules/request-handler/main.tf Outdated
Comment thread ak-deployment/ak-aws/serverless/modules/response-handler/main.tf Outdated
Comment thread ak-py/src/agentkernel/secret/manager.py Outdated
Comment thread docs/specs/749-secret-resolution/plan.md Outdated
Comment thread docs/specs/749-secret-resolution/plan.md
Comment thread docs/specs/749-secret-resolution/spec.md
Comment thread docs/specs/749-secret-resolution/spec.md

@amithad amithad left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall: solid, spec-driven implementation. The secret/ package, the _SecretConfig block, the Terraform wiring and the tests match the approved design and the house patterns (pluggable by default, justified config fields, classes not scripts, secret/ imports core only). The main gap is that the spec set no longer describes what shipped for the examples and CI. CI: all 64 checks pass.

Spec verdict (reviewed before the code)

  • design.md, spec.md and plan.md all reviewed. Every path:line, default and count cited against the base branch verified (the 34 main.tf files, every config.py, Terraform module and CI line reference), with one exception: the "70 models" count is 68 (nit inline).
  • Cross-document consistency holds: spec.md covers every design.md requirement and plan.md covers every spec.md component. The one break is the examples/CI decision, which changed during implementation and is now recorded in the code, READMEs and skills but in none of the three documents (inline on spec.md:811).
  • Design approval staging and the scope narrowing vs #749 were already raised by Copilot and are not repeated here.

Spec conformance (implementation)

  • Implemented as specified: SecretProvider ABC, concrete SecretManager with current()/reset(), SecretCache (monotonic TTL, lock-free reads, guarded eviction), SecretProviderFactory with require_extra + resolve_dotted, env/aws_ssm providers, SecretProviderContract; _SecretConfig with exactly the three justified fields and no enabled/manager type; no module-level functions and nothing outside secret/ imports it; ssm_enabled on both roots, count-gated, account-scoped ssm:GetParameter per tier, AK_SECRET__PREFIX injected, provider type never injected; docs and skills sync done except the new dev skill.
  • Deviated (inline): the two AWS examples drop the OPENAI_API_KEY injection instead of keeping it optional; CI now seeds SSM where the spec says "no CI change"; the ak-dev-new-secret-provider skill required by plan.md Iteration 9 is absent.
  • Beyond the spec: examples/cli/openai_secret and its e2e matrix entry (useful and green in CI). Worth one line in spec.md Examples and plan.md Iteration 7 so the plan matches the PR.

Findings: 0 blockers, 4 suggestions, 2 nits, 1 question (all inline).

Not anchorable to a diff line

  • [suggestion] Tests: spec.md Testing and plan.md Iteration 8 step 1 promise "the environment wins for every provider" and "an empty variable reaches aws_ssm" driven through the manager with the fake SSM client recording zero or one get_parameter call. Only the dict-provider variants exist (test_environment_wins_over_cache_and_provider, test_empty_environment_variable_is_a_miss). Two small tests in test_secret_manager.py using the _FakeSSMClient fixture would close it.
  • [suggestion] Docs-site React pages: README, intro.md and the docs pages now list the capability, but docs/src/pages/index.tsx (ak-add-capabilities pills) and features.tsx (Core Capabilities cards) do not. Scheduling and threads have no card either, so this is optional; a Secrets pill on the ak-add-capabilities entry is the cheapest consistency fix.

Positives

  • _KEY_PATTERN.fullmatch and the ${var.account_id}-scoped ARNs address the earlier Copilot findings.
  • The _DictSecretProvider test double is itself held to SecretProviderContract; the cache eviction race has a deterministic test; no secret value can reach a log record or exception message; the CI seed passes the value via --cli-input-json file://, never argv.

Skipped as duplicates of existing feedback: design approval staging (spec.md:17), scope narrowing vs #749 (design.md:167), Stage-1 implementation detail (design.md:88).

Comment thread docs/specs/749-secret-resolution/spec.md Outdated
Comment thread docs/specs/749-secret-resolution/spec.md Outdated
Comment thread docs/specs/749-secret-resolution/plan.md Outdated
Comment thread examples/aws-containerized/openai-dynamodb/uv.lock Outdated
Comment thread docs/docs/advanced/secrets.md Outdated
Comment thread .github/INTEGRATION_TESTS.md
…ssm example to remove direct OpenAI API key injection

- Introduced `ak-dev-new-secret-provider` skill with step-by-step guidance for extending secret resolution.
- Updated AWS examples to exclusively use AWS SSM for OpenAI API key resolution, eliminating Terraform state secrets.
- Added CI seeding for SSM parameters in affected examples to validate the SSM path during tests.
- Adjusted CI roles for `ssm:PutParameter` permission and documented changes in READMEs and dev skills
# Conflicts:
#	.agents/skills/ak-dev-architecture/SKILL.md
#	docs/docs/core-concepts/configuration.md
Comment thread .github/test-config.yaml
# Conflicts:
#	ak-py/src/agentkernel/skills/ak-add-capabilities/evals/evals.json
@amithad amithad assigned induwara-yaala and unassigned amithad Sep 28, 2026

This branch was successfully deployed

1 active (outdated) deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants