Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .agent/plugins/marketplace.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "confit",
"interface": {
"displayName": "ConfIT"
},
"plugins": [
{
"name": "confit",
"source": {
"source": "local",
"path": "./"
},
"policy": {
"installation": "AVAILABLE",
"authentication": "ON_INSTALL"
},
"category": "testing"
}
]
}
17 changes: 17 additions & 0 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"$schema": "https://json.schemastore.org/claude-code-marketplace.json",
"name": "confit",
"description": "ConfIT \u2014 declarative API integration testing for .NET. Skills for setting up a test suite and authoring test definitions from an API contract.",
"owner": {
"name": "Rahul Garg",
"url": "https://github.com/techygarg"
},
"plugins": [
{
"name": "confit",
"description": "Set up a ConfIT suite, write component tests from your controller and mocks, and write API tests from a spec.",
"source": "./",
"category": "testing"
}
]
}
20 changes: 20 additions & 0 deletions .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "confit",
"version": "1.0.0",
"description": "Agent skills for ConfIT \u2014 set up a declarative API test suite, write component tests from your controller and its mocks, and write black-box API tests from an OpenAPI spec.",
"author": {
"name": "Rahul Garg",
"url": "https://github.com/techygarg"
},
"homepage": "https://github.com/techygarg/ConfIT",
"repository": "https://github.com/techygarg/ConfIT",
"license": "MIT",
"keywords": [
"testing",
"api-testing",
"integration-testing",
"component-testing",
"dotnet",
"confit"
]
}
54 changes: 54 additions & 0 deletions .codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
{
"name": "confit",
"version": "1.0.0",
"description": "Agent skills for ConfIT — set up a declarative API test suite, write component tests from your controller and its mocks, and write black-box API tests from an OpenAPI spec.",
"author": {
"name": "Rahul Garg"
},
"homepage": "https://github.com/techygarg/ConfIT",
"repository": "https://github.com/techygarg/ConfIT",
"license": "MIT",
"keywords": [
"testing",
"api-testing",
"integration-testing",
"component-testing",
"contract-testing",
"test-automation",
"openapi",
"swagger",
"graphql",
"wiremock",
"xunit",
"dotnet",
"confit"
],
"skills": "./skills/",
"interface": {
"displayName": "ConfIT",
"shortDescription": "Set up a declarative API test suite, write component tests from your code, and API tests from your spec.",
"longDescription": "ConfIT is a .NET library for declarative API integration testing — tests are YAML or JSON files, not C# code. This plugin gives Codex three skills. Suite Setup wires ConfIT into a project: startup mode (in-process, command/AppLauncher, or integration), suite.config.yaml, the fixture and test class, auth profiles and tag filters. Component Tests is for a developer mid-implementation: it works from the controller plus the mocks behind it, tracing the call path to discover what the service calls, and driving error paths by changing what a mocked dependency returns. Integration Tests is for testing an already-deployed service black box: it works from an OpenAPI spec, a Postman collection or a live endpoint, assumes no access to the service source, and handles environment selection, auth profiles and the shared-persistent-state problem. All three read ConfIT's own CI-verified example suites rather than carrying templates that go stale.",
"developerName": "Rahul Garg",
"category": "Testing",
"capabilities": [
"Interactive",
"Read",
"Write"
],
"defaultPrompt": [
"Set up a ConfIT component test suite for my API.",
"I just implemented this controller — write component tests for it.",
"Discover what outbound calls this endpoint makes and mock them.",
"My component test gets a 404 from the mock — fix the interaction.",
"Here's our OpenAPI spec — write API tests against staging.",
"Convert this Postman collection into ConfIT tests.",
"Point the suite at the QA environment with a bearer token.",
"My ConfIT suite passes once then fails on the second run."
],
"websiteURL": "https://github.com/techygarg/ConfIT",
"privacyPolicyURL": "https://docs.github.com/en/site-policy/privacy-policies/github-general-privacy-statement",
"termsOfServiceURL": "https://docs.github.com/en/site-policy/github-terms/github-terms-of-service",
"brandColor": "#2F6FEB",
"screenshots": []
}
}
28 changes: 28 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,34 @@ ConfIT uses [Semantic Versioning](https://semver.org/).

---

## [3.2.0]

### Added

- **`mock.enableLogs`** — WireMock request logging is now switchable from `suite.config.yaml` instead of only from fixture code. Beyond debugging a stub that will not match, it turns an unknown dependency surface into a listing: run a component test with no `mock:` block and every outbound call is logged as an unmatched request, which is enough to write the interactions from. Works whatever language the service under test is written in, since it observes HTTP rather than code. See [Mock Interactions](doc/mock-interactions.md#discovering-what-a-service-calls).

- **Agent skills** — three [Agent Skills](https://docs.claude.com/en/docs/claude-code/skills) in [`skills/`](skills), distributed as agent plugins: `confit-suite-setup` (wire a suite), `confit-component-tests` (developer, mid-implementation — works from the controller plus the mocks behind it) and `confit-integration-tests` (QA, post-deployment — black box, works from a spec, collection or live endpoint and assumes no access to the service source). See [AI Agent Skills](doc/ai-skills.md).

- **Plugin manifests** — the repository is its own Claude Code marketplace (`/plugin marketplace add techygarg/ConfIT`, then `/plugin install confit@confit`), and carries a Codex manifest. Further agents are onboarded by adding one manifest directory each.

- **`example/README.md`** — maps the three startup modes to their example projects, states what every suite structurally needs, and lists what is demo-specific so none of it is copied into a consuming project.

- **Validation tooling in [`tools/`](tools)** — `check-testcases.py` statically validates test definitions (missing expected bodies, invalid `depends:`, unresolvable `{{variables}}`, `mock:` blocks in an integration suite, unregistered files, and matcher problems such as an unknown name, a missing closing parenthesis, or a wildcard in a `semantic` path). `verify-suite.sh` checks project wiring. Both exit non-zero on error and run in CI via the new `make skills` target, and both read ground truth from `src/ConfIT/` — supported frameworks, built-in matcher names — rather than hardcoding it.

### Changed

- **Skills read ConfIT's own `example/` and `doc/` instead of shipping templates.** Embedded copies of the fixture, `suite.config.yaml` and `.csproj` drifted from the real projects, so they were removed; each skill now resolves the repository it ships inside and reads the suites CI verifies.

### Fixed

- **`example/User.IntegrationTests`** — `TestReader.GetTestsForAFile` was called with its arguments reversed; removed `AuthTokenProvider.cs`, which implemented `IAuthTokenProvider` but was never referenced, since no `SuiteBootstrapper` overload accepts a custom provider (auth is configured declaratively). The `qa` environment now reads its URL from `${QA_API_URL}` instead of a hardcoded host.
- **`example/User.ComponentTests`** — `appsettings.Tests.json` carried an `environmentVariables` key that nothing reads; `UserDbInitializer.Seed()` had a commented-out body and now seeds real reference data, demonstrating the `onStarted` hook.
- **`example/User.ComponentTests.AppLauncher`** — configuration comments referred to a `Startup.SeedDatabase` method that does not exist, and now describe the actual `IsLocalComponentTests` / `UserDbContext` mechanism; the namespace was aligned with the project folder.

Other than `mock.enableLogs`, the library is unchanged — the skills are distributed as agent plugins, not as package content.

---

## [3.1.0]

### Added
Expand Down
39 changes: 38 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@ make test # unit + component tests only
make unit # unit tests only
make component # component tests only
make integration # wipe DB, start services, run integration tests, stop services
make ci # full pipeline: build + unit + component + integration
make skills # validate the agent skills' scripts against every example suite
make ci # full pipeline: build + unit + component + skills + integration
make clean # stop services, remove build artefacts and SQLite DB
make services-stop# kill running User.Api / JustAnotherService processes
make help # list all targets with descriptions
Expand Down Expand Up @@ -337,6 +338,42 @@ The `example/` projects are not just demos — they run in CI as regression gate

---

## Agent Skills

Three agent skills live in `skills/` at the repository root and are documented in
[doc/ai-skills.md](doc/ai-skills.md):

- **`confit-suite-setup`** — wiring a suite: startup mode, `suite.config.yaml`, fixture, test class, auth, filters
- **`confit-component-tests`** — developer, mid-implementation: works from the controller plus the mocks behind it
- **`confit-integration-tests`** — QA, post-deployment: black box, works from a spec/collection, assumes no source

Authoring is split by **persona**, not by feature. The two produce nearly the same artifact but are
different jobs — different input, state model, matcher instinct and achievable test matrix. Do not
merge them back.

The skills are **not** shipped in the NuGet package — they are distributed as agent plugins
only. The repository root doubles as a plugin root (`.claude-plugin/`, `.codex-plugin/`; further
agents get one manifest directory each). Installing a plugin clones the whole repository, which is
what lets the skills read `example/` and `doc/` directly.

`skills/` must stay at the repository root: the manifests name `./skills/`, and
`skills/confit-suite-setup/scripts/reference-path.sh` resolves the reference by finding a
directory holding `example/`, `doc/` and `skills/` as siblings. Work on the skills locally with
`claude --plugin-dir .` rather than copying them anywhere.

**`confit-suite-setup` must not gain template files again.** It was rebuilt to read the live
`example/` suites precisely because embedded copies of the fixture, config and `.csproj` went
stale — `example/` took 21 commits in twelve months. New setup patterns go into `example/`, where
CI runs them, and `example/README.md` records what is structural versus demo-specific.

Shared executables live in `tools/`, not inside a skill: `tools/check-testcases.py` (test
definitions) and `tools/verify-suite.sh` (project wiring). Both must report zero errors against
every suite in `example/` — `make skills` enforces this and runs as part of `make ci`. Each reads
ground truth from `src/ConfIT/` (supported frameworks, built-in matcher names) rather than
hardcoding it, so they do not drift.

---

## Curated References

- [.NET API docs](https://learn.microsoft.com/en-us/dotnet/api/) — official Microsoft reference
Expand Down
16 changes: 14 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ API_PORT := 5170
SVC_PORT := 9999
DB := example/User.Api/User.db

.PHONY: default help build unit component component.applauncher test integration services-start services-stop ci clean
.PHONY: default help build unit component component.applauncher test integration services-start services-stop skills ci clean

default: build test

Expand Down Expand Up @@ -62,8 +62,20 @@ services-stop: ## Kill any running User.Api / JustAnotherService processes
@lsof -ti :$(API_PORT) 2>/dev/null | xargs kill -9 2>/dev/null || true
@lsof -ti :$(SVC_PORT) 2>/dev/null | xargs kill -9 2>/dev/null || true

# ── Agent skills ────────────────────────────────────────────────────────────────
skills: ## Validate the agent-skill tooling against every example suite
@for p in example/User.ComponentTests example/User.IntegrationTests example/User.ComponentTests.AppLauncher; do \
printf " %-42s" "$$p"; \
python3 tools/check-testcases.py "$$p" > /tmp/confit-skills.log 2>&1 \
|| { echo "FAILED"; cat /tmp/confit-skills.log; exit 1; }; \
bash tools/verify-suite.sh "$$p" >> /tmp/confit-skills.log 2>&1 \
|| { echo "FAILED"; cat /tmp/confit-skills.log; exit 1; }; \
echo "ok"; \
done
@echo " ✓ Skill scripts clean on all example suites"

# ── Pipelines ───────────────────────────────────────────────────────────────────
ci: build test component.applauncher integration ## Full pipeline: build + unit + component + integration
ci: build test component.applauncher skills integration ## Full pipeline: build + unit + component + skills + integration

clean: services-stop ## Stop services, remove build artefacts and SQLite DB
@rm -f $(DB)
Expand Down
12 changes: 11 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,15 @@ Component and integration tests share a large common surface — how tests are d

→ **[Test Execution Flow](doc/test-execution-flow.md)** — ASCII flow diagrams showing what happens at runtime across all three suite types.

→ **[AI Agent Skills](doc/ai-skills.md)** — three skills that let a coding agent set up a suite, write component tests from your controller, and write API tests from your spec. Install as a plugin:

```
/plugin marketplace add techygarg/ConfIT
/plugin install confit@confit
```

Codex has its own manifest; more agents are added over time. The skills read the live [`example/`](example) suites rather than carrying templates, so what they show you is what CI proves works.

---

## Documentation
Expand Down Expand Up @@ -71,12 +80,13 @@ Component and integration tests share a large common surface — how tests are d
|---|---|
| [Reading Failure Output](doc/failure-output.md) | Per-field failure messages, path notation, suite summary table, debugging tips |
| [Extending ConfIT](doc/extending-confit.md) | `ITestOutputLogger`, `ITestProcessor` / `ITestProcessorFactory` hooks, custom semantic matchers, `IAuthTokenProvider` |
| [AI Agent Skills](doc/ai-skills.md) | `confit-suite-setup`, `confit-component-tests`, `confit-integration-tests` — plugin install, why authoring splits by persona, plus two standalone validation scripts |

---

## Example Projects

The `example/` directory contains a working reference implementation:
The `example/` directory contains working reference implementations of all three startup modes — see [`example/README.md`](example/README.md) for the mode-to-project map and what is demo-specific:

| Project | Role |
|---|---|
Expand Down
8 changes: 8 additions & 0 deletions doc/Package.Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,14 @@ await Execute(testName, test, sourceFile);

---

## Agent Skills

Two agent skills live in the repository — one for wiring up a suite, one for turning an OpenAPI spec, ASP.NET controller, or GraphQL schema into test definitions. They install as an agent plugin (Claude Code, Codex, more over time) and are not part of this package.

→ [Agent Skills documentation](https://github.com/techygarg/ConfIT/blob/main/doc/ai-skills.md)

---

## Full Documentation

→ [github.com/techygarg/ConfIT](https://github.com/techygarg/ConfIT)
Loading
Loading