feat(playground): run ExtendDB on DocumentDB - #454
Open
xgerman wants to merge 3 commits into
Open
Conversation
Adds documentdb-playground/extenddb/, a playground demonstrating ExtendDB (open-source DynamoDB wire-protocol server) configured with its MongoDB storage backend pointed at a DocumentDB cluster, giving a DynamoDB-compatible API surface backed by DocumentDB. Includes: - Dockerfile building ExtendDB from source with the mongodb feature (no official Mongo-backend image is published upstream) - Kubernetes manifests: namespace/PVC, one-shot init Job, serve Deployment + Service - scripts/ for build, deploy, connectivity smoke test, and cleanup - demo/demo.py: small boto3 walkthrough of the classic DynamoDB Movies table - README covering architecture, quick start, config, the ddbsh interactive shell, and troubleshooting (notably the directConnection/replicaSet=rs0 interaction with DocumentDB's gateway) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 73ace353-38c7-44b2-8a9f-0ee833727c53 Signed-off-by: German Eichberger <geeichbe@microsoft.com>
…ve testing Fixes found by deploying and testing this playground end-to-end against a real DocumentDB instance (kind cluster) with the actual ExtendDB binary, following up on a rubber-duck review of the original implementation: - deploy.sh: strip replicaSet=rs0 from the connection string (matching lightrag/keda-autoscaling). A prior mongosh-based test suggested keeping it was safe with directConnection=true, but the real ExtendDB Rust mongodb v3 driver strictly validates the requested replica set name and rejects the gateway's response even in direct-connection mode. - deploy.sh: append tlsAllowInvalidCertificates=true -- the Rust driver's default certificate verifier rejects DocumentDB's self-signed gateway cert with UnknownIssuer. - deploy.sh: restrict envsubst substitution to explicit variable lists so it stops blanking out unrelated variable references in the manifests. - deploy.sh: capture full init Job logs with a fallback so admin credentials (printed once) are never silently lost by a failed keyword grep. - manifests/init-job.yaml: extenddb init --backend mongodb reads the Mongo connection string from the --config file itself, not from the connection-string env var override (that override only applies at serve time) -- write a seed TOML first, then run init --overwrite. Use bash instead of sh (dash doesn't support set -o pipefail). - manifests/init-job.yaml, manifests/serve.yaml: set HOME to the PVC state dir so the self-signed TLS cert/key extenddb init generates under the home-relative .extenddb/tls/ path land on the PVC-backed state dir instead of the init Job's ephemeral container home dir, so the separate serve Deployment pod can find them. - manifests/init-job.yaml, manifests/serve.yaml: add securityContext (runAsUser/runAsGroup/fsGroup 10001) so the non-root container can reliably write to the PVC regardless of the CSI driver's default mount ownership. - manifests/init-job.yaml: add ttlSecondsAfterFinished to release the shared ReadWriteOnce PVC promptly. - manifests/serve.yaml: document the readiness/liveness probe caveat -- extenddb healthcheck doesn't verify backend connectivity. - Dockerfile: install libssl3 in the runtime stage defensively. - demo.py: use Decimal instead of float literals -- boto3's DynamoDB resource rejects native Python floats. - README: document a confirmed, currently-blocking upstream compatibility gap -- ExtendDB's MongoDB backend unconditionally uses MongoDB's snapshot read concern for CreateTable/PutItem/etc., which this operator's DocumentDB gateway does not support (Error code 115: CommandNotSupported). CreateTable succeeds; PutItem and most other data-plane operations currently fail with InternalServerError. Also documents the extenddb manage create-account/create-user/ put-user-policy/create-access-key workflow needed to get a SigV4 access key (the init admin credentials are management-API only, not a SigV4 key), and updates the replicaSet/directConnection troubleshooting writeup with the corrected, doubly-verified explanation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 73ace353-38c7-44b2-8a9f-0ee833727c53 Signed-off-by: German Eichberger <geeichbe@microsoft.com>
Configure majority transaction reads, add local-source image builds, and expand the playground smoke test to cover CRUD and transactional operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c9b74922-c628-425e-bf1b-484b223c52b5 Signed-off-by: German Eichberger <geeichbe@microsoft.com>
documentdb-triage-tool
Bot
added
documentation
|
🤖 Auto-triaged by documentdb-triage-tool. Applied: Reasoningcomponent from path globs (playground, docs); effort from diff stats (1235+0 LOC, 12 files); LLM: Adds a new ExtendDB-backed playground with multi-file changes spanning build scripts, Kubernetes manifests, IAM setup, smoke tests, and read-concern configuration. If a label is wrong, remove it manually and ping |
xgerman
marked this pull request as ready for review
September 8, 2026 16:08
xgerman
requested review from
WentingWu666666 and
hossain-rayhan
as code owners
September 8, 2026 16:08
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
There are a few user-facing correctness issues (hardcoded credential secret name and misleading docs/error messages) that can cause confusing failures when running the new playground.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new documentdb-playground/extenddb/ playground that deploys ExtendDB (DynamoDB-compatible API) on top of a Kubernetes-hosted DocumentDB instance, including build/deploy/test/demo workflows and documentation for running it end-to-end.
Changes:
- Introduces ExtendDB image build workflow (remote ref or local working tree) and optional kind image load.
- Adds Kubernetes manifests plus deploy/cleanup scripts to wire ExtendDB to an existing DocumentDB connection string and initialize the backend.
- Adds a DynamoDB API smoke test via AWS CLI and a small boto3 demo, with detailed README guidance (including transaction read concern compatibility).
File summaries
| File | Description |
|---|---|
| documentdb-playground/extenddb/scripts/test-connection.sh | AWS CLI smoke test (CRUD + transactional APIs) via port-forwarded ExtendDB endpoint |
| documentdb-playground/extenddb/scripts/deploy.sh | Deploy script to resolve DocumentDB connection info, init ExtendDB, and start the server |
| documentdb-playground/extenddb/scripts/cleanup.sh | Namespace teardown for the ExtendDB playground |
| documentdb-playground/extenddb/scripts/build-image.sh | Docker build wrapper for the ExtendDB MongoDB-backend image (with optional kind load) |
| documentdb-playground/extenddb/README.md | End-to-end instructions, rationale, and troubleshooting for the ExtendDB-on-DocumentDB setup |
| documentdb-playground/extenddb/manifests/serve.yaml | Deployment + Service for the long-running ExtendDB server (including majority tx read concern) |
| documentdb-playground/extenddb/manifests/namespace.yaml | Namespace + PVC for ExtendDB state/config/TLS material |
| documentdb-playground/extenddb/manifests/init-job.yaml | One-shot Job to run extenddb init --backend mongodb with PVC-backed HOME/config seed |
| documentdb-playground/extenddb/documentdb.yaml | Sample DocumentDB instance manifest for the playground |
| documentdb-playground/extenddb/Dockerfile | Multi-stage image build for ExtendDB with the MongoDB backend enabled |
| documentdb-playground/extenddb/demo/requirements.txt | Python demo dependency list |
| documentdb-playground/extenddb/demo/demo.py | boto3 demo exercising common DynamoDB operations against ExtendDB |
Review details
- Files reviewed: 12/12 changed files
- Comments generated: 6
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+35
to
+43
| if command -v kind >/dev/null 2>&1; then | ||
| # Auto-detect the current kind cluster from the active kube context unless | ||
| # KIND_CLUSTER was set explicitly. | ||
| if [ -z "${KIND_CLUSTER}" ]; then | ||
| CTX="$(kubectl config current-context 2>/dev/null || true)" | ||
| case "${CTX}" in | ||
| kind-*) KIND_CLUSTER="${CTX#kind-}" ;; | ||
| esac | ||
| fi |
Comment on lines
+46
to
+49
| DOCDB_USER=$(kubectl get secret docdb-credentials -n "$DOCUMENTDB_NAMESPACE" \ | ||
| -o jsonpath='{.data.username}' | base64 -d) | ||
| DOCDB_PASSWORD=$(kubectl get secret docdb-credentials -n "$DOCUMENTDB_NAMESPACE" \ | ||
| -o jsonpath='{.data.password}' | base64 -d) |
Comment on lines
+61
to
+66
| SVC_IP=$(kubectl get svc "documentdb-service-${DOCUMENTDB_CLUSTER}" -n "$DOCUMENTDB_NAMESPACE" \ | ||
| -o jsonpath='{.spec.clusterIP}' 2>/dev/null) || true | ||
| if [ -n "$SVC_IP" ]; then | ||
| SVC_DNS="documentdb-service-${DOCUMENTDB_CLUSTER}.${DOCUMENTDB_NAMESPACE}.svc.cluster.local" | ||
| MONGO_URI=$(echo "$MONGO_URI" | sed "s/$SVC_IP/$SVC_DNS/g") | ||
| fi |
Comment on lines
+32
to
+36
| │ ┌──────────────┐ MongoDB wire protocol ┌────────────┐│ | ||
| │ │ ExtendDB │──────────────────────────▶│ DocumentDB ││ | ||
| │ │ (DynamoDB │ connection_string with │ (Gateway) ││ | ||
| │ │ API server) │ replicaSet=rs0 └─────┬──────┘│ | ||
| │ └──────────────┘ │ │ |
Comment on lines
+91
to
+93
| EXTENDDB_ACCESS_KEY_ID=<from step 3> \ | ||
| EXTENDDB_SECRET_ACCESS_KEY=<from step 3> \ | ||
| ./demo/demo.py |
Comment on lines
+57
to
+61
| if not access_key or not secret_key: | ||
| sys.exit( | ||
| "Set EXTENDDB_ACCESS_KEY_ID and EXTENDDB_SECRET_ACCESS_KEY " | ||
| "(printed by ../scripts/deploy.sh after 'extenddb init')." | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
majority, since DocumentDB does not supportsnapshotread concernTransactWriteItems, andTransactGetItemsin the smoke testValidation
CommandNotSupported/ read-concern error 115 failuresbash -npassed for all playground scriptsRelated