Repository navigation
Conversation
A green 'Tests skipped' job records it on the run, and Finalize Deploy still prunes images. main ignores the flag.
Run workflow on dev with 'rollback' ticked skips the build and runs the existing rollback job, which restores the previously live image. The run stays green when the rollback succeeds.
ci(deploy): SKIP_DEV_TESTS flag and a dev rollback button
The fallback https://api.civicdataspace.in does not resolve, so every download_url in DCAT/Croissant/Dublin Core exports was a dead link when PUBLIC_API_URL was unset. Use https://api.datakeep.civicdays.in, the prod API host, in settings, the export adapter and .env.example. Closes #238
fix(api): point PUBLIC_API_URL fallback at the live prod API
The export view caught every ValueError around export_dataset() and sent str(exc) back with a 400, so internal errors (crosswalk, dates, rdflib) reached anonymous users verbatim and were reported as client errors. Validate standard and format against export_options() before exporting and answer with a fixed message plus the options. Every other failure now falls to the logged generic 500. Closes #240
fix(api): stop returning exception text from metadata export
DVC never worked: uploaded files were never under DVC_REPO_PATH, so every DVC call raised and createMajorVersion always failed. Remove the manager, the setup command, version-change detection and the post_save signal. createMajorVersion now writes the ResourceVersion row directly inside transaction.atomic(), behind IsAuthenticated plus a check that the caller owns the dataset or may change it. BREAKING CHANGE: the setup_dvc management command and DVC_* settings are gone.
Media moves to S3 only, through django-storages. Two storages share one bucket under <STORAGE_ENV>/public and <STORAGE_ENV>/private: public assets get unsigned, cacheable URLs; private files get 5-minute presigned URLs. The default storage is private, so anything that falls back to it is never world-readable by accident. Empty access keys use the boto3 credential chain (ECS/EC2 task role). boto/urllib3 loggers are capped at WARNING because at DEBUG they log every signed request, Authorization header included. Tests swap all storages for InMemoryStorage and never reach S3.
Every FileField/ImageField now names a storage: logos, cover images, profile pictures and publication block files go to public storage; resource files and chart images go to private storage. The storages are callables, so the migrations record a function reference, not bucket settings. upload_to paths drop their "files/public/" and "files/" prefixes, which only repeated what MEDIA_ROOT used to add; the storages already put every key under <env>/public or <env>/private.
S3 has no FieldFile.path, so every reader that used it would raise NotImplementedError once media left local disk. - local_copy() yields a temporary local copy (keeping the extension) for pandas, libmagic and the YAML/XML parsers; data indexing, entry counts and MIME validation use it. - The lru_cache on the loaders is dropped: it keyed on temp paths that are deleted on exit, so it only held DataFrames in memory. - Downloads stream with FileResponse instead of reading whole objects. - Charts only check that the file exists; they read the indexed table. - Search documents index the storage URL instead of a rewritten disk path.
Links issued before the move pointed at /public/<name> on disk. That route now redirects to the file's public S3 URL, after stripping the old "files/public/" or "files/" prefix from the name. Only public files redirect. The old static() route also served resources and drafts whenever DEBUG was on; anything outside public storage, or a name that tries to traverse paths, is a 404 here.
Rows created on disk carry a "files/public/" or "files/" prefix that the S3 keys no longer use. For each stored name the command puts the file at the normalised key in the storage its field names, taken from the old S3 key or, with --source, from the old disk tree, then repoints the row. Dry run by default; --execute copies and renames. Safe to re-run (names already in place are skipped) and deletes nothing.
Regression tests for the local-disk/DVC to S3 migration: S3 storage options (unsigned public URLs, 5-minute presigned private URLs, private default), which storage each file field names, local_copy() and MIME checks on storage without paths, prefix-free upload names, the legacy /public redirect (public files only, no traversal), createMajorVersion's owner-only version write, and migrate_media_to_s3 (dry run, disk and old S3 sources, missing files, failed copies).
Replace the DVC references in SECURITY.md and AGENTS.md with the S3 storage layout and access rules, and add the no-co-author commit rule to AGENTS.md.
Agents kept tripping over setup the file never mentioned (.env and logs/ are needed before tests boot; pre-commit pins python3.11) and over S3-era rules that lived only in code review. Add a quick start, a commands table with the SDK test command, task checklists for mutations, model changes and bug fixes, a files-and-media section (no FieldFile.path, local_copy, FileResponse, public vs private storage), and git safety rules (self-contained commits, no AI co-author trailer, never stash the user's work). Existing rules are unchanged.
CivicDataLab's scrapers fill "Source identifier" on every dataset they upload (dataspace-scrapers/<scraper_id>/<dataset_key>) and read it back to recognise their own datasets, so a dataset is never created twice and their upload records can be rebuilt from DataSpace. The data migration creates the item, or switches it back on if an admin switched it off; reversing it deletes nothing. Tests: tests/test_source_identifier_migration.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fier feat(metadata): add source identifier dataset metadata item
…blications Add useCasesTable, collaborativesTable, aiModelsTable and publicationsTable, shaped like datasetsTable: client filters and sort options limited to an allowlist per module, limit/offset paging, the total after filters, and counts per status for the dashboard tabs (ignoring the status filter so tab labels stay stable). Each query shows the same rows as the module's existing list query, plus published (for AI models: public and active) rows when includePublic is set. The existing list queries are unchanged. StatusCount and status_counts() are shared in api/utils/qs_utils.py. No migration.
The tab counts in datasetsTable were an inline copy of what status_counts() in api/utils/qs_utils.py now does for the other four tables. Call the helper instead, so all five tables count statuses in one place. status_counts() takes an optional item_type so the dataset table keeps its DatasetStatusCount type; the GraphQL schema is unchanged.
feat(api): table queries for use cases, collaboratives, AI models, publications
The charts dashboard loads every built chart and chart image through getChartData and searches them in the browser. chartsTable returns one page of both kinds in a single list, with the total count and per-status counts, using the shared table helpers. Both tables are merged in one SQL UNION, so counting, sorting and slicing happen in the database; only the rows of the page are then loaded. Visibility and permission are the same as getChartData. Rows keep their types (TypeResourceChart, TypeResourceChartImage), so existing fragments work unchanged. getChartData is untouched.
Sorting and paging now go through the same helper as the other table queries. Only filtering stays per table, because Django cannot filter after a UNION. Drops the extra id tie-breaker, matching the others.
feat(api): add chartsTable with filters, sorting and pagination
This commit introduces a new Event model with fields for name, location, dates, and status, along with a corresponding GraphQL schema for creating, updating, and querying events. The schema includes input types for event creation and updates, as well as queries for retrieving events based on various filters. Additionally, utility functions for handling Luma embeds and event permissions are implemented to ensure proper validation and access control.
An event whose owner was deleted (user set to NULL) and has no organization was reported as individual by the API but not by Event.is_individual_event. Use the same check in both.
Each event gets a 12-character public_id; its link is the name words plus that id (urlSlug), so renames never break links and two events can share a name. eventBySlug finds the event by the id at the end of the link. The slug is no longer unique and the counter loop is gone. Also drops the 0055_event dependency on 0049_merge_publication_and_accessmodel, which does not exist.
feat(api): permanent public links for all modules
Resource slugs were unique and the free one was picked before the insert, so two uploads of the same file name at the same moment could clash. Nothing looks a resource up by slug, so the unique rule is dropped. Fixes #229
don't index summary for now
Summaries, AI model and publication descriptions, and user bios have no effective length limit, but their `raw` sub-fields used the keyword analyzer, which emits the whole value as one token. Lucene rejects tokens over 32,766 bytes, so Elasticsearch refused the whole document and the record never reached search. Nothing queries these sub-fields, so they are removed. The fields stay indexed and returned in results. This also makes the workaround in 4ddee76 unnecessary, so summary goes back into the use case search fields. Existing indexes keep the old mapping until rebuilt, and sync_search_mappings never removes fields. After deploying, each environment needs: python manage.py search_index --rebuild -f --models api.UseCase \ api.Collaborative api.AIModel api.Publication api.Organization \ authorization.User
Implement S3 storage, event model, and CI improvements
amit0539
had a problem deploying
to
development
November 4, 2026 09:24 — with 
GitHub Actions
Failure
GitHub Actions
Failure
Bring in main's history, keeping this branch's code wherever the two differ. main contains 01bf459 (Revert "Dev"), which would have silently removed the dataset data API, along with its SDK methods, docs and tests, and undone formatting and type fixes. Those changes are not carried over, so 01bf459 is in the history but has no effect on this branch's code. Conflicts were resolved to this branch: main still had the removed DVC versioning and lacked the publicId and url_slug fields. Kept from main: - PromptFormat conversion for prompt_format in TypeResource - executable bit on docker-entrypoint.sh - django-debug-toolbar 6.0.0 and isort 7.0.0
Bump django-debug-toolbar and isort to latest major versions
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| count=count, | ||
| ) | ||
| except DataFetchError as e: | ||
| return Response({"error": str(e)}, status=400) |
amit0539
had a problem deploying
to
development
November 4, 2026 09:50 — with
GitHub Actions
Failure
GitHub Actions
Failure
amit0539
had a problem deploying
to
development
November 4, 2026 12:19 — with
GitHub Actions
Failure
GitHub Actions
Failure
fix(api): stop same-name uploads failing on the resource slug rule
anantjain341
had a problem deploying
to
development
November 4, 2026 15:07 — with
GitHub Actions
Failure
GitHub Actions
Failure
…opies Removing the keyword-analyzed raw sub-fields changed the documents only; existing indexes kept the old mapping, so a long use case summary still failed to index on save and every deploy failed at search_index --populate. Migration 0064 drops each search index whose live mapping still has such a sub-field and creates it again with the current mapping; the deploy then refills it. Also removes the same sub-field from dataset description. Fixes #266
fix(search): rebuild indexes that still have single-token long text copies
Dev added its own edit check to createMajorVersion. Keep the shared ChangeResourcePermission there instead, so resource versioning follows the same dataset edit rule as every other resource mutation.
fix(api): add permission checks to dataset resources, charts and access models
Adds migration 0065 joining main's 0054_rebuild_long_text_search_indexes with dev's 0064, so dev to main releases keep a single migration leaf.
This was referenced Nov 5, 2026
chore: merge main back into dev after the long summary hotfix
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
Brief description of the changes introduced by this PR.
Type of Change
Please delete options that are not relevant.
Related Issues
Fixes #(issue number)
Closes #(issue number)
Relates to #(issue number)
Changes Made
Testing
Test Instructions
Please describe how reviewers can test your changes:
Screenshots (if applicable)
Add screenshots to help explain your changes.
Checklist
black .,isort ., andflake8and fixed any issuesAdditional Notes
Add any additional notes or context about the PR here.