feat: organization-scoped event custom fields with management UI and filter integration - #1815
feat: organization-scoped event custom fields with management UI and filter integration#1815ejsmith wants to merge 32 commits into
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds custom field support for events by updating the event model and repository logic, removing legacy query visitor code, and updating test infrastructure to support the new behavior.
- Removed obsolete EventFieldsQueryVisitor usage and file.
- Introduced automatic custom field creation in the event repository.
- Updated the PersistentEvent model to implement virtual custom fields and modified test configurations accordingly.
Reviewed Changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/Exceptionless.Tests/Search/PersistentEventQueryValidatorTests.cs | Removed legacy query visitor call in tests. |
| tests/Exceptionless.Tests/Search/EventIndexTests.cs | Updated repository query to include organization filtering. |
| tests/Exceptionless.Tests/Migrations/FixDuplicateStacksMigrationTests.cs | Set log level to Trace for migration tests. |
| tests/Exceptionless.Tests/AppWebHostFactory.cs | Added Kibana container configuration. |
| src/Exceptionless.Core/Repositories/Queries/Visitors/EventFieldsQueryVisitor.cs | Removed unused visitor implementation. |
| src/Exceptionless.Core/Repositories/EventRepository.cs | Added auto-creation of custom fields and related tenant logic. |
| src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs | Removed legacy dynamic mapping for event index custom fields. |
| src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs | Added index creation for custom fields. |
| src/Exceptionless.Core/Models/PersistentEvent.cs | Updated Idx property type and implemented IHaveVirtualCustomFields. |
Files not reviewed (1)
- src/Exceptionless.Core/Exceptionless.Core.csproj: Language not supported
niemyjski
changed the title
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…filter integration Add organization-level custom field management that allows premium users to define typed, searchable event fields (keyword, string, numeric, date, boolean) with full lifecycle management including soft-delete, index slot reuse, and saved view protection. Key changes: - Custom field CRUD on organization controller with validation and plan gating - EventCustomFieldService for document change handling and field lifecycle - Premium feature detection in saved views (UsesPremiumFeatures flag) - Filter integration with operator support per field type - Angular UI: management dialog, filter picker integration - Svelte UI: custom fields management and filter components - Comprehensive test coverage (107+ custom field tests) - Max 20 active fields per organization, slot-based index allocation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
0a8855a to
b277880
Compare
…seline, fix Prettier - Restore accidentally deleted ControllerManifestTests.cs (was removed during squash) - Generate controller-manifest.json baseline with custom fields endpoints included - Fix Prettier formatting in sidebar.svelte Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…heck Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…finedAsync The previous baseline was missing PostPredefinedAsync because UPDATE_SNAPSHOTS ran against a stale build. Rebuilt and regenerated correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…es, ContainsKey pattern - PersistentEvent.GetCustomFields(): consolidate value-type guard into the .Where() predicate - EventCustomFieldService: use .OfType<PersistentEvent>() to replace the outer .Where() null filter and the redundant inner .Where(d => d is not null) in the document loop - CustomFieldIndexingTests: fix regex type names (integer→int, boolean→bool; add string and float) - CustomFieldApiTests: fix 'integer' index type to 'int' (integer is not a supported type) - CustomFieldApiTests: replace ContainsKey+indexer with TryGetValue for SessionHasError assertion The 'integer' index type bug was silent: ConvertValue returns null for unknown types so the field would be stored but never actually indexed, with no error surfaced to the user. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ring
ConvertToKeyword and ConvertToString were calling value.ToString() without
a format provider. On servers with a non-en-US locale (e.g. de-DE), float,
double, and decimal values produce '1,5' instead of '1.5', breaking
Elasticsearch keyword-field searches because the stored value and the query
value use different decimal separators. DateTime values were also formatted
with the current thread culture instead of ISO 8601.
Fix: introduce a FormatInvariant helper that uses CultureInfo.InvariantCulture
for numeric types and the roundtrip 'O' ISO-8601 format for DateTime/DateTimeOffset.
The bool.ToString() behaviour ("True"/"False") is preserved for backwards
compatibility with already-indexed data.
Add regression tests that explicitly set CultureInfo to de-DE and assert
that the output always uses an invariant decimal separator. Also add tests
for long coercion in ConvertToBool, and for DateTime/DateTimeOffset keyword
formatting.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Focused custom-field and session directionAfter re-reading the current implementation, I think the session compatibility concern is real, but the larger proposed redesign is more machinery than this PR needs. Contract to preserve
Changes required before merge
Scope I do not think is required hereI would not require an administrative preflight feature, a global lifecycle lock, a persisted compatibility cutoff, an eight-stage deployment, new operator policy, or a general backfill/reindex system. Foundatio already serializes slot allocation per entity/tenant/type, and this branch already serializes user creation for quota enforcement. Exact invariant validation plus dual-read compatibility is the proportional fix. Verification needed
This keeps the useful part of the feedback—session continuity and deterministic reserved slots—without turning this PR into a broader custom-field control plane. |
|
Follow-up audit for the focused custom-field and session direction is complete. Feedback disposition
RCA and final fixThe session continuity concern was real: pooled system slots replaced retained legacy session indexes. The branch now centralizes the three system descriptors, validates their exact reserved invariants, and performs legacy/current dual reads with correct positive, exists, missing, range, and negation semantics. The remaining valid issue was the public contract, not another migration framework: automatic arbitrary extended-data indexing was removed, but the PR description and filtering docs still called the change additive and even suggested a
The PR description now reflects the current four endpoints, three system fields, synchronous soft-delete policy, known limitations, and breaking search behavior. Verification
Thermo-nuclear structural review found no PR-caused file crossing 1,000 lines, no justified code-judo simplification beyond the existing small canonical system descriptor/visitor, and no reason to add a general backfill, dual-write, or migration control plane for this accepted one-time cutover. |
# Conflicts: # src/Exceptionless.Web/ClientApp/src/lib/features/events/premium-filter.test.ts # src/Exceptionless.Web/ClientApp/src/routes/(app)/+layout.svelte # tests/http/organizations.http
# Conflicts: # src/Exceptionless.Web/ClientApp/src/routes/(app)/+layout.svelte
# Conflicts: # src/Exceptionless.Web/ClientApp/src/lib/features/shared/components/faceted-filter/faceted-filter-number.svelte.test.ts
|
Summary
Adds explicit, opt-in, organization-scoped typed event custom fields backed by pooled Elasticsearch slots, with API, MCP, saved-view, and Svelte management/filter integration.
This remains a forward-only indexing cutover. User fields are indexed only after a definition exists; historical events are not backfilled. The only dual-read compatibility fields are Exceptionless-owned
@ref:session,sessionend, andhaserror.Production hardening
AppFilterscopes and preserves the explicit free-organization global-admin path throughApiFilterPolicy.Customer regression
Focused HTTP ingestion coverage configures
DatabaseVersionaskeyword, posts both"4.90"and"4.90 build 1234 30-Aug-2024", reloads Elasticsearch_source, verifies exactData/pooledIdxstrings, and proves4.90matches while4.9does not. Adoubledefinition intentionally converts"4.90"to4.9and skips the development string without altering rawData.Upgrade and rollout
Verification at
e53fbcce1EventCustomFieldServiceTests: 71/71 passed.PersistentEventCustomFieldsTests: 14/14 passed.git diff --checkpassed.Local Docker/Aspire dogfood was unavailable because Docker Desktop was unhealthy. GitHub CI provides the Elasticsearch-backed ingestion/query and Aspire browser proof; representative retained-index startup timing remains a staging gate.
Production blockers / known limitations
Foundatio.Repositories.Elasticsearch8.0.2-preview.elasticsearch-mapping-issue-investigation.0.3. An official immutable release must contain the required mapping behavior and later large-mapping fix, then this PR must pin that release.Breaking behavior
Arbitrary extended-data fields without an active organization definition are no longer automatically indexed or searchable through
data.*. Existing saved views and integrations relying on automatic user-field indexing need a one-time definition rollout. Existing API shapes remain compatible.