feat(clp-s)!: Adds CLP+ as an experimental feature. by davidlion · Pull Request #2275 · y-scope/clp

davidlion · 2026-05-12T03:58:06Z

Description

Adds CLP+ (clpp), an experimental extension of clp-s that parses unstructured text logs into structured, query-able fields using log-surgeon. Enabled via --experimental; existing clp-s behaviour is unchanged when the flag is absent.

CLP+ storage & compression

CLP+ uses a (log-surgeon parsing spec](https://github.com/y-scope/log-surgeon/blob/log-mechanic/rust/docs/parsing-spec-file.md) to parse the unstructured text during ingestion.
- Note that for pure unstructured logs, CLP-S requires the log-converter to first change them to JSON.
The parsing spec contains rules (named regex patterns) that may contain sub-rules (named regex captures). These patterns split each log message into:
- Leaf rule matches: a rule match containing no sub-rule. These are stored in CLP-S columns.
- A log shape: a string containing static text of a log message with the leaf rule matches replaced with %log-surgeon.qualified.leaf_rule_name% placeholders. (Similar to CLP log types.)
- Parent rule matches: a rule match containing at least one sub-rule. These are stored as log shape metadata, where the parent rule matches for each log shape are "views" into the log shape.
The parsing spec is stored in the archive, so extraction and search need only --experimental. Reading a CLP+ archive without it is an error.
int/float leaves are encoded as native numeric columns, enabling range queries.
Some additional statistics are stored to support special "stat" queries (see below).

Search

Extends the KQL grammar with function-call syntax for new filter and and projection functions.
Adds the filter function shape(txt) to match against the shape of txt where it is a log message or parent rule field (e.g. shape(msg): "INFO * static log text * %qualified.leaf_rule_name% *").
Leaf rules are queryable by their rule path (e.g. message.blockID.genStamp), including wildcards and numeric comparisons.
Whole-message and parent-rule value filters depend on query decomposition, which is still under development and not yet reliable. It is gated behind -DCLP_BUILD_CLPP_DECOMPOSITION=ON (default OFF).
Adds stats.archives, stats.log_shapes, and stats.schema_tree queries for inspecting archive metadata.

Projection

Reworks Projection to track a per-node mode mask (Value/Shape/Decompose) rather than a flat column set, so a single query can request raw text, the shape, and decomposed leaves at different nesting levels.
Added two functions that can be run on log message or parent rule fields.
- shape(txt): output txt's shape as {"txt": {"shape": "..."}}
- decompose(txt): output txt's shape and all of x's leaf rule fields as {"txt": {"shape": "...", "leaf": [...]}}.

Documentation

Adds user docs for CLP+ covering concepts, compression/extraction, search, and projection output.

Build & tests

Switches log-surgeon to its Rust implementation and adds a fast_float dependency.
Adds integration-tests/tests/binary_tests/test_clpp.py covering round-trip, search, projection, and value encoding; decomposition-dependent tests are skipped unless the build flag is set.
Extends the clp-s unit tests to run against both plain and CLP+ archives.

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

Verified compression and extraction are lossless by round-tripping the (open-source Hive, Hadoop, OpenStack, and MongoDB datasets](https://docs.yscope.com/clp/main/user-docs/resources-datasets) and comparing the extracted output against clp-s and the original input (made using the log-converter for fully unstructured datasets).
Verified search over a compressed archive returns the expected result counts across leaf, shape(), and non-wildcard message and parent rule queries.
- Wildcard message and parent rule search has know issues and is disabled by default.
Verified every projection mode produces the expected JSON structure by comparing against known-good output.
Unit tests and the integration test suite pass.

…r than hijacking ClpStringT.

objects. Log-surgeon search side updated, but clp side changes not complete (may be in broken state).

different types); small refactoring.

Keep throws at the root usage to ensure nothing is missed. This reverts commit b46a68a.

bugs, but still need negation and leaf position support.

…on; Updates log-surgeon build chain.

gibber9809

Partial review focusing mostly on the parsing/write side. Should be mostly small changes, though I did spot one potential bug with find_log_shape_id/ClpMatcher that may have slightly bigger scope.

Also took a look at log-converter since I ran into a crash trying to use this branch's version.

gibber9809 · 2026-09-15T15:27:38Z

+     * - Forwards `m_archive_writer->update_log_shape_dict`'s return values on failure.
+     * - Forwards `m_archive_writer->update_parent_rule_shapes`'s return values on failure.
+     */
+    auto parse_str_field(


Minor nit - I think the name of this function might be a little bit too generic, maybe something like parse_log_surgeon_string_field or parse_log_message might be a bit better? (Or at least parse_string_field if you disagree with those).

That is fair, I'm also not happy with that name. Iirc, I used to have something closer to parse_log_message, but I was thinking that really it can parse any string not just log messages. parse_unstructured_string_field (or unstructured_text_field) is probably might be more accurate, but I'm fine with parse_log_message since realistically that is the purpose.

If you have a preference to anything lmk and I'll switch to that.

Yeah lets go with parse_log_message then.

gibber9809 · 2026-09-15T15:56:39Z

+/**
+ * An open `ParentRule` unordered object scope during `parse_str_field`. `match` identifies the
+ * scope within a `parse_str_field` call. `schema_start` is the scopes starting position in the
+ * schema and used when closing the unordered object.
+ */
+struct ParentScope {
+    log_surgeon::Match const* match;
+    size_t schema_start;
+    SchemaNode::id_t tree_node_id;
+};


I think that the schema_start name is mechanically accurate, but maybe not the clearest terminology since it mixes mechanism with functionality.

Instead of schema_start, maybe something like scope_handle, schema_handle, or rule_handle would make the functionality more clear? schema_start is aligned more with how we've been naming this in the rest of the code, but I don't think our existing naming is that good for this feature.

Ya I don't like the name either (arguably it isn't super accurate either because that index isn't the "start of a schema" it is the scope's start within the schema).

I feel like I always associate "handle" with something that wraps a resource/object. I don't think I've ever used it for an index. What do you think of scope_schema_idx, scope_start_schema_idx. I got pretty confused working with schema entries so I personally would prefer there be some sort of relationship. At one point I was trying to refactor this stuff even more, but it was taking too long.

If you feel strongly about any name, I'll go with what you prefer.

I think scope_schema_idx sounds good to me.

gibber9809 · 2026-09-15T18:30:33Z

+auto convert_string_to_double(std::string_view raw, double& converted) -> bool {
+    auto const res{fast_float::from_chars(raw.begin(), raw.end(), converted)};
+    return res.ptr == raw.end() && std::errc{} == res.ec;
+}
+


Beyond the scope of this PR, but I'm making a note to check if this fast_float library can help speed up the restore_encoded_float utility from clp-s.

I think that is worthwhile. Fwiw, the reason I left fast_float in is because I did notice a non-trivial improvement when the encoding was still buggy (and trying to convert everything).

gibber9809 · 2026-09-15T18:51:59Z

+                log_msg_node_id,
+                *m_archive_writer
+        )};
+        SchemaNode::id_t node_id{0};


Suggested change

SchemaNode::id_t node_id{0};

SchemaNode::id_t node_id{-1};

Should probably use -1 or std::optional<SchemaNode::id_t>, just because 0 is a valid node ID, even if by construction the leaf node here should never be node 0.

gibber9809 · 2026-09-15T19:09:36Z

+    [[nodiscard]] auto get_next_id() const -> uint64_t { return m_next_id; }
+


Suggested change

[[nodiscard]] auto get_next_id() const -> uint64_t { return m_next_id; }

Not used, as far as I can tell?

gibber9809 · 2026-09-17T16:15:55Z

+            // The closing placeholder delimiter is always the next '%' after the opening one as a
+            // column name cannot contain a delimiter.


Is this (making sure rule names can't contain %) enforced when parsing the spec file in log-surgeon right now?

That's correct. Although, now I realize it isn't explicitly stated in the log-surgeon docs. Iirc the only symbol is _.

gibber9809 · 2026-09-17T16:18:26Z

+            if (std::string_view::npos == close) {
+                result.emplace_back(Segment::Type::Literal, shape.substr(open));
+                return result;
+            }


Can we hit this case for a valid text shape?

No, good point. Do you think throwing is the best option since this basically means a bug occurred or there is corruption? Or what do you think is the best option?

Yeah I think throw/error is probably the right thing to do.

gibber9809 · 2026-09-17T20:49:50Z

+    compressor.write_numeric_value(m_parent_rule_shapes.size());
+    for (auto const& shape : m_parent_rule_shapes) {
+        compressor.write_numeric_value(shape.m_name.size());
+        compressor.write_string(shape.m_name);
+        compressor.write_numeric_value(shape.m_start);
+        compressor.write_numeric_value(shape.m_size);


Suggested change

compressor.write_numeric_value(m_parent_rule_shapes.size());

for (auto const& shape : m_parent_rule_shapes) {

compressor.write_numeric_value(shape.m_name.size());

compressor.write_string(shape.m_name);

compressor.write_numeric_value(shape.m_start);

compressor.write_numeric_value(shape.m_size);

compressor.write_numeric_value<uint64_t>(m_parent_rule_shapes.size());

for (auto const& shape : m_parent_rule_shapes) {

compressor.write_numeric_value<uint64_t>(shape.m_name.size());

compressor.write_string(shape.m_name);

compressor.write_numeric_value<uint64_t>(shape.m_start);

compressor.write_numeric_value<uint64_t>(shape.m_size);

Just because if we compile this for a 32-bit target (like wasm) we can accidentally change the archive format if we're writing/reading size_t directly. Could also choose uint32_t or something here, just need to be explicit with whatever we choose here + on the read side.

Does this make sense to you:

If 32-bit targets are a legitimate case, we probably want to explicitly limit everything to uint32_. Otherwise, I think we'd need to explicit write the size of size_t of the archive writer (for the reader to use).

Right now all of the field sizes are explicitly defined in the format (there's a lot of 64-bit fields that really don't need to be 64-bit, but we can't change them because of backwards compatibility). When we added support for wasm what we did was we went through all of the 64-bit fields in the archive format that need to get treated as a size_t in the code after decompression and added an extra check to ensure that the uint64 in the archive could be converted to size_t on the target platform (using the ReaderUtils::try_uint64_to_size_t utility). Basically the 32-bit wasm path handles this problem by explicitly error-ing out on some archives.

But yeah, since for legitimate use-cases these values should fit within a uint32_t anyway I'd be fine with just using uint32_t in the format for all the places I pointed out in my review and making our lives easier.

gibber9809 · 2026-09-17T20:52:44Z

+namespace clpp {
+auto LogShapeStat::compress(clp_s::ZstdCompressor& compressor) const
+        -> ystdlib::error_handling::Result<void> {
+    compressor.write_numeric_value(m_count);


Suggested change

compressor.write_numeric_value(m_count);

compressor.write_numeric_value<uint64_t>(m_count);

Same as my other comment -- should update the read and write side to use some explicit integer type besides size_t.

gibber9809 · 2026-09-17T20:54:29Z

+template <typename Element, typename Index>
+auto Array<Element, Index>::compress(clp_s::ZstdCompressor& compressor)
+        -> ystdlib::error_handling::Result<void> {
+    compressor.write_numeric_value(m_array.size());


Suggested change

compressor.write_numeric_value(m_array.size());

compressor.write_numeric_value<uint64_t>(m_array.size());

Same as other comments -- need to use explicit integer type for read/write side to avoid issue with size_t changing size between 32/64-bit platforms.

merging from main.

davidlion and others added 30 commits March 25, 2026 21:59

Revert deps changes.

c1c7835

missing query decomp and encoding

7dcaa89

add int encoding

5f08f63

help

dc8cd7a

garf

900c819

disable all targets but clp-s

26212d3

Fix schema consumption.

90083c3

always reparse schema...

c6e677a

Merge remote-tracking branch 'upstream/main' into clpp

f46495b

Deps tweaks; per-branch build dirs.

79f98c6

Add filter count stats.

7c8fffc

Switch filter count to QueryRunner.

1410812

Add new hive ints.

fd44181

Add int and str filter counts.

57ba5ba

Add dict stats.

5b2ddd8

Improve search metrics.

142c397

Update log surgeon; Add decomposed query cache.

cd8c7e1

Remove log.

5a3238b

vibrating

f394f90

Check metadata properly.

6b348a8

Store schema in archive; read schema with log surgeon; some refactoring.

f05db0e

Remove unnecessary include.

9527aa7

Fix logtype schema map.

5317c29

Refactor clpp query creation.

518af89

Compression changes for log-mechanic.

b6bdfff

Finish compression port.

b3d1924

Create new literal type for types requiring query decomposition rathe…

edccbd3

…r than hijacking ClpStringT.

Begin renaming LS terms.

b3ade7d

More renaming.

a58c84d

More renaming.

472bb75

davidlion added 8 commits August 21, 2026 12:56

Refactor open parent rule scope updating.

d71df7b

Integration tests with elasticsearch dataset.

6b92b8e

Forgot to remove when merging upstream profiler.

a2352d9

Change schema API and store log shape ID inside LogMessage unordered

b65e11a

objects. Log-surgeon search side updated, but clp side changes not complete (may be in broken state).

Schema + search refactor.

816a027

Gate decomposition behind a compilation flag.

59c9712

Merge remote-tracking branch 'upstream/main' into clpp

037ec85

Refactor reading to fix SFAs; add some unit tests.

06ba562

davidlion changed the title ~~Clpp~~ feat(clp-s)!: Adds CLP+ as an experimental feature. Sep 11, 2026

davidlion added 13 commits September 11, 2026 02:56

Fix nexist behaviour; fix when names expand to multiple nodes (due to

1b4615f

different types); small refactoring.

Tweak decomposition error handling.

b46a68a

Fixed field-name / left-side wildcard behaviour; minor refactoring.

6c72138

Revert "Tweak decomposition error handling."

772db1b

Keep throws at the root usage to ensure nothing is missed. This reverts commit b46a68a.

Remove old clp/clo/glt usage of log-surgeon.

7241c97

Remove old clp/clo/glt usage of log-surgeon.

236e4b1

Remove some remaining snippets outside of log-converter.

33d0d68

Missed deleting file.

d13f79a

Update search with the new version of log-surgeon. Fixes correctness

ae6d37b

bugs, but still need negation and leaf position support.

Merge remote-tracking branch 'upstream/main' into drop-clp-ls

a6a6ceb

Dropped now unused dep tracking; added back doc version.

f5015c9

feat(log-converter): Update log converter to use the latest log-surge…

40212ec

…on; Updates log-surgeon build chain.

Add negation support.

ff49860

davidlion marked this pull request as ready for review September 17, 2026 20:12

davidlion requested a review from a team as a code owner September 17, 2026 20:12

Merge branch 'log-converter-ls-update' into clpp

a3afc9a

gibber9809 requested changes Sep 17, 2026

View reviewed changes

davidlion added 4 commits September 18, 2026 19:03

Remove profiling redefinition in cmakelists that wasn't removed when

2e960ca

merging from main.

Address some review comments.

1576180

Correctly handle the positions of leaf queries from decomposition.

54ead04

Docs pass.

901068c

		[[nodiscard]] auto get_next_id() const -> uint64_t { return m_next_id; }

		// The closing placeholder delimiter is always the next '%' after the opening one as a
		// column name cannot contain a delimiter.

	compressor.write_numeric_value(m_count);
	compressor.write_numeric_value<uint64_t>(m_count);

	compressor.write_numeric_value(m_array.size());
	compressor.write_numeric_value<uint64_t>(m_array.size());

Conversation

davidlion commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

CLP+ storage & compression

Search

Projection

Documentation

Build & tests

Checklist

Validation performed

Uh oh!

gibber9809 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

davidlion commented May 12, 2026 •

edited

Loading