Fix XXE in XML batch parsing - #452
Conversation
gibson9583
left a comment
There was a problem hiding this comment.
Requesting changes before merge.
The new batch endpoint can report success without accepting a batch, and its response collector retains all processed message payloads. Please address the two inline comments.
The Oracle integration check is also failing in the new CSV batch test: Expected source transformed content but the server stored none. Please investigate and get a passing Oracle run before merging. I have not established whether the missing content is caused by this PR.
f1d7369 to
0fb4c05
Compare
|
@gibson9583, the oracle issue was opened with issue #454 and fixed in #455 |
gibson9583
left a comment
There was a problem hiding this comment.
Items raised are non-blocking. Approved
tonygermano
left a comment
There was a problem hiding this comment.
My requests are for documentation improvements and not to challenge the code.
Can you expand the commit messages? Some of these changes are not small, and it would be good to have a summary of the changes and justification for why they are needed, especially when they touch multiple files or appear at a glance to change workflows.
Normally batch processing is controlled by the channel definition. What is the expected behavior when calling the new API method batchMessagesWithObj on a channel which doesn't have batch processing enabled? Is the only difference between calling this method and the existing method the return value? I think the method description implies that might be the case, but it should also describe what happens when called on a channel with batch processing disabled.
You should have write access to the branch. Feel free to reword. The commits are thematically made, already, so there should not be much explanation I can think of - and no issue is open to link to (nor do I think these are nuanced enough to warrant one).
Not sure - but the change is to return the message ID's from process batch. It does not change the processing. Whatever happened previously would still happen, just now you have an endpoint which gives all of the message ID's instead of just 1. |
|
This rejects batches whose DOCTYPE declares only internal entities and points nowhere. I added a case to jonbartels flagged the same behaviour change on #441 on 15 Sep and offered a changelog entry, and abhinavagarwal07 asked there about a release note. Neither got a reply, and nothing here mentions it. #453 goes the other way, permitting the DOCTYPE and setting only |
You and Tony are right. The rest of the app blocks DTDs entirely. I had tried to leave a small hole in the XSLT on the basis that someone might be using XHTML source messages, but it did not work, and I see no reasonable way to make it work. Updated that PR to block DTD entirely, making it consistent with this PR (and the majority of the engine, which already blocks DTDs). |
tonygermano
left a comment
There was a problem hiding this comment.
See inline suggestion and question. I was going to have clanker write the missing commit messages bodies, but I did not know if they would be retained if other changes go through.
0fb4c05 to
8266577
Compare
8266577 to
822f8ee
Compare
pacmano1
left a comment
There was a problem hiding this comment.
The DOCTYPE question is settled, but the behavior change still needs a release note. A batch whose DOCTYPE declares only internal entities splits on 4.5 and gets a 500 after this PR. Can the PR body call that out so it lands in the 4.6.x notes? Same ask jonbartels made on #441.
822f8ee to
551aea6
Compare
|
I have a non blocking followup PR at mgaffigan#8 This PR would make some corrections to how a failed message is handled. I think its valuable but adding 500 lines to PR 452 is would further slow the review process. I am approving |
processMessage returns a single message id, taken from the response
handler's selected result. For a batch that is only the first or last
message, leaving a caller no way to learn the ids of the others. The
smoke test harness needs them to assert per-message content.
POST /channels/{channelId}/batchMessagesWithObj returns every id. The
response handler is already threaded through dispatchBatchMessage, so
CollectingResponseHandler only has to record each id as it is set. It
keeps the ids and not the DispatchResults, so a long batch does not
retain the processed messages, maps and content of the messages that
have already finished.
Processing is unchanged. Whether a batch is split is still governed by
the channel's Process Batch setting; on a channel with batch processing
disabled the dispatch takes the single-message path and the endpoint
returns a one-element list.
EngineController.dispatchRawMessage gains a five-argument overload. The
existing four-argument form delegates to it with a SimpleResponseHandler,
so its callers are unaffected, but the interface gains an abstract method
and out-of-tree implementations will need updating.
Signed-off-by: Mitch Gaffigan <mitch.gaffigan@comcast.net>
A batch fixture produces several messages, and the harness could only submit a payload and assert against one of them. Assertion files for a fixture that expects N messages now live in numbered subdirectories, 01 through NN; a fixture that expects one message keeps its flat layout and reads exactly as before. The generator rejects numbering that does not start at 01 or that leaves gaps, and rejects a fixture that mixes loose assertion files with numbered ones. An empty source_rejected file declares that the server must refuse the submission. It cannot be combined with assertion files, since a refused submission produces no message to assert against. source_raw, destNN_raw and destNN_encoded are now assertable. OieServer.submitMessage calls the new batchMessagesWithObj endpoint so the harness learns every id the payload produced, and runMessage fails if the count does not match what the fixture expects. Failure output renders every message rather than one, and now includes the source map. ci/tests/120-delimited-batch covers the new layout with a CSV batch channel, independent of any XML change. Note that routing every submission through batchMessagesWithObj leaves processMessage without end-to-end coverage. Signed-off-by: Mitch Gaffigan <mitch.gaffigan@comcast.net>
Signed-off-by: Mitch Gaffigan <mitch.gaffigan@comcast.net>
CVE-2026-82578 found an XXE vulnerability in XML batch parsing. This closes that vulnerability and adds regression tests. setNamespaceAware(true) is not a behavior change. XPath.evaluate(String, InputSource, QName) built its DOM with a namespace aware DocumentBuilder, so the batch splitter has always been namespace aware. Without it, split messages lose the xmlns declarations they inherit from the batch element. Signed-off-by: Mitch Gaffigan <mitch.gaffigan@comcast.net>
551aea6 to
77e86a5
Compare
|
I also addressed the commit updates that @tonygermano cited. @mgaffigan hit me in Discord tomorrow and lets get this merged |
Commit messages were updated.
CVE-2026-82578 found an XXE vulnerability in XML batch parsing. This closes that vulnerability and adds regression tests.
Review notes:
Note: