Expected behaviour
When bagit-python uses multiprocessing for manifest generation or validation, it should always clean up the pool before returning control to the caller. If a worker fails, the pool should be terminated and joined so child processes do not remain alive, orphaned, or defunct in long-running services.
Current behaviour
When processes > 1, failures inside Pool.map() can skip the normal close() and join() path. The caller receives the exception, but the BagIt worker processes may remain behind. In services like Archivematica, repeated failures can accumulate leftover child processes.
Steps to reproduce
Start an ingest that reaches the "Verify AIP" step and force that step to fail during BagIt validation, for example by corrupting or removing a file from the AIP before verification runs. After the failure is reported, inspect the operating system process list with a tool such as ps. In affected versions, defunct Python child processes from the BagIt multiprocessing pool may remain after the failed validation.
Your environment (version of Archivematica, operating system, other relevant details)
This affects Archivematica 1.16 or newer.
For Artefactual use:
Before you close this issue, you must check off the following:
Expected behaviour
When
bagit-pythonuses multiprocessing for manifest generation or validation, it should always clean up the pool before returning control to the caller. If a worker fails, the pool should be terminated and joined so child processes do not remain alive, orphaned, or defunct in long-running services.Current behaviour
When
processes > 1, failures insidePool.map()can skip the normalclose()andjoin()path. The caller receives the exception, but the BagIt worker processes may remain behind. In services like Archivematica, repeated failures can accumulate leftover child processes.Steps to reproduce
Start an ingest that reaches the "Verify AIP" step and force that step to fail during BagIt validation, for example by corrupting or removing a file from the AIP before verification runs. After the failure is reported, inspect the operating system process list with a tool such as
ps. In affected versions, defunct Python child processes from the BagIt multiprocessing pool may remain after the failed validation.Your environment (version of Archivematica, operating system, other relevant details)
This affects Archivematica 1.16 or newer.
For Artefactual use:
Before you close this issue, you must check off the following: