Small one, raised before I write about it in case I have misread it. Line numbers at commit a7dd338386a4fae9.
metrics/cuad/compute_score.py:117-121: np.trapz returns NaN when the integral is undefined — for example a degenerate or empty recall array, where there is nothing to integrate. That NaN is collapsed to 0.
In an area-under-precision-recall metric, 0 is not "undefined". It is the worst attainable score. So a case where the AUPR could not be computed is recorded as the model achieving the worst possible result, and the two are indistinguishable in any aggregate.
This is CUAD, a legal-contract benchmark, so the aggregate is the number people quote.
Suggestion
As an option: propagate the NaN, or return None and exclude the sample, so an uncomputable area is visible as uncomputable rather than as a floor value.
Disclosure
This appears as one instance in a short methods paper about evaluation instruments that state an invariant and do not apply it — most instances in it are from my own code. I would rather you saw it first, and if it is intended I will record that.
Small one, raised before I write about it in case I have misread it. Line numbers at commit
a7dd338386a4fae9.metrics/cuad/compute_score.py:117-121:np.trapzreturnsNaNwhen the integral is undefined — for example a degenerate or empty recall array, where there is nothing to integrate. ThatNaNis collapsed to0.In an area-under-precision-recall metric,
0is not "undefined". It is the worst attainable score. So a case where the AUPR could not be computed is recorded as the model achieving the worst possible result, and the two are indistinguishable in any aggregate.This is CUAD, a legal-contract benchmark, so the aggregate is the number people quote.
Suggestion
As an option: propagate the
NaN, or returnNoneand exclude the sample, so an uncomputable area is visible as uncomputable rather than as a floor value.Disclosure
This appears as one instance in a short methods paper about evaluation instruments that state an invariant and do not apply it — most instances in it are from my own code. I would rather you saw it first, and if it is intended I will record that.