Evaluating ClusteredPredictor

[kowshikthopalli]

Hi
Thank you for this amazing toolbox. I have a question regarding how to use clustered predictor for evaluation.

  predictor =  ClusteredPredictor(score_function=THR(score_type='identity'),model=model, split='doubledip')
  predictor.calibrate(calib_loader,alpha=0.1)
  class_specific_thresholds = predictor.q_hat
  result_dict = predictor.evaluate(test_loader)

However predictor.evaluate is not implemented in ClusteredPredictor but inherits from the SplitPredictor which does not take clustering into account. Can you tell me how to evaluate please. ?

[Jianguo99]

Hi @kowshikthopalli,

Thank you for your question. You can directly use ClusteredPredictor.evaluate. Although this method is inherited from SplitPredictor, it correctly uses the cluster-specific thresholds during evaluation.

When you call predictor.calibrate(), it sets predictor.q_hat as a vector containing individual thresholds for each cluster. Then during evaluation, the implementation automatically applies the appropriate threshold for each sample based on its cluster assignment.

We appreciate your interest in our toolbox and hope this helps with your implementation. Please feel free to reach out if you have any further questions.

Best regards,
Authors