Unable to calculate VTSS for user-provided videos

[cellzero]

Thanks for the great works! And I wonder how to calculate VTSS for user-provided videos?

The official VTSS inference demo needs the ground truth labels to resize predicted labels to the same scale as ground truth labels (pr_labels = rescale(pr_labels, gt_labels), line 83, training_suitability_assessment/inference.py). However, user-provided videos do not have the ground truth values. I have noticed the No Label Testing part in FusionDataset class (datasets.py), but the method simply set the gt_label to -1, after rescale() all the predicted values will be set to -1.

Thank you for your time and assistance!

[qiuqiuh]

Thank you for your question, in the code, if gt_label is set to -1, the predicted result will not be scaled, i.e. it will be a score between 0 and 1, and you can scale the value in any way you want.

[cellzero]

Thank you for your reply.

I estimated the distribution of the gt_labels using Figure 6 from the paper, and it was able to obtain similar results. Here are the estimations I used:

official_score_distribution_dict = {
    1.1: 180, 1.3: 360, 1.5: 375, 1.7: 160, 1.9:  99,
    2.1:  80, 2.3:  79, 2.5:  98, 2.7: 155, 2.9: 245,
    3.1: 330, 3.3: 395, 3.5: 390, 3.7: 365, 3.9: 475,
    4.1: 687, 4.3: 800, 4.5: 500, 4.7: 130, 4.9:  10,
}

I also tried outputting the results without re-scaling, which ranged approximately from -0.05 to 0.068. I'm curious if this range aligns with your results.