About the metric

Spot the Trend
1

There is something really weird about the metric:
suppose ground truth is : (55613, 90222), (101241, 101544), (256613, 296764)

A. if we predict : (55613, 90222**+1000**), (101241, 101544), (256613, 296764), an error of 1000 min of the first stop, we get a score : 0.9978. This makes the IOU completely useless

B. if we predict : (55613**+1**, 90222), (101241, 101544), (256613, 296764), an error of +1 on the first start, we get a score : 0.9798. So an error of just 1 min is much worse than an error of 1000 min on the full interval

C. * if we predict : (55613**+3**, 90222), (101241, 101544), (256613, 296764), an error of +3 on the first start, we get a score : 0.94986

  • if we predict : (101241, 101544), (256613, 296764), completely missing the first interval, we get a score of : 0.9545
    So it’s better to miss a full interval rather than making a +3 error on the first start. This doesn’t make sense to me.

Can you please confirm this is the intended behaviour of the metric.

2

Hi @mchahhou :wave:

Your thoughts are correct on the metric, this is the intended behavior. The intersection over union and the end time are less important than the start error and a small mistake in the start prediction strongly affects the result.

Onward Team