Hi Team Onward,
I have a couple of questions about the scoring function.
Whatever I submit, even the sample submission file or a very promising solution with respect to the training dataset, I get a zero score, so I’m a little confused. Could you provide an official Python implementation of the scoring function?
Does the scoring function accept as valid the reverse of a valid solution? The challenge is about sorting image traces, but how can one choose between a sorting solution and its mirrored one? E.g. if [3,1,4,0,2] is the correct order of vectors for a given image made of 5 traces, I think that [2,0,4,1,3] should be accepted as correct too. Is there any preference condition between the two vectors?
Thank you for your insightful question and observation.
The sample_submission file is intended to produce a zero score for the Parallel Perspectives challenge. The scoring algorithm’s implementation for this challenge is kept confidential.
Mirrored submissions will be accepted as the data has no context to describe orientation. The Live Scoring algorithm will be edited in the next few days to give equal scoring regardless of orientation.
Please let us know if you continue to have questions about this unique puzzle challenge.
In relation to the first question posed by @francescolucchi , I encountered a similar issue—an assigned score of 0 to a solution that appeared promising. To investigate whether the problem was specific to my solution, I uploaded a random one and received the same 0 score. Considering the equation:
Hi @gabriel.soares.gamma here is a bit more information on the scoring algorithm and why your solutions received a score of 0.
Scores range from 0 to 1 where a score of 0 means less than 75% of the pieces are correct, and a score of 1 means that all pieces are correct. The first weight is used for assessing overall similarity, and the second for the sum of dissimilarities for mismatched pieces. With the current weights, the score will be 0 for solutions that have more than 25% of mismatched elements per image.
Since there are perfect solutions on the leaderboard already, there is a chance they will score perfectly on the private data as well. How would prizes be distributed in that scenario?
As more participants have reached the top-ranking spots for Parallel Perspectives, the Challenges Team has diversified and expanded the hold-out data. With this update, we feel that the finalists can be sorted.
