Can we make submission for both Paths?

Reflection Connection
1

Are we eligible to make submissions for both the one-shot and free form paths that exist in this competition?
If we can only make 1 submission, what if our one shot classifier performs better than all other algorithms for either path? Would this mean we would also win the free form path of the competition as that path doesn’t have any restrictions on the types of models that can be used?

2

Hi @kylepeters08721

For this challenge we expect only one final solution per team. Any model that is designed specifically to solve the one-shot problem will also be eligible for the freeform classification prize. However, in case your algorithm performs better than other participants in both paths you will be awarded only with the higher (one-shot) prize.

Good luck!

Onward Team

3

Hi Onward team,

Can you elaborate more, what you call “freeform classifier”? If I somehow use labels of train data – does this immediately move me into the category of freeform classifiers?

4

Hi @dmitry.ulyanov.msu

By “freeform classifier” we mean any classification model, regardless of its one-shot capabilities. The distinction between two paths applies only to evaluation. You can use the labels from training data in any way you like - your model will be then assigned scored for both freeform and one-shot classification.

Onward Team

5

Thanks for your response, though it’s still not completely clear to me how freeform classification score will be assigned. Will there be a separate dataset without extra classes? How do you determine class prediction based on top3 retrieval results that we submit in json?

6

Hi @dmitry.ulyanov.msu

There will be a separate holdout dataset, but it will have a similar structure to the current test dataset in terms of classes included.

The only difference between the two paths is which classes of reference images are considered - for the freeform classification we calculate the score only for query images that belong to one of the training classes. It’s equivalent to setting class_weight of one-shot classes to 0. Images from new classes are still present in image corpus and might be incorrectly retrieved.

Onward Team

7

Hi @team

I have several questions about the final evaluation.

  1. The overview page states:

“For the final evaluation, the submissions that reach a freeform classification score threshold of >0.5 on the Predictive Leaderboard will be invited to send Onward their fully reproducible code for evaluation against holdout data.”

Does this mean the public test set is for freeform classification and only contains samples from the 8 classes?

  1. What happens if no one reaches the 0.5 criteria?

  2. The overview also mentions:

“For participants interested in the one-shot path, we’ll allow all participants to submit their code through the challenge page once the challenge is completed on March 22, 2024. Further instructions will be provided for one-shot submission at the close of the challenge.”

Is it allowed to submit code that was not submitted to the prediction leaderboard? In other words, can we still develop and change the code after the deadline of March 22, 2024? If that is the case, what is the real deadline for the one-shot path?

8

Hi @daisuke0530

Here are the answers to your quesionts:

  1. The public test set is not limited to freeform classification (it contains also samples for new, unseen classes) - but the public scoring in Predictive Leaderboard is a freeform classification score - it’s calculated only based on 8 training classes.
  2. If no one reaches the 0.5 criteria we will be accepting the top 12 scoring teams to submit for both freeform and one-shot paths. It will be up to each team to indicate the paths they would like to be considered for within their documentation.
  3. The Challenge closes on March 22 2024, and the deadline for final submissions is March 29 2024

Onward Team

2 Likes