As a reminder, to pass submission check the result dataset must include 50 unique classes of y_given with 30 samples of x_output each. Remember, as described in the Starter Notebook, your submission must contain y_given and not y_hat.
Here is some code to to ensure that your submission dataset format is correct:
# Path to your result dataset file
submission = "result_dataset.pt"
# Loading the dataset using class SignalDataset which provided inside the Starter Notebook's files (CVAE_functions.py).
dataset_scoring = SignalDataset(torch.load(submission))
scoring_loader = DataLoader(dataset_scoring, batch_size=len(dataset_scoring), shuffle=False)
x, y = next(iter(scoring_loader))
unique_y, counts = torch.unique(y, dim=0, return_counts=True)
# Checking if the number of unique classes in y is exactly 50
print(f'You result includes GIVEN_Y and has exactly 50 unique classes: {len(unique_y) == 50}')
# Checking if each class in y has exactly 30 samples
print(f'You result includes exactly 30 samples for each class in GIVEN_Y: {torch.all(counts == 30).item()}')
When you run the code above you must get TRUE in both print statements to pass the submission check.
Hi @daisuke0530
Yes, y_given affects the score.
As described at the end of the Starter Notebook, one component of the score is the reconstructed error. This evaluates how closely y_hat approximates y_given.
Therefore, when you provide us with a file containing the generated X0/X1, we can calculate y_hat, right? So to compare y_hat and y_given, we need to know the specific y_given for which these values were generated.
I hope that was helpful,
Best regards. Onward Team.
Here’s another recommendation: If you are facing issues with a submission not passing due to an incorrect format, please check whether your tensors were created on a GPU (CUDA) or a CPU.
To successfully pass the submission process, they should be deserialised onto a CPU device.
This can be achieved as follows (just as an example):
x_output.cpu(), y_output.cpu()
before you compile them into the final result_dataset.pt file.
Hi Guys
I took to heart the idea of submitting the given y0 and y1. Progress! It passed the formatting and I got a score of 0. I then made one little change and received a new error message:
Scoring algorithm failed
I regard this as hopeful since they were actually trying to score it.
Anyway, for me the key was to use 30 entries each of y0 and y1. Note that if you use 50, the submit file is too big. This means throwing away the last 20 y0 and y1 entries. I have some additional thoughts on the submission process for later.
cheers, Eric
i have the same error however it passes the check my tensors shape are torch.Size([1500, 50, 2]) torch.Size([1500, 50, 2]) and both of float 32 dtype @discourse-admin
I am getting this error - “Tensor dataset has incorrect structure, each element must include x and y pairs.” but the above code you have given me gives True in both print statements. Let me know where am I going wrong.
Hi @lori.eric9797@mabdelrazik34 and @shravankumar224 it sounds like you are all making progress on your submissions. You should be able to upload the submission file without hitting the upload limit.
The print statement below prints out the shapes for the different elements in the tensor. The comments on each line are the correct sizes from the sample submission:
print(f"Tensor shapes:\n x shape: {x.shape}\n" # torch.Size([1500, 1, 50, 2])
f" y shape: {y.shape}\n" # torch.Size([1500, 1, 50, 2])
f" unique_y shape: {unique_y.shape}\n" # torch.Size([1500, 1, 50, 2])
f" Number of unique classes: {len(unique_y)}\n" # 50
f" Counts shape: {counts.shape}\n" # torch.Size([50])
f" Number of samples per class {counts.unique()[0]}") # 30
Also, double check that that your tensors are deserialized onto a CPU device using the in the thread above.
I am still getting a score of zero - I checked the above outputs or print statements - all of them are aligned as per the requirement except the
f" unique_y shape: {unique_y.shape}\n" # torch.Size([1500, 1, 50, 2])
for me it is showing ----unique_y shape: torch.Size([50, 1, 50, 2])
Do you think this is the problem for my - 0 score. Can you please clarify and also check my submission if possible?
Hi @shravankumar224 your submission was successful. With a score of zero that means your submission was accepted and is in the correct format. Now it is time to work on tuning your model to improve your score.
I am having trouble loading the submission. I think I have followed all your advice like:
Check that data format is float32
Check that the tensors are in cpu (not gpu)
Run the following 2 print statements and see if you get True:
print(f'You result includes GIVEN_Y and has exactly 50 unique classes: {len(unique_y) == 50}')
# Checking if each class in y has exactly 30 samples
print(f'You result includes exactly 30 samples for each class in GIVEN_Y: {torch.all(counts == 30).item()}')
print(f"Tensor shapes:\n x shape: {x.shape}\n" # torch.Size([1500, 1, 50, 2])
f" y shape: {y.shape}\n" # torch.Size([1500, 1, 50, 2])
f" unique_y shape: {unique_y.shape}\n" # torch.Size([1500, 1, 50, 2])
f" Number of unique classes: {len(unique_y)}\n" # 50
f" Counts shape: {counts.shape}\n" # torch.Size([50])
f" Number of samples per class {counts.unique()[0]}") # 30
Here I think you made a typo. For unique_y.shape I get (50, 1 , 50, 2) but you say it should be [1500,1, 50,2].
Apart from this everything matches with the expectation but I still am not able to submit. I get algorithm error without any specification for the cause of the error.
Hi @discourse-admin,
Yes, I have checked that my tensors are in cpu just like you ask us do. I have followed your advice on all 4 criteria that you’ve mentioned. Can I send you my submission.pt file so that you can let me know from your end what seems to be a problem? If yes, can you please tell me your email id. Many thanks!
I sent an email to you about this. You also sent me a message with an email but it got deleted (i think) so i am unable to access it. So, I sent you an email on challenges@thinkonward.com with my submission tensor attached
I have checked your shared dataset, it is correct and as I see your last submission was successful, but has zero score. You are on the right way. Keep going.