Hi,
Is there anyone who has tested the provided starter_notebook.ipynb? I have tried to run it, And it seems has a few of bugs in dataset.py and cannot work at all. I tried to fix two bugs, but still has a fatal error in the training step.
May the host or anyone clarify it?
thanks,
Ken
On my side, I keep running into a ‘ValueError: setting an array element with a sequence’ error on the dataset creation step. I tried downgrading the numpy version, making changes to the dataset.py file, etc., but no success yet.
Hi @stonekingxk and @ogneebho.mukhopadhya
Can you please double check that you have unpacked reflection-connection-train-data.zip and reflection-connection-test-data.zip into the ROOT/data directory?
In case the problem still occurs, please share the following details to help us solve the problem:
- Specifications of your environment (OS version, versions of Python, libraries specified in requirements.txt and CUDA in case you’re using GPU acceleration)
- Error message
Onward Team
The project directory should have the following structure:
.
├── data
│ ├── image_corpus
│ ├── query
│ ├── test
│ │ ├── Boring
│ │ ├── Bright_Chaotic
│ │ ├── Bright_Planar
│ │ ├── Channel
│ │ ├── Converging_Amplitudes
│ │ ├── Fault
│ │ ├── Salt
│ │ └── Transparent_Planar
│ └── train
│ ├── Boring
│ ├── Bright_Chaotic
│ ├── Bright_Planar
│ ├── Channel
│ ├── Converging_Amplitudes
│ ├── Fault
│ ├── Salt
│ └── Transparent_Planar
└── src
└── siamnet
├── snn
│ └── dataset
└── utils
Please verify that your local structure is the same.
Onward Team
Thanks,
1, linux; python: 3.9.13, pytorch: 1.13, only cpu; source files are in src/, data files are in data/ as the instruction.
2. errors:
(1) ImportError: cannot import name ‘_flip_byte_order’ from ‘torchvision.datasets.utils’ (/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/datasets/utils.
I fixed it by deleting _flip_byte_order’ in dataset.py
(2) → 118 data.append(np.array(images, dtype=np.float32))
119 #data.append(np.array(im, dtype=np.float32))
120 targets.append(np.full(len(images), target))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (210,) + inhomogeneous part.
I fixed it by modifying images to im. But I am not sure it is correct way.
(3) then plt figures can not work for contrastive_dataset, triplet_dataset, not fixed
(4) Keep running, but still can not train.
IndexError Traceback (most recent call last)
Cell In [17], line 1
----> 1 siamese_trainer.train(model, train_loader, scheduler, n_epochs=n_epochs)
File ~/Desktop/Daddy_code/Reflection_Connection/src/siamnet/snn/train.py:190, in Trainer.train(self, model, train_loader, scheduler, val_loader, start_epoch, n_epochs, dry_run)
187 scheduler.step
188 # training stage
→ 190 train_loss = self.training_epoch(model,
191 train_loader,
192 epoch=epoch,
193 dry_run = dry_run)
194 if dry_run:
195 print(‘Dry run test completed’)
File ~/Desktop/Daddy_code/Reflection_Connection/src/siamnet/snn/train.py:72, in train_contrastive(model, train_loader, device, optimizer, criterion, epoch, dry_run, log_interval)
69 if criterion is None:
70 criterion = ContrastiveLoss()
—> 72 for batch_idx, ((images_1, images_2), targets) in enumerate(train_loader):
73 images_1, images_2, targets = images_1.to(device), images_2.to(device), targets.to(device)
74 optimizer.zero_grad()
File ~/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py:628, in _BaseDataLoaderIter.next(self)
625 if self._sampler_iter is None:
626 # TODO(Bug in dataloader iterator found by mypy · Issue #76750 · pytorch/pytorch · GitHub)
627 self._reset() # type: ignore[call-arg]
→ 628 data = self._next_data()
629 self._num_yielded += 1
630 if self._dataset_kind == _DatasetKind.Iterable and
631 self._IterableDataset_len_called is not None and
632 self._num_yielded > self._IterableDataset_len_called:
File ~/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py:671, in _SingleProcessDataLoaderIter._next_data(self)
669 def _next_data(self):
670 index = self._next_index() # may raise StopIteration
→ 671 data = self._dataset_fetcher.fetch(index) # may raise StopIteration
672 if self._pin_memory:
673 data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)
File ~/.local/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py:58, in _MapDatasetFetcher.fetch(self, possibly_batched_index)
56 data = self.dataset.getitems(possibly_batched_index)
57 else:
—> 58 data = [self.dataset[idx] for idx in possibly_batched_index]
59 else:
60 data = self.dataset[possibly_batched_index]
File ~/.local/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py:58, in (.0)
56 data = self.dataset.getitems(possibly_batched_index)
57 else:
—> 58 data = [self.dataset[idx] for idx in possibly_batched_index]
59 else:
60 data = self.dataset[possibly_batched_index]
File ~/Desktop/Daddy_code/Reflection_Connection/src/siamnet/snn/dataset/wrappers.py:63, in ContrastiveDataset.getitem(self, index)
61 if self.train:
62 target = np.random.randint(0, 2)
—> 63 img1, label1 = self.train_data[index], self.train_labels[index].item()
64 if target == 1:
65 siamese_index = index
IndexError: index 858 is out of bounds for dimension 0 with size 561
It is too difficult to debug your provided starter code.
Hi @stonekingxk
Issue (2) is caused by an inhomogeneous shape of images in dataset. Unfortunately the fix that you implemented causes further problems as it modifies the expected shape of data tensor.
We noticed that line numbering is different than expected in the error traceback you provided. Can you please verify that the following line of code is included after reading the image from disk:
113 im = im.convert(self.colormode).resize((118, 143))
It is essential to resize each image to the common shape before appending to the list of images for each class.
Please note that starter notebook was prepared to work with pytorch 2.x and a compatible version of torchvision. Running it with pytorch 1.x might result in unexpected behavior.
Onward Team
Yes, I still keep this line: im = im.convert(self.colormode).resize((118, 143)).
Even in Kaggle platform with only cpu: Pytorch 2.0.0+cpu. It has the same error:
ValueError Traceback (most recent call last)
Cell In[6], line 1
----> 1 dataset = SeismicDataset(ROOT, build=True, greyscale=False, train=True)
SeismicDataset.init(self, root, train, transform, target_transform, build, greyscale, compatibility_mode)
81 self.compatibility_mode = compatibility_mode
83 if build:
—> 84 self.data, self.targets = self._load_from_fs(root)
85 return
87 self.data, self.targets = self._load_data()
File /kaggle/working/src/siamnet/snn/dataset/dataset.py:117, in SeismicDataset._load_from_fs(self, root)
115 if len(images) == 0:
116 continue
→ 117 data.append(np.array(images, dtype=np.float32))
118 targets.append(np.full(len(images), target))
120 return torch.from_numpy(np.concatenate(data, axis=0)), torch.from_numpy(np.concatenate(targets, axis=0))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (210,) + inhomogeneous part.
Hi @stonekingxk
We have tried replicating the error you are running into on fresh conda environments on Windows, macOS, and Linux. We have not been able to replicate the error on any of these systems. Can you download a fresh copy of the starter notebook and associated python code, and overwrite your existing versions of the starter notebook and Python assets in the /src directory? Once you have replaced them, please run the notebook from start to finish with no modifications and validate that it is working with the fresh copies.
Onward Team
I have tried a few times in kaggle and my linux workstation before, with the same errors. I believed I followed your instruction and have not made mistakes. And there is at least one other guy above mentioned the same error. May you try it in kaggle or google colab? Thanks.
Rerun in Kaggle with the same detailed errors below:
/kaggle/working/src/siamnet/snn/dataset/dataset.py:117: FutureWarning: The input object of type ‘Image’ is an array-like implementing one of the corresponding protocols (__array__, __array_interface__ or __array_struct__); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using np.array(obj). To retain the old behaviour, you have to either modify the type ‘Image’, or assign to an empty array created with np.empty(correct_shape, dtype=object).
data.append(np.array(images, dtype=np.float32))
ValueError Traceback (most recent call last)
Cell In[6], line 1
----> 1 dataset = SeismicDataset(ROOT, build=True, greyscale=False, train=True)
File /kaggle/working/src/siamnet/snn/dataset/dataset.py:84, in SeismicDataset.init(self, root, train, transform, target_transform, build, greyscale, compatibility_mode)
81 self.compatibility_mode = compatibility_mode
83 if build:
—> 84 self.data, self.targets = self._load_from_fs(root)
85 return
87 self.data, self.targets = self._load_data()
File /kaggle/working/src/siamnet/snn/dataset/dataset.py:117, in SeismicDataset._load_from_fs(self, root)
115 if len(images) == 0:
116 continue
→ 117 data.append(np.array(images, dtype=np.float32))
118 targets.append(np.full(len(images), target))
120 return torch.from_numpy(np.concatenate(data, axis=0)), torch.from_numpy(np.concatenate(targets, axis=0))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (210,) + inhomogeneous part.
Good news, I can run the code in another machine. But there are still incompatible issues in kaggle platform or some old machines. Anyway, Thanks!
Great to hear you were able to get it up and running @stonekingxk. We are looking to see what the issues are on Kaggle and older machines. Thank you for your patience and trying out a couple different machines!
Happy solving!
Onward Team
Thanks a lot for your help!
I have the same issue the files for classes in snn module is not present , all the classes like Datasets , Siemese network doesn’t exist .
Hi @ahmedmohamed.h1002 and welcome to Challenges!
Can you tell us a bit more about the issue and include any error messages? Also remember to join the challenge and download and unzip the dataset into the directory in the structure outlined above.
Onward Team
After I extracted the train file , the data is the train and test folder only without the starter classes
Hi @ahmedmohamed.h1002 make sure you add the Python files from the Starter Notebook download to your directory. They are zipped with the Starter Notebook download and not with the train/test data download. Check the data tab to download the .zip file with the classes in the Python files.
Onward Team
Oh I saw it , sorry for that I downloaded the files into colab and didn’t notice them
Thanks for answering
1 Like
Glad you were able to get it resolved and best of luck on the challenge!
Onward Team
I also have this “ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (210,) + inhomogeneous part.” I have the latest torchvision and pytorch.
After I upgrade numpy to latest version, this error is gone.
For anyone who encounters this error, try to upgrade your numpy.
1 Like