Zero Input Size

[Michael-wzl]

When running python train.py --config configs/example_smirk.txt, an error occurred:
Epoch 1, Iter 24: 0%| | 0/45 [00:41<?, ?it/s, loss=4.09]
Traceback (most recent call last):
File "/home/zlwang/SPARK/TrackerAdaptation/train.py", line 344, in
adapt_tracker(args, wrapper, dataset_train, dataset_val, dataset_test)
File "/home/zlwang/SPARK/TrackerAdaptation/train.py", line 108, in adapt_tracker
for views in dataloader_train:
File "/home/zlwang/SPARK/TrackerAdaptation/../MultiFLARE/utils/dataset.py", line 123, in iter
for batch in super().iter():
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 628, in next
data = self._next_data()
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torch/utils/data/dataset.py", line 295, in getitem
return self.dataset[self.indices[idx]]
File "/home/zlwang/SPARK/TrackerAdaptation/../MultiFLARE/utils/dataset.py", line 146, in getitem
self.cache[idx] = self.dataset[idx]
File "/home/zlwang/SPARK/TrackerAdaptation/adapt/crop_dataset.py", line 135, in getitem
crop = self.transforms(crop.permute(2,0,1)) # HW3 to 3HW
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torchvision/transforms/transforms.py", line 95, in call
img = t(img)
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torchvision/transforms/transforms.py", line 346, in forward
return F.resize(img, self.size, self.interpolation, self.max_size, self.antialias)
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torchvision/transforms/functional.py", line 476, in resize
return F_t.resize(img, size=output_size, interpolation=interpolation.value, antialias=antialias)
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torchvision/transforms/functional_tensor.py", line 469, in resize
img = interpolate(img, size=size, mode=interpolation, align_corners=align_corners, antialias=antialias)
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torch/nn/functional.py", line 3950, in interpolate
return torch._C._nn.upsample_bilinear2d(input, output_size, align_corners, scale_factors)
RuntimeError: Input and output sizes should be greater than 0, but got input (H: 0, W: 0) output (H: 224, W: 224)

Why will this happen and how to solve it?

[KelianB]

Before images are passed to SMIRK, they are tightly cropped around the face using facial landmarks. The error indicates that this step failed and returned a 0x0 crop box.
What you can do is add print(i) at the top of the __getitem__ method in crop_dataset.py and see the frame index that triggers the error. Then multiply it by 4 (this is the sample_idx_ratio value, which determines how we sample the input sequence for training) and look at that frame in your pre-processing data. Is the face maybe out of frame, too dark, very blurry or something like that?

[Michael-wzl]

Most of the face, except for one of the ears, is within the frame in the original image, but only half of the face remains in the cropped image. Is it possible to skip these problematic images, or is it possible to improve the effect of cropping?

[Michael-wzl]

Another error occurred:
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2132 8528
(The upper part is the last output of print(i, i*4) at the top of the getitem method)
Epoch 1, Iter 1: 0%| | 0/24 [00:01<?, ?it/s]
Traceback (most recent call last):
File "/home/zlwang/SPARK/TrackerAdaptation/train.py", line 343, in
adapt_tracker(args, wrapper, dataset_train, dataset_val, dataset_test)
File "/home/zlwang/SPARK/TrackerAdaptation/train.py", line 130, in adapt_tracker
run_dict = wrapper(views, training=True, losses=True)
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zlwang/SPARK/TrackerAdaptation/adapt/wrapper.py", line 89, in forward
decoded: DecodedValues = self.decode(encoded, views, training)
File "/home/zlwang/SPARK/TrackerAdaptation/adapt/wrapper.py", line 83, in decode
decoded = self.decoder(encoded, views, training)
File "/home/zlwang/miniforge3/envs/SPARK/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zlwang/SPARK/TrackerAdaptation/adapt/face_decoder.py", line 154, in forward
cam.K = avatar.cams_K_train[seq_idx].clone()
IndexError: index 7 is out of bounds for dimension 0 with size 7

Is this error also the result of bad input data? But I have checked the original image; every part of the face is within the frame, and the face is not blurry and well-lighted.

[stas-polukeev]

@Michael-wzl
i did a workaround where in crop_dataset in get_item before resize i check if one of the sizes is zero, if so i return None instead of a dict
then in colate() in crop_dataset i check if in one item in batch is None I return None
then in tracker adaptation train.py i basically check
in the cycle for view in train_dataloader:
if view is None:
continue
this way you just skip the broken images and it works fine

[stas-polukeev]

@Michael-wzl
if my instructions are unclear, I'm sorry
you could tag me, and I'll provide exact code lines and files where the changes should be made

[Michael-wzl]

@stas-polukeev Thanks for the advice! Your instructions are pretty clear to me. But it would be great if you could provide the exact code lines and files so that I can verify my understanding.

[KelianB]

@stas-polukeev Feel free to make a pull request btw, i'll gladly review and merge it.

[stas-polukeev]

@KelianB Thanks, I've submitted a pull request! not very elegant, but it works. May be it's better to add some failure flag to the dict instead of changing it to None, but as the image being non-zero is central to this pipeline, if something is crashed that would be probably better for debugging
@Michael-wzl you can check the pull request

[KelianB]

Closing this now as the PR has been merged. @Michael-wzl let me know if you still run into issues.