“nan" problem in training custom dataset

[ohmyhair151263]

Epoch: [18] [3500/4384] eta: 0:08:08 lr: 0.000003 loss: 31.5320 (30.7038) loss_vfl: 0.6807 (0.6671) loss_bbox: 0.1400 (0.1503) loss_giou: 0.4664 (0.4601) loss_fgl: 1.0752 (1.0601) loss_vfl_aux_0: 0.7646 (0.7457) loss_bbox_aux_0: 0.1571 (0.1619) loss_giou_aux_0: 0.5047 (0.4856) loss_fgl_aux_0: 1.1022 (1.0939) loss_ddf_aux_0: 0.0534 (0.0495) loss_vfl_aux_1: 0.7661 (0.7237) loss_bbox_aux_1: 0.1372 (0.1529) loss_giou_aux_1: 0.4784 (0.4662) loss_fgl_aux_1: 1.0835 (1.0638) loss_ddf_aux_1: 0.0085 (0.0080) loss_vfl_aux_2: 0.7612 (0.7056) loss_bbox_aux_2: 0.1392 (0.1508) loss_giou_aux_2: 0.4685 (0.4611) loss_fgl_aux_2: 1.0754 (1.0603) loss_ddf_aux_2: 0.0014 (0.0015) loss_vfl_aux_3: 0.7124 (0.6810) loss_bbox_aux_3: 0.1400 (0.1503) loss_giou_aux_3: 0.4668 (0.4602) loss_fgl_aux_3: 1.0765 (1.0602) loss_ddf_aux_3: 0.0002 (0.0003) loss_vfl_aux_4: 0.6865 (0.6691) loss_bbox_aux_4: 0.1401 (0.1503) loss_giou_aux_4: 0.4664 (0.4601) loss_fgl_aux_4: 1.0752 (1.0601) loss_ddf_aux_4: 0.0001 (0.0001) loss_vfl_pre: 0.7646 (0.7465) loss_bbox_pre: 0.1566 (0.1618) loss_giou_pre: 0.5061 (0.4844) loss_vfl_enc_0: 0.7573 (0.7581) loss_bbox_enc_0: 0.1973 (0.1956) loss_giou_enc_0: 0.5716 (0.5552) loss_vfl_dn_0: 0.5078 (0.4945) loss_bbox_dn_0: 0.2006 (0.2113) loss_giou_dn_0: 0.4881 (0.4648) loss_fgl_dn_0: 1.1960 (1.2022) loss_ddf_dn_0: 0.2418 (0.2392) loss_vfl_dn_1: 0.4163 (0.4028) loss_bbox_dn_1: 0.1490 (0.1491) loss_giou_dn_1: 0.3783 (0.3514) loss_fgl_dn_1: 1.0866 (1.0820) loss_ddf_dn_1: 0.0355 (0.0327) loss_vfl_dn_2: 0.4009 (0.3814) loss_bbox_dn_2: 0.1394 (0.1393) loss_giou_dn_2: 0.3552 (0.3362) loss_fgl_dn_2: 1.0661 (1.0647) loss_ddf_dn_2: 0.0066 (0.0064) loss_vfl_dn_3: 0.3857 (0.3712) loss_bbox_dn_3: 0.1330 (0.1367) loss_giou_dn_3: 0.3452 (0.3318) loss_fgl_dn_3: 1.0660 (1.0648) loss_ddf_dn_3: 0.0003 (0.0003) loss_vfl_dn_4: 0.3799 (0.3678) loss_bbox_dn_4: 0.1316 (0.1363) loss_giou_dn_4: 0.3448 (0.3314) loss_fgl_dn_4: 1.0670 (1.0650) loss_ddf_dn_4: 0.0000 (0.0000) loss_vfl_dn_5: 0.3794 (0.3678) loss_bbox_dn_5: 0.1316 (0.1363) loss_giou_dn_5: 0.3448 (0.3314) loss_fgl_dn_5: 1.0670 (1.0650) loss_vfl_dn_pre: 0.5098 (0.4947) loss_bbox_dn_pre: 0.2113 (0.2182) loss_giou_dn_pre: 0.4908 (0.4658) time: 0.4990 data: 0.0280 max mem: 20978
tensor([[[nan, nan, nan, nan],
[nan, nan, nan, nan],
[nan, nan, nan, nan],
...,
[nan, nan, nan, nan],
[nan, nan, nan, nan],
[nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    ...,

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]]], device='cuda:0', grad_fn=<SelectBackward0>)

Traceback (most recent call last):
File "/home/GZH/net/DEIM/./train.py", line 86, in
main(args)
File "/home/GZH/net/DEIM/./train.py", line 56, in main
solver.fit()
File "/home/GZH/net/DEIM/engine/solver/det_solver.py", line 76, in fit
train_stats = train_one_epoch(
^^^^^^^^^^^^^^^^
File "/home/GZH/net/DEIM/engine/solver/det_engine.py", line 65, in train_one_epoch
loss_dict = criterion(outputs, targets, **metas)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/GZH/net/DEIM/engine/deim/deim_criterion.py", line 276, in forward
indices = self.matcher(outputs_without_aux, targets)['indices']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/GZH/net/DEIM/engine/deim/matcher.py", line 101, in forward
cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), box_cxcywh_to_xyxy(tgt_bbox))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/GZH/net/DEIM/engine/deim/box_ops.py", line 53, in generalized_box_iou
assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
It comes up every time I'm halfway through training