"Segmentation fault" error

[JulienPercheron]

Hello,

I tried to use this lib to classify points in the LIDAR HD database but i keep getting errors when I launch the script. The command I enter is :

python run.py \
task.task_name=predict \
predict.src_las="./TestLIDARHD.las" \
predict.output_dir="./out" \
predict.gpus=[1] \
datamodule.batch_size=50

The result is always either just

Segmentation fault

or

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted

I think it comes from the limitation of my hardware but I wanted to be sure through this discussion.

My specs are :

• NVIDIA Quadro P400
• Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz
• 32Go of RAM
• Using WSL as I'm on a Windows system

Do you know if there's a solution or not ?
Thanks !

Julien

[JulienPercheron]

Following this discussion,

I tried to run Myria3D on a Linux system and encountered an unexpected error :

/home/jdelanoy/Documents/Semis_2021_0848_6516_LA93_IGN69.las ['/home/jdelanoy/Documents/Semis_2021_0848_6516_LA93_IGN69.las']
  0%|                                                                                                                             | 0/1 [00:00<?, ?it/s][2023-05-31 16:20:47,235][torch.distributed.nn.jit.instantiator][INFO] - Created a temporary directory at /tmp/tmp4318z03x
[2023-05-31 16:20:47,236][torch.distributed.nn.jit.instantiator][INFO] - Writing /tmp/tmp4318z03x/_remote_module_non_sriptable.py
Centers:   0%|                                                                                                                  | 0/400 [00:00<?, ?it/s]
0it [00:42, ?it/s]
  0%|                                                                                                                             | 0/1 [00:52<?, ?it/s]
Error executing job with overrides: ['task.task_name=predict', 'predict.src_las=/home/jdelanoy/Documents/Semis_2021_0848_6516_LA93_IGN69.las', 'predict.output_dir=./out', 'predict.gpus=[0]', 'datamodule.batch_size=50']
Traceback (most recent call last):
  File "/home/jdelanoy/myria3d/run.py", line 113, in <module>
    launch_predict()
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/hydra/main.py", line 48, in decorated_main
    _run_hydra(
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/utils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/jdelanoy/myria3d/run.py", line 72, in launch_predict
    predict(config)
  File "/home/jdelanoy/myria3d/myria3d/utils/utils.py", line 159, in timed
    result = method(*args, **kwargs)
  File "/home/jdelanoy/myria3d/myria3d/predict.py", line 63, in predict
    for batch in tqdm(datamodule.predict_dataloader()):
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1224, in _next_data
    return self._process_data(data)
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1250, in _process_data
    data.reraise()
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/torch/_utils.py", line 457, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/jdelanoy/miniconda3/envs/myria3d/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "/home/jdelanoy/myria3d/myria3d/pctl/dataset/iterable.py", line 54, in get_iterator
    sample_data = self.points_pre_transform(sample_points)
  File "/home/jdelanoy/myria3d/myria3d/pctl/points_pre_transform/lidar_hd.py", line 36, in lidar_hd_pre_transform
    assert points[color].max() <= COLORS_NORMALIZATION_MAX_VALUE
ValueError: no field of name Red

The file I used was a tile taken from this link covering terrain around the city of Lyon (the one which include the Parilly park to be more precise).
To go from .laz to .las I used the laszip tool (tho the link to the download seems to be down at the moment so i used a .exe found at this adress.
The command entered was :

python run.py task.task_name=predict predict.src_las=/home/jdelanoy/Documents/Semis_2021_0848_6516_LA93_IGN69.las predict.output_dir=./out predict.gpus=[0] datamodule.batch_size=50

The computer's GPU was a NVIDIA GeForce RTX 3080 and it was running on Ubuntu 22.04 (popOS).

The error might be linked to the file this time (same file used as the first issue) but I don't know what I can do at this point.

Thank you for your understanding.
Julien

[CharlesGaydon]

Hi @JulienPercheron,
thanks for the well-detailed issue.

---

On Windows, I think you were right to say :

I think it comes from the limitation of my hardware but I wanted to be sure through this discussion.

A simple workaround to that is to reduce the datamodule.batch_size to something like 25 (or even lower - but better not go too low, unexpected things happen in pytorch lightning when you go to values like 1 or 2 haha).

---

On Linux, I am happy to report that this issue is easy to resolve: the AI model expects colorized Lidar (RGB+NIR). Here is a link that explains how to do it: #75 (comment).

On a side note: one user reported replacing colors with the Intensity channel. But I would advice to use actual orthoimages instead via the aforementioned method.

Keep me posted!

[JulienPercheron]

Hello,

Thanks for your answer !
I finally got to try your solutions and here are the results :

On Windows

Same error, unfortunately. It seems that my hardware really is the limiting factor and I can't do anything about it.
Since I didn't get that from the documentation, maybe it could be useful to specify somewhere that you need a pretty good hardware to launch the lib ?

On Linux

I tried the solution described on #75 and after struggling to get it to work (mostly due to the multiple dependencies) I succeeded to get first a colorized Lidar, and then classify it using Myria3D. However, it is to be noted that I got a GPU out of memory exception twice before getting it to work. Reducing the batch_size to 5 worked but that means that maybe it could be useful to specify hardware requirements in the documentation since it seems really high.
Another thing, I didn't get that you needed a colored Lidar to get the classification working by reading the doc. I saw in the issue #75 that you wanted to put the info in the doc in the Preparing data for training section. I would suggest to not put it here because if you only want to use pre-trained models you might skip this part of the doc, at least that's what I did.

Anyway, thank you for the help provided on the issue and I hope this discussion can help other people too !

Julien