Mochi Fine Tuning: Stage 2 - "video_fps" Error.

[MushroomFleet]

setup & install of Finetuner went ok,,

• Stage 1 complete correctly;

• During Stage 2 of the Fine Tuning process, every single video caused an error 'video_fps' although they were all checked to be 30 FPS videos as required.

> python3 /home/ubuntu/models/demos/fine_tuner/encode_videos.py videos_prepared/ --model_dir weights/ --num_gpus 1 --shape 37x480x848 --overwrite

Timing load_encoder
Timing load_decoder
Stage                   Time(s)    Percent
load_encoder               1.42     29.02%
load_decoder               3.47     70.98%
...
Processing videos_prepared/20241127_115021_1f3903f8_segment3.mp4                                                            
Traceback (most recent call last):                                                                                          
  File "/home/ubuntu/models/demos/fine_tuner/encode_videos.py", line 130, in batch_process                                  
    preprocess(
  File "/home/ubuntu/models/demos/fine_tuner/encode_videos.py", line 44, in preprocess                                      
    fps = metadata["video_fps"]
KeyError: 'video_fps'                                                                                                       
Error processing videos_prepared/20241127_115021_1f3903f8_segment3.mp4: 'video_fps'  
...

Not sure what the problem is here?

[MushroomFleet]

seems like Stage 3 proceeds regardless of the above error, but unsure

[MushroomFleet]

bash ./demos/fine_tuner/run.bash -c ./demos/fine_tuner/configs/lora.yaml -n 1

Starting training with 1 GPU(s), mode: single_gpu
Using config: ./demos/fine_tuner/configs/lora.yaml
model=weights/dit.safetensors, optimizer=, start_step_num=0
Found 27 training videos in videos_prepared
Loaded 0/27 valid file pairs.
Traceback (most recent call last):
  File "/home/ubuntu/models/demos/fine_tuner/train.py", line 396, in <module>
    main()
  File "/home/ubuntu/models/.venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/models/.venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/models/.venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/models/.venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/ubuntu/models/demos/fine_tuner/train.py", line 201, in main
    train_dl = torch.utils.data.DataLoader(
  File "/home/ubuntu/models/.venv/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 376, in __init__
    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
  File "/home/ubuntu/models/.venv/lib/python3.10/site-packages/torch/utils/data/sampler.py", line 164, in __init__
    raise ValueError(
ValueError: num_samples should be a positive integer value, but got num_samples=0
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

videos.mp4 + captions.txt + embed.pt = all in place however:

• Found 27 training videos in videos_prepared
• Loaded 0/27 valid file pairs.

cannot proceed. ValueError: num_samples should be a positive integer value, but got num_samples=0

unfortunately, steps to do not provide a route to start training.

[junsukha]

check bc is installed. If not, apt-get install bc
Make sure DURATION=$(printf "%.1f" "$(echo "($NUM_FRAMES / 30) + 0.09" | bc -l)") line in preprocess.bash works.
stage1 might appear to be completed but actually it's not.