Molmo-7B-D: Couldn't load caption model: Unknown Error

[2snEM6]

I've downloaded the whole folder from https://huggingface.co/allenai/Molmo-7B-D-0924/tree/main and I've added it to the list of models.

However, when trying to caption an image using Molmo, I get: Couldn't load caption model: Unknown Error

Are there any logs stored somewhere that I can read and investigate?

Thanks!

[FennelFetish]

Unknown Error means it probably crashed without even raising an exception.
This could be due to low VRAM, or more likely, setup failed/installed incompatible packages.

Did you test other models or is this only a problem with Molmo?

When using the run scripts, logs are written to last.log inside the qapyq folder. This file is overwritten and cleared during startup.

If that's empty, you could try with this example script (the same that I also posted in your other issue):
Place it as test.py in qapyq's top folder, change the path of model and image, and run it in the same virtual environment.

from infer.backend_molmo import MolmoBackend

def main():
    config = {
        "model_path": "/mnt/ai/Models/MM-LLM/Molmo-7B-D-0924/",
        "gpu_layers": -1,
        "vis_gpu_layers": -1,
        "quantization": "none",
        "max_tokens": 20
    }

    molmo = MolmoBackend(config)

    prompts = [{"caption": "Describe this image in two sentences."}]
    answers = molmo.caption("/home/rem/Pictures/red-tree-with-eyes.jpeg", prompts)
    print(answers)

if __name__ == "__main__":
    main()

Running on Linux:

cd qapyq
source .venv/bin/activate
python test.py

Running on Windows (didn't test):

cd qapyq
.venv\Scripts\python.exe test.py

[zixzaxlad]

I’m dealing with the same issue, but I’m using the InternVL2 model. I made some modifications to test.py, and although the flash attention isn’t working, it still gives me an answer. So, I think the model is functioning correctly.

Loading model configuration...
Initializing InternVL2Backend with given configuration...
D:\AI\qapyq.venv\Lib\site-packages\timm\models\layers_init_.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
FlashAttention2 is not installed.
flash-attention package not found, consider installing for better performance: No module named 'flash_attn'.
Current flash-attenton does not support window_size. Either upgrade or use attn_implementation='eager'.
Warning: Flash attention is not available, using eager attention instead.
Phi3ForCausalLM has generative capabilities, as prepare_inputs_for_generation is explicitly overwritten. However, it doesn't directly inherit from GenerationMixin. From 👉v4.50👈 onwards, PreTrainedModel will NOT inherit from GenerationMixin, and this model will lose the ability to call generate and other related functions.

• If you're using trust_remote_code=True, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
• If you are the owner of the model architecture code, please modify your model class such that it inherits from GenerationMixin (after PreTrainedModel, otherwise you'll get an exception).
• If you are not the owner of the model architecture class, please contact the model code owner to update it.
  Warning: Flash attention is not available, using eager attention instead.
  Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 2/2 [00:04<00:00, 2.19s/it]
  Model initialized successfully.
  Running inference on image...
  The seen_tokens attribute is deprecated and will be removed in v4.41. Use the cache_position model input instead.
  You are not running the flash-attention implementation, expect numerical differences.
  Inference completed.
  Answers: {'caption': 'The image features a young woman with long, dark hair and large hoop earrings, gazing thoughtfully into the distance. She is set against a blurred natural background, suggesting an outdoor setting.'}

[2snEM6]

Unknown Error means it probably crashed without even raising an exception. This could be due to low VRAM, or more likely, setup failed/installed incompatible packages.

Did you test other models or is this only a problem with Molmo?

When using the run scripts, logs are written to last.log inside the qapyq folder. This file is overwritten and cleared during startup.

If that's empty, you could try with this example script (the same that I also posted in your other issue): Place it as test.py in qapyq's top folder, change the path of model and image, and run it in the same virtual environment.

from infer.backend_molmo import MolmoBackend

def main():
    config = {
        "model_path": "/mnt/ai/Models/MM-LLM/Molmo-7B-D-0924/",
        "gpu_layers": -1,
        "vis_gpu_layers": -1,
        "quantization": "none",
        "max_tokens": 20
    }

    molmo = MolmoBackend(config)

    prompts = [{"caption": "Describe this image in two sentences."}]
    answers = molmo.caption("/home/rem/Pictures/red-tree-with-eyes.jpeg", prompts)
    print(answers)

if __name__ == "__main__":
    main()

Running on Linux:

cd qapyq
source .venv/bin/activate
python test.py

Running on Windows (didn't test):

cd qapyq
.venv\Scripts\python.exe test.py

Just tried this and it works! However using the UI doesn't.

Interstingly, if I change:

qapyq/run.bat

Line 15 in 47b54ce

set PYTHON="%VENV_DIR%\Scripts\pythonw.exe"

to use python.exe instead of pythonw.exe it works.

[FennelFetish]

Just tried this and it works! However using the UI doesn't.

Interstingly, if I change:

qapyq/run.bat

Line 15 in 47b54ce

set PYTHON="%VENV_DIR%\Scripts\pythonw.exe"

to use python.exe instead of pythonw.exe it works.

Good catch! I could reproduce the error. Apparently, I didn't test this well enough as I didn't expect changing the run script could cause issues, sorry.

I updated the run.bat and it does work for me now, also when using pythonw. I suspect the start command broke the pipes.
Can you verify?

Thanks!