Any advice to train on Chinese Dataset using Caption Contrastive Fine-tuning?

[WeihongM]

Hello, thx for the impressive work. I find the model perform poor when use Chinese caption. If I want to use caption contrastive finetuning loss to train on LLM support Chinese (such as qwen), which dataset do you advise me to use?

[raytrun]

Thank you for your interest in our work.
The WuKong Dataset is a large-scale Chinese image-text pairs dataset and could be a good choice, given its substantial volume of data. However, I’m not entirely sure about the quality of the captions in this dataset. You may need to check and see if it’s suitable.

[MrPanda007]

Thank you for your interest in our work. The WuKong Dataset is a large-scale Chinese image-text pairs dataset and could be a good choice, given its substantial volume of data. However, I’m not entirely sure about the quality of the captions in this dataset. You may need to check and see if it’s suitable.

@raytrun Thanks for you great work.
Now, I use your cc-finetuned llama model to finetune eva model with cc15m and my own chinese data(1 millions), but it performances worse than chinese-clip.
If I want to use llm2clip model in Chinese, shuold I use your cc-finetune LLama model to train clip, or I need to finetune both llama and clip model?