Event Retrieval from Video - HCMAIC 2023

1. Our team name: Science AIO
2. My roles: Leader - Researcher - Developer

System pipeline

Usage

Setup

conda create -n py38 python==3.8
conda activate py38
pip install git+https://github.com/openai/CLIP.git
pip install -r requirements.txt

Data Preparation

• Download all the images here. We have applied various image compression methods. The results are compacted into 6GB.
• account.txt: is used for login and submission.
• dataframe_Lxx.csv: detection database

You can choose 1 of the following 3 versions for testing.

version	description	json	faiss bin v1	faiss bin v2
full	contains 3 frames in each scene (begin, middle, end)	full_path_v1.json	full_faiss_v1.bin	full_faiss_v2.bin
standard	contains 2 frames in each scene (begin, end)	full_path_v3.json	full_faiss_v3.bin	full_faiss_v4.bin
lightweight	contains 1 frame in each scene (middle)	full_path_v5.json	full_faiss_v5.bin	full_faiss_v6.bin

Run

python app.py

Run this URL in your browser: http://0.0.0.0:5001/home?index=0

Note: I use 2 versions of CLIP to increase the diversity of displayed results.

Documents

Faiss

1. Faiss: Facebook AI Research Search Similarity (Docs)

CLIP and Prompt Engineering for CLIP

1. Learning Transferable Visual Models From Natural Language Supervision - 2021 (Paper - GitHub - Blog)
2. How to Try CLIP: OpenAI's Zero-Shot Image Classifier (Blog)
3. Learning to Prompt for Vision-Language Models - CoOp - 2022 (Paper)
4. Towards Robust Prompts on Vision-Language Models - 2023 (Paper)
5. Prompt Engineering: The Magic Words to using OpenAI's CLIP - 2021 (Blog)

TransNet

1. TransNet: A Deep Network for Fast Detection of Common Shot Transitions (GitHub)
2. TransNet V2: Shot Boundary Detection Neural Network (GitHub)