NeuSyRE: A Neuro-Symbolic Visual Understanding and Reasoning Framework based on Scene Graph Enrichment

Downloads

• VG dataset
• CSKG embeddings
• COCO dataset
• Faster RCNN pretrained network
• SGG pretrained network
• Image Captioning Pretrained Network and Preprocessed Dataset
• Image Generation Pretrained Network

Code

Scene Graph Generation and Enrichment

• Scene Graph Generation evaluation on VG/COCO dataset
• Scene Graph Enrichment using CSKG with evaluation

Downstream Tasks

• Image Captioning using COCO dataset
• Image Captioning evaluation
• Image Generation using VG dataset

Important Directories

VG dataset

• Generated scene graphs of all images (in Eval_IO/vg/0_images) in the dataset are saved to Eval_IO/vg/1_pred_scene_graphs
• Enriched scene graphs are saved to Eval_IO/vg/2_enriched_scene_graphs
• Generated images are saved to Eval_IO/vg/3_generated_images

COCO dataset

• Generated scene graphs of all images (in Eval_IO/coco/0_images) in the dataset are saved to Eval_IO/coco/1_pred_scene_graphs
• Enriched scene graphs are saved to Eval_IO/coco/2_enriched_scene_graphs
• Generated captions are saved to Eval_IO/coco/3_captions