A golang-based data loader which can be used from Python. Focused on a VectorDB stack at the moment, fetching and processing data per sample at GB/s speeds.

[NeurIPS'24] HippoRAG is a novel RAG framework inspired by human long-term memory that enables LLMs to continuously integrate knowledge across external documents. RAG + Knowledge Graphs + Personali…

PyTorch code for hierarchical k-means -- a data curation method for self-supervised learning

scalable and robust tree-based speculative decoding algorithm

FastFit ⚡ When LLMs are Unfit Use FastFit ⚡ Fast and Effective Text Classification with Many Classes

Python & JS/TS SDK for running AI-generated code/code interpreting in your AI app

Recursive Enriching Pterodactyl Tree Augmented Retrieval (REPTAR) is a system that uses a recursive summarization approach to generate thoughtful summaries of text data.

A Collection of Pydantic Models to Abstract IRL

This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?

Convert PDF to markdown quickly with high accuracy

DSPy: The framework for programming—not prompting—foundation models

An extensible benchmark for evaluating large language models on planning

Inference code for Persimmon-8B

Forward-Looking Active REtrieval-augmented generation (FLARE)

The official repo of Qwen-VL (通义千问-VL) chat & pretrained large vision language model proposed by Alibaba Cloud.

[ACL 2023] One Embedder, Any Task: Instruction-Finetuned Text Embeddings

(AAAI 2024) BLIVA: A Simple Multimodal LLM for Better Handling of Text-rich Visual Questions

Curate better data for LLMs

Get up and running with Llama 3.2, Mistral, Gemma 2, and other large language models.

Llama 2 Everywhere (L2E)

Chat language model that can use tools and interpret the results

ReactJS library for "Cells, Generators, and Lenses": object-oriented UI components to compose LLM-powered writing interfaces that support iteration and exploration.

Structured Text Generation

[ICLR 2023] Codebase for Copy-Generator model, including an implementation of kNN-LM

Datasets collection and preprocessings framework for NLP extreme multitask learning

🚀🎬 ShortGPT - Experimental AI framework for youtube shorts / tiktok channel automation