[WIP] Resources for AI engineers. Also contains supporting materials for the book AI Engineering (Chip Huyen, 2025)

A curated list of Large Language Model resources, covering model training, serving, fine-tuning, and building LLM applications.

Linux tool to show progress for cp, mv, dd, ... (formerly known as cv)

A playbook for systematically maximizing the performance of deep learning models.

Data and software for building the ACL Anthology.

MS MARCO(Microsoft Machine Reading Comprehension) is a large scale dataset focused on machine reading comprehension and question answering

Tools for merging pretrained large language models.

IBM-Generative-AI is a Python library built on IBM's large language model REST interface to seamlessly integrate and extend this service in Python programs.

A Detailed Introduction to My Favorite Statistical Measure, Hoeffding's D

Accelerate your Hugging Face Transformers 7.6-9x. Native to Hugging Face and PyTorch.

🐍💯pySBD (Python Sentence Boundary Disambiguation) is a rule-based sentence boundary detection that works out-of-the-box.

Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.

All things prompt engineering

A Python library for calculating a large variety of metrics from text

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools

A collection of large question answering datasets

A collection of large question answering datasets

The prime repository for state-of-the-art Multilingual Question Answering research and development.

A framework for few-shot evaluation of language models.

The simplest, fastest repository for training/finetuning medium-sized GPTs.

OpenLLaMA, a permissively licensed open source reproduction of Meta AI’s LLaMA 7B trained on the RedPajama dataset

Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"

A collaborative catalog of NLP resources for Indic languages

An open collection of implementation tips, tricks and resources for training large language models

A modular RL library to fine-tune language models to human preferences

Shoonya - Platform to Annotate and label data at scale.

Code Repository for the IndicXNLI paper.

Argilla is a collaboration tool for AI engineers and domain experts to build high-quality datasets