A Datacenter Scale Distributed Inference Serving Framework

GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embeddings

A Docker-powered service for PDF document layout analysis. This service provides a powerful and flexible PDF analysis service. The service allows for the segmentation and classification of differen…

Negima is a Python package to extract phrases in Japanese text by using the part-of-speeches based rules you defined.

Wikipediaを用いた日本語の固有表現抽出データセット

OCR, layout analysis, reading order, table recognition in 90+ languages

A daily digest web app that scrapes and summarizes blogs, Reddit threads, GitHub trending, and Hacker-News-trending articles all in one place.

A blazing fast inference solution for text embeddings models

Official implementation of "TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models"

A simple yet powerful tool to turn traditional container/OS images into unprivileged sandboxes.

Simple RL training for reasoning

AJIMEE-Bench (Advanced Japanese IME Evaluation Benchmark)

Hackable and optimized Transformers building blocks, supporting a composable construction.

Supercharge Your Model Training

Preferred Generation Benchmark

Bringing BERT into modernity via both architecture changes and scaling

SONAR, a new multilingual and multimodal fixed-size sentence embedding space, with a full suite of speech and text encoders and decoders.

A modern alternative to ls

Scripts for creating a Japanese-English parallel corpus and training NMT models

YAST - Yet Another SPLADE or Sparse Trainer

Yomitoku is an AI-powered document image analysis package designed specifically for the Japanese language.

A Python package for intrinsic dimension estimation

Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https://arxiv.org/abs/2309.08351)