index.md

Low Latency System

A collection of everything about low latency systems.

The goal of this project is to collect resources that help people to get better performance in their system.

Leave a message

If you have anything about this project, please submit a GitHub issue here.

Linux OS tuning

• Redhat Linux Performance Tuning Guide An essential manual of tuning Linux OS. Concepts and tools are introduced in this article. It's a perfect entry of performance optimization. It works with RHEL and Centos. Other distributions may have different toolchains. But the concepts are similar. Please select the right version number for your system.

• Low Latency Performance Tuning for Red Hat Enterprise Linux 7 Based on concepts introduced in the article above, this article focuses on how to get low latency. To access this article needs a Red Hat account.

• Red Hat Enterprise Linux Network Performance Tuning Guide How to tune network stack in Linux kernel. Most of low latency trading system use kernel bypass to reduce network latency.

• Talks on Redhat Summit: Performance analysis and tuning of Red Hat Enterprise Linux, This talk series shows how engineers think about performance tuning and their practices.

  • Videos: 2018 Part 1, 2018 Part 2
  • Notes of 2015 talk

System tuning guides from server vendors

Server vendors have their tuning guides. Generally, a tuning guide from a server vendor contains similar sections which are hardware tuning, OS tuning, network tuning, benchmark test, etc. In a hardware tuning section, there would be some options about turning on high performance mode on a server. Some subtle differences would be in OS tuning section.

HP

• Configuring and tuning HP ProLiant Servers for low-latency applications This document updates when HP releases new sku. The latest version is 201710.

• HPE Trade and Match Server: HPE ProLiant XL1x0r Gen9 servers using Intel Xeon E5-1680 v3 processors A solution that uses 1 overclocking CPU to get higher CPU frequency than dual socket servers.

• HPE Gen10 Servers Intelligent System Tuning HP's new tech to remove jitters in G10 system.

• HP Gen8 technologies for low latency, high-performance trading, and exchanges

• HP Gen8 technologies for low latency, high performance trading and exchanges

• Optimizing Linux performance on HP Integrity Superdome X

Supermicro

• Supermicro HFT-Optimized Servers and the ExaNIC

Dell

• Configuring Low-Latency Environments on Dell PowerEdge Servers

• Optimizing the High Frequency Trading GatiRT* Application on the latest Intel® Architecture Server

• Reducing OS-Jitter, (Frame Latency, Latency, Stutter)

lenovo

• Linux Performance and Tuning Guidelines
  • 中文版

Tuning Guides from Intel and AMD

Few years ago, Intel Xeon CPU was the only choice for a low latency system. It provides enough cores, CPU frequency, cache capacity, etc. Overclocked dual 2600 series CPU was the mainstream solution. Some scenarios would choose a single 1600 series CPU for higher frequency. Xeon CPU has a problem. They only could be overclocked via the base clock. That means only 1 digit percentage of frequency increment. Core i5/i7 series CPU could provide a high overclocked ratio. But the core number was a problem until core i9 released. A top core i9 CPU like i9-7980XE has 18 cores with 4.2 GHz turbo frequency. Some vendors can make all cores running at 4.6+ GHz. A single socket solution removes the latency between CPUs. Servers based on i9 is a mainstream solution now. The same thing happened with AMD. AMD's EPYC CPU is another choice now.

Intel

• Optimizing Computer Applications for Latency: configuring the hardware

• Optimizing Computer Applications for Latency: tuning applications

• Optimizing for low latency

AMD

• Performance tuning guidelines

Network Tuning Guides

Solarflare

• Solarflare Server Adapter User Guide This user guide contains a chapter Performance Tuning on Linux which

Mellanox

• Performance Tuning Guide for Mellanox Network Adapters

  • out date
  • the latest

• How to tune libvma and other tools from mellanox to improve network performance.

  • VMA Performance Tuning Guide
  • Mellanox AcademyIntroduction to InfiniBand

Other

• Network tuning and performance

Benchmarks and tools

• STAC
  • SUT ID: SFC170206 STAC-Network IO UDP over 10GbE using OpenOnload on RHEL 6.6 with Solarflare SFN 8522-PLUS Adapters on HPE ProLiant XL170r Gen9 Trade & Match Servers This test discloses the configurations.

Tools

• Sockperf

• iperf

Faster than 10G network

• Extreme Ethernet Server Networking: 10GbE, 25GbE, 40GbE & 50GbE

• A lecture talk about Linux kernel networking

• A great blog from cloudflare

  • How to receive a million packets
  • Why we use the Linux kernel's TCP stack

• Multi-queue network interfaces with SMP on Linux

  • 中译本：linux下基于SMP架构的多队列网卡的调优

Low Level API

• IB verbs

• ef_vi

• libexanic

Kernel bypass

Solarflare

• OpenOnload Google Talk Compared different accelerate technology approach that Solarflare used in OpenOnload compared latency, bandwidth by using openonload and kernel

• ASICs argued with FPGAs

• TCP on FPGA made practical and fast

Infiniband, RMDA

• THE GEEK IN THE CORNER Programming odds and ends — InfiniBand, RDMA, and low-latency networking for now.

  A blog for RDMA programming and concept

  • An introduction of infiniband and RDMA

• RDMA programming concepts

• InfiniBand Technology Overview An advanced introduction to IB

• introduction to Infiniband just a simple introduction. The only advantage is there are many diagrams.

• Introduction to InfiniBand An introduction to IB from Mellanox whitepaper.

• RDMA aware programming user manual

• OpenOnload NIC / Infiniband RDMA

• Designing Cloud and Grid Computing Systems with InfiniBand and High-Speed Ethernet

• Get Started with Infiniband

Other

• PF_RING

Zero Copy

• Zero Copy I: User-Mode Perspective

Time tech

• IEEE 1588 & PTP USING EMBEDDED LINUX SYSTEMS

• How to Benchmark Code Execution Times in Intel IA-32 and IA-64 Instruction Set Architectures

Jitter detection

• sysjitter from solarflare

  A tool that measures the jitter experienced by user-level threads running on each CPU core of a computer. Such jitter may be caused by interrupts, OS threads, system management interrupts or other processes on the system.

  All openOnload download is here http://www.openonload.org/download/

• Erez's hiccup tools

  A tool used for measure OS hiccup.

• HP-timetest I don't get this tool now. Need help.

• Google Group: Mechanical-Sympathy

  • A Toolkit to Measure Basic System Performance & OS Jitter
  • Systemic to reduce Linux OS jitter

Unsort

These resources will be moved to right topics later.

• Performance Analysis

• A Low-Latency Solution for High Frequecy Trading From IBM and Solarflare It's a very old solution that looks like a system has not been tuned well.

• Low Latency in Linux kernel An article from 2000. Very very old.

• Optimizing System Performance

• Design Best Practices for Latency Optimization

• Low Latency Optimization

• Intel i350 NIC datasheet

• Network in Linux kernel

• Understanding TCP/IP network stack and writing network programming

• Trading floor architecture

• Latency in high performance trading system

• How does an algorithmic trading system architecture look like

• Evolution and practice: Low-latency Distributed Applications in Finance

• Technology in banking – a problem in scale and complexity

• Low-latency networks for trading infrastructure Fujitsu An introduction of Fujitsu's optical infrastructure.

• Topics in High Performance Messaging

• intelligent trading technology from A-Team

• nextplatform

Other

• A blog from a guy working in LMAX
  • Reducing system jitters part 1
  • Reducing system jitters part 2

Report, Whitepaper

• Tolly Test Report:Mellanox Spectrum vs. Broadcom StrataXGS Tomahawk 25GbE & 100GbE Performance Evaluation
• The Next Frontiers for Low Latency Programmers
• Webinar from Mellanox: The latest Trend of Low Latency Programming
• Verbs Programming, Raw Ethernet
• Low Latency Optimization
• Tolly Test Report:Mellanox Spectrum vs. Broadcom StrataXGS Tomahawk 25GbE & 100GbE Performance Evaluation
• The Next Frontiers for Low Latency Programmers
• Webinar from Mellanox: The latest Trend of Low Latency Programming
• Verbs Programming, Raw Ethernet

FPGA

• algo-logic

• A Low Latency Library in FPGA Hardware for High Frequency Trading (HFT) An article from alog-logic. Here is the external link. The origin link on algo-logic.com is 404 error.

• Efficient Event Processing through Reconfigurable Hardware for Algorithmic Trading

• NetFPGA-10G Information A FPGA solution in 2012. Should update to latest news.

• FPGA networking processing card

• FPGA smart NIC card A list of FPGA solution of NIC. A 2013 article, it's a little expired.

• FPGA options for ultra low latency HFT with stunningly convincing Youtube video with Algo Logic with some technical papers A 2013 article also.

• Xeon+FPGA platform for data center

• Intel hybrid CPU+FPGA

• FPGA developer A FPGA site that focuses on tech not finance solution.

• How are HFT systems implemented on FPGA nowadays A QA on stackexchange

Programming languages

The choices are not so many for low latency system.

Java

• Zing. A JVM solution Zing is a JVM solution for performance critical usage.
• Ultra Low Latency Trading Systems: A blog about techniques used to build www.SubMicroTrading.com

C++

• CppCon 2017: Fedor Pikus “C++ atomics, from basic to advanced. What do they really do?”
• ACCU 2017: Lock-free programming with modern C++ - Timur Doumler
• CppCon 2014: Herb Sutter "Lock-Free Programming Part I"
• CppCon 2014: Herb Sutter "Lock-Free Programming Part II"

About me

• My LinkedIn Page
• [email protected]
• Wechat if you know: trador