Skip to content

Easily load data from kafka to ClickHouse with high performance

Notifications You must be signed in to change notification settings

3-shake/clickhouse_sinker

This branch is 1 commit ahead of, 667 commits behind housepower/clickhouse_sinker:master.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

c325281 · May 6, 2019

History

27 Commits
May 6, 2019
May 6, 2019
Oct 13, 2018
Oct 13, 2018
May 6, 2019
May 6, 2019
May 6, 2019
May 6, 2019
Nov 7, 2018
May 6, 2019
May 6, 2019
May 6, 2019
Oct 13, 2018
Jul 2, 2018
May 6, 2019

Repository files navigation

clickhouse_sinker

clickhouse_sinker is a sinker program that consumes kafka message and import them to ClickHouse.

Features

  • Easy to use and deploy, you don't need write any hard code, just care about the configuration file
  • Custom parser support.
  • Support multiple sinker tasks, each runs on parallel.
  • Support multiply kafka and ClickHouse clusters.
  • Bulk insert (by config bufferSize and flushInterval).
  • Loop write (when some node crashes, it will retry write the data to the other healthy node)
  • Uses Native ClickHouse client-server TCP protocol, with higher performance than HTTP.

Install && Run

By binary files (suggested)

Download the binary files from release, choose the executable binary file according to your env, modify the conf files, then run ./clickhouse_sinker -conf conf

By source

  • Install Golang

  • Go Get

go get -u github.com/3-shake/clickhouse_sinker/...

cd $GOPATH/src/github.com/3-shake/clickhouse_sinker

go get -u github.com/kardianos/govendor

# may take a while
govendor sync
  • Build && Run
go build -o clickhouse_sinker bin/main.go

## modify the config files, then run it
./clickhouse_sinker -conf conf

Support parsers

  • Json

Supported data types

  • UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64
  • Float32, Float64
  • String
  • FixedString
  • DateTime (not support, cause we use timestamp UInt32, that's enough)

Configuration

See config example

Custom metric parser

  • You just need to implement the parser interface on your own
type Parser interface {
	Parse(bs []byte) model.Metric
}

See json parser

About

Easily load data from kafka to ClickHouse with high performance

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 99.8%
  • Makefile 0.2%