CosId Universal, flexible, high-performance distributed ID generator
CosId aims to provide a universal, flexible and high-performance distributed ID generator. Two types of ID generators are currently provided:
SnowflakeId: Stand-alone TPS performance：4,096,000 JMH Benchmark , It mainly solves two major problems ofSnowflakeId: machine number allocation problem and clock backwards problem and provide a more friendly and flexible experience.SegmentId: Get a segment (Step) ID every time to reduce the network IO request frequency of theIdSegmentdistributor and improve performance.IdSegmentDistributor:RedisIdSegmentDistributor:IdSegmentdistributor based on Redis.JdbcIdSegmentDistributor: The Jdbc-basedIdSegmentdistributor supports various relational databases.
SegmentChainId(recommend):SegmentChainId(lock-free) is an enhancement ofSegmentId, the design diagram is as follows.PrefetchWorkermaintains asafe distance, so thatSegmentChainIdachieves approximatelyAtomicLongTPS performance (Step 1000): 127,439,148+/s JMH Benchmark .PrefetchWorkermaintains a safe distance (safeDistance), and supports dynamicsafeDistanceexpansion and contraction based on hunger status.
SnowflakeId is a distributed ID algorithm that uses
Long(64 bits) bit partition to generate ID. The general bit allocation scheme is :timestamp(41 bits) +machineId(10 bits) +sequence(12 bits) = 63 bits 。
- 41 bits
timestamp= (1L<<41)/(1000/3600/365) approximately 69 years of timestamp can be stored, that is, the usable absolute time isEPOCH+ 69 years. Generally, we need to customizeEPOCHas the product development time. In addition, we can increase the number of allocated bits by compressing other areas， The number of timestamp bits to extend the available time. - 10 bits
machineId= (1L<<10) = 1024 That is, 1024 copies of the same business can be deployed (there is no master-slave copy in the Kubernetes concept, and the definition of Kubernetes is directly used here) instances. Generally, there is no need to use so many, so it will be redefined according to the scale of deployment. - 12 bits
sequence= (1L<<12) * 1000 = 4096000 That is, a single machine can generate about 409W ID per second, and a global same-service cluster can generate 40960001024=419430W=4.19 billion (TPS).
It can be seen from the design of SnowflakeId:
- 👍 The first 41 bits are a
timestamp,So SnowflakeId is local monotonically increasing, and affected by global clock synchronization SnowflakeId is global trend increasing. - 👍
SnowflakeIddoes not have a strong dependency on any third-party middleware, and its performance is also very high. - 👍 The bit allocation scheme can be flexibly configured according to the needs of the business system to achieve the optimal use effect.
- 👎 Strong reliance on the local clock, potential clock moved backwards problems will cause ID duplication.
- 👎 The
machineIdneeds to be set manually. If themachineIdis manually assigned during actual deployment, it will be very inefficient.
It mainly solves two major problems of SnowflakeId: machine number allocation problem and clock backwards problem and provide a more friendly and flexible experience.
Currently CosId provides the following three
MachineIddistributors.
cosid:
snowflake:
machine:
distributor:
type: manual
manual:
machine-id: 0Manually distribute
MachineId
cosid:
snowflake:
machine:
distributor:
type: stateful_setUse the stable identification ID provided by the
StatefulSetofKubernetesas the machine number.
cosid:
snowflake:
machine:
distributor:
type: redisUse Redis as the distribution store for the machine number.
cosid:
snowflake:
clock-backwards:
spin-threshold: 10
broken-threshold: 2000The default DefaultClockBackwardsSynchronizer clock moved backwards synchronizer uses active wait synchronization strategy, spinThreshold (default value 10 milliseconds) is used to set the spin wait threshold, when it is greater than spinThreshold, use thread sleep to wait for clock synchronization, if it exceeds BrokenThreshold (default value 2 seconds) will directly throw a ClockTooManyBackwardsException exception.
public class MachineState {
public static final MachineState NOT_FOUND = of(-1, -1);
private final int machineId;
private final long lastTimeStamp;
public MachineState(int machineId, long lastTimeStamp) {
this.machineId = machineId;
this.lastTimeStamp = lastTimeStamp;
}
public int getMachineId() {
return machineId;
}
public long getLastTimeStamp() {
return lastTimeStamp;
}
public static MachineState of(int machineId, long lastStamp) {
return new MachineState(machineId, lastStamp);
}
}cosid:
snowflake:
machine:
state-storage:
local:
state-location: ./cosid-machine-state/The default LocalMachineStateStorage local machine state storage uses a local file to store the machine number and the most recent timestamp, which is used as a MachineState cache.
cosid:
snowflake:
share:
clock-sync: trueThe default SnowflakeId will directly throw a ClockBackwardsException when a clock moved backwards occurs, while using the ClockSyncSnowflakeId will use the ClockBackwardsSynchronizer to actively wait for clock synchronization to regenerate the ID, providing a more user-friendly experience.
SnowflakeId snowflakeId = SafeJavaScriptSnowflakeId.ofMillisecond(1);The Number.MAX_SAFE_INTEGER of JavaScript has only 53 bits. If the 63-bit SnowflakeId is directly returned to the front end, the value will overflow. Usually we can convert SnowflakeId to String type or customize SnowflakeId Bit allocation is used to shorten the number of bits of SnowflakeId so that ID does not overflow when it is provided to the front end.
cosid:
snowflake:
share:
friendly: truepublic class SnowflakeIdState {
private final long id;
private final int machineId;
private final long sequence;
private final LocalDateTime timestamp;
/**
* {@link #timestamp}-{@link #machineId}-{@link #sequence}
*/
private final String friendlyId;
}public interface SnowflakeFriendlyId extends SnowflakeId {
SnowflakeIdState friendlyId(long id);
SnowflakeIdState ofFriendlyId(String friendlyId);
default SnowflakeIdState friendlyId() {
long id = generate();
return friendlyId(id);
}
} SnowflakeFriendlyId snowflakeFriendlyId=new DefaultSnowflakeFriendlyId(snowflakeId);
SnowflakeIdState idState = snowflakeFriendlyId.friendlyId();
idState.getFriendlyId(); //20210623131730192-1-0
cosid:
segment:
enabled: true
distributor:
type: redisInitialize the
cosidtable
create table if not exists cosid
(
name varchar(100) not null comment '{namespace}.{name}',
last_max_id bigint not null default 0,
last_fetch_time bigint not null,
constraint cosid_pk
primary key (name)
) engine = InnoDB;spring:
datasource:
url: jdbc:mysql://localhost:3306/test_db
username: root
password: root
cosid:
segment:
enabled: true
distributor:
type: jdbc
jdbc:
enable-auto-init-cosid-table: false
enable-auto-init-id-segment: trueAfter enabling enable-auto-init-id-segment:true, the application will try to create the idSegment record when it starts to avoid manual creation. Similar to the execution of the following initialization sql script, there is no need to worry about misoperation, because name is the primary key.
insert into cosid
(name, last_max_id, last_fetch_time)
value
('namespace.name', 0, unix_timestamp());
cosid:
segment:
enabled: true
mode: chain
chain:
safe-distance: 5
prefetch-worker:
core-pool-size: 2
prefetch-period: 1s
distributor:
type: redis
share:
offset: 0
step: 100
provider:
bizC:
offset: 10000
step: 100
bizD:
offset: 10000
step: 100cosid:
snowflake:
provider:
bizA:
# timestamp-bit:
sequence-bit: 12
bizB:
# timestamp-bit:
sequence-bit: 12IdGenerator idGenerator = idGeneratorProvider.get("bizA");In actual use, we generally do not use the same IdGenerator for all business services, but different businesses use different IdGenerator, then IdGeneratorProvider exists to solve this problem, and it is the container of IdGenerator , You can get the corresponding IdGenerator by the business name.
Kotlin DSL
val cosidVersion = "1.3.2";
implementation("me.ahoo.cosid:spring-boot-starter-cosid:${cosidVersion}")<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>demo</artifactId>
<properties>
<cosid.version>1.3.2</cosid.version>
</properties>
<dependencies>
<dependency>
<groupId>me.ahoo.cosid</groupId>
<artifactId>spring-boot-starter-cosid</artifactId>
<version>${cosid.version}</version>
</dependency>
</dependencies>
</project>spring:
application:
name: ${service.name:cosid-example}
datasource:
url: jdbc:mysql://localhost:3306/test_db
username: root
password: root
redis:
url: redis://localhost:6379
cosid:
namespace: ${spring.application.name}
snowflake:
enabled: true
# epoch: 1577203200000
clock-backwards:
spin-threshold: 10
broken-threshold: 2000
machine:
# stable: true
# machine-bit: 10
# instance-id: ${HOSTNAME}
distributor:
type: redis
# manual:
# machine-id: 0
state-storage:
local:
state-location: ./cosid-machine-state/
share:
clock-sync: true
friendly: true
provider:
bizA:
# timestamp-bit:
sequence-bit: 12
bizB:
# timestamp-bit:
sequence-bit: 12
segment:
enabled: true
mode: chain
chain:
safe-distance: 5
prefetch-worker:
core-pool-size: 2
prefetch-period: 1s
distributor:
type: redis
share:
offset: 0
step: 100
provider:
bizC:
offset: 10000
step: 100
bizD:
offset: 10000
step: 100
- The development notebook : MacBook Pro (M1)
- All benchmark tests are carried out on the development notebook.
- Deploying Redis on the development notebook.
gradle cosid-core:jmh
# or
java -jar cosid-core/build/libs/cosid-core-1.3.2-jmh.jar -bm thrpt -wi 1 -rf json -f 1Benchmark Mode Cnt Score Error Units
SnowflakeIdBenchmark.millisecondSnowflakeId_friendlyId thrpt 4020311.665 ops/s
SnowflakeIdBenchmark.millisecondSnowflakeId_generate thrpt 4095403.859 ops/s
SnowflakeIdBenchmark.safeJsMillisecondSnowflakeId_generate thrpt 511654.048 ops/s
SnowflakeIdBenchmark.safeJsSecondSnowflakeId_generate thrpt 539818.563 ops/s
SnowflakeIdBenchmark.secondSnowflakeId_generate thrpt 4206843.941 ops/s
gradle cosid-redis:jmh
# or
java -jar cosid-redis/build/libs/cosid-redis-1.3.2-jmh.jar -bm thrpt -wi 1 -rf json -f 1 RedisChainIdBenchmarkBenchmark Mode Cnt Score Error Units
RedisChainIdBenchmark.atomicLong_baseline thrpt 5 144541334.198 ± 5578137.471 ops/s
RedisChainIdBenchmark.step_1 thrpt 5 1874168.687 ± 310274.706 ops/s
RedisChainIdBenchmark.step_100 thrpt 5 114226113.524 ± 15789563.078 ops/s
RedisChainIdBenchmark.step_1000 thrpt 5 127439148.104 ± 1833743.699 ops/s
In statistics, a percentile (or a centile) is a score below which a given percentage of scores in its frequency distribution falls (exclusive definition) or a score at or below which a given percentage falls (inclusive definition). For example, the 50th percentile (the median) is the score below which (exclusive) or at or below which (inclusive) 50% of the scores in the distribution may be found.
java -jar cosid-redis/build/libs/cosid-redis-1.3.2-jmh.jar -bm sample -wi 1 -rf json -f 1 -tu us step_1000Benchmark Mode Cnt Score Error Units
RedisChainIdBenchmark.step_1000 sample 1336271 0.024 ± 0.001 us/op
RedisChainIdBenchmark.step_1000:step_1000·p0.00 sample ≈ 0 us/op
RedisChainIdBenchmark.step_1000:step_1000·p0.50 sample 0.041 us/op
RedisChainIdBenchmark.step_1000:step_1000·p0.90 sample 0.042 us/op
RedisChainIdBenchmark.step_1000:step_1000·p0.95 sample 0.042 us/op
RedisChainIdBenchmark.step_1000:step_1000·p0.99 sample 0.042 us/op
RedisChainIdBenchmark.step_1000:step_1000·p0.999 sample 0.042 us/op
RedisChainIdBenchmark.step_1000:step_1000·p0.9999 sample 0.208 us/op
RedisChainIdBenchmark.step_1000:step_1000·p1.00 sample 37.440 us/op
gradle cosid-jdbc:jmh
# or
java -jar cosid-jdbc/build/libs/cosid-jdbc-1.3.2-jmh.jar -bm thrpt -wi 1 -rf json -f 1 MySqlChainIdBenchmarkBenchmark Mode Cnt Score Error Units
MySqlChainIdBenchmark.atomicLong_baseline thrpt 5 145294642.937 ± 224876.284 ops/s
MySqlChainIdBenchmark.step_1 thrpt 5 35058.790 ± 36226.041 ops/s
MySqlChainIdBenchmark.step_100 thrpt 5 74575876.804 ± 5590390.811 ops/s
MySqlChainIdBenchmark.step_1000 thrpt 5 123131804.260 ± 1488004.409 ops/s
java -jar cosid-jdbc/build/libs/cosid-jdbc-1.3.2-jmh.jar -bm sample -wi 1 -rf json -f 1 -tu us step_1000Benchmark Mode Cnt Score Error Units
MySqlChainIdBenchmark.step_1000 sample 1286774 0.024 ± 0.001 us/op
MySqlChainIdBenchmark.step_1000:step_1000·p0.00 sample ≈ 0 us/op
MySqlChainIdBenchmark.step_1000:step_1000·p0.50 sample 0.041 us/op
MySqlChainIdBenchmark.step_1000:step_1000·p0.90 sample 0.042 us/op
MySqlChainIdBenchmark.step_1000:step_1000·p0.95 sample 0.042 us/op
MySqlChainIdBenchmark.step_1000:step_1000·p0.99 sample 0.042 us/op
MySqlChainIdBenchmark.step_1000:step_1000·p0.999 sample 0.083 us/op
MySqlChainIdBenchmark.step_1000:step_1000·p0.9999 sample 0.208 us/op
MySqlChainIdBenchmark.step_1000:step_1000·p1.00 sample 342.528 us/op