B00tLoad/SnowflakeService
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

1.0.0-rc1

Browse Source 
- README.md: Updated documentation to reflect project scope and functionality
- pom.xml: logback bump, rem okhttp3, add javalin, brotli4j, minor beautification, version number
- logback-{docker, bare}.xml: Added additional rolling files for debug and trace
- SnowflakeService.java: Added documentation, removed orchestration, added webserver
- SnowflakeIDGenerator.java: added documentation, removed unused methods
- ConfigurationValues.java: added documentation
This commit is contained in:
Alix
2024-05-20 19:11:02 +02:00
parent 6c0882eb1a
commit 97ee513e21
7 changed files with 174 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files

81
README.md
View File

@@ -1,7 +1,7 @@
# SnowflakeService
A tool/microservice to centrally generate snowflake IDs. Can be run distributed.
A tool/microservice to centrally generate snowflake IDs.
![GitHub License](https://img.shields.io/github/license/B00tLoad/SnowflakeService)
![GitHub Issues or Pull Requests](https://img.shields.io/github/issues/B00tLoad/SnowflakeService)
@@ -12,7 +12,8 @@ This utility uses:
- The format and name of [Twitter's Snowflake IDs](https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake)
- [Javalin](https://github.com/javalin/javalin)
- [QOS.CH's Logback](https://github.com/qos-ch/logback)
- [Square's OkHttp3](https://github.com/square/okhttp)
- [Google's gson](https://github.com/google/gson)
- [hyperxpro's Brotli4j](https://github.com/hyperxpro/Brotli4j)
- [B00tLoad_'s ConfigurationUtilities](https://github.com/B00tLoad/Configurationutilities)
## Installation
@@ -20,17 +21,17 @@ This utility uses:
### Docker (Recommended)
```bash
docker pull bootmediaalix/snowflake-service
docker run bootmediaalix/snowflake-service -e %{set required .env, see below} -p 95674:95674 -v /data/b00tload-services/snowflake:%desired path on host%
docker pull bootmediaalix/snowflakeservice
docker run bootmediaalix/snowflakeservice -e %{set required .env, see below} -p 95674:95674 -v /data/b00tload-services/snowflake:%desired path on host%
```
### Containerless
A containerless installation is possible, although not supported. For development convenience the application base directory is located in `~/.b00tload-services/snowflake` instead of `/data/b00tload-services/snowflake`.
If you want to work containerless you are on your own.
## Environment Variables
To run this project, you will need to add the following environment variables to your .env file
To run this project, you may add the following environment variables to your .env file
`EPOCH` - the starting time of the snowflake
`EPOCH` - the starting time of the snowflake (defaults to 01.01.2024 12:00 AM)
`MACHINE_ID_BITS` - the amount of bits used for the machine ID
@@ -38,20 +39,6 @@ To run this project, you will need to add the following environment variables to
`MACHINE_ID` - the ID of the generator
**_or_**
`ORCHESTRATOR_IP` - an IP where a snowflake orchestrator is available to fetch all above values
**_or_**
`EPOCH` - the starting time of the snowflake
`MACHINE_ID_BITS` - the amount of bits used for the machine ID
`SEQUENCE_BITS` - the amount of bits used for the sequence counter
with neither `MACHINE_ID` nor `ORCHESTRATOR_IP` set, the Service will start as an orchestrator itself with MACHINE_ID = 0.
## API Reference
@@ -59,64 +46,16 @@ with neither `MACHINE_ID` nor `ORCHESTRATOR_IP` set, the Service will start as a
#### Get an ID
```http
GET /flake
GET /generate
```
Response example:
```json
{
"snowflake": ""
"id": "50990430426234880"
}
```
#### Register worker (internal)
```http request
POST /orchestra/register
```
| Header | Description |
|:-------|:----------------------------------------------------------------------------------|
| `name` | **Required**. A name to recognize the worker by (e.g. hostname+pid+random number) |
Response example:
Http Status: 200
```json
{
"machineBits": "",
"workerid": "",
"sequenceBits": "",
"epoch": ""
}
```
Response on error:
If the version of the client and the orchestrator do not match the orchestrator will respond with a HTTP code 426. The required version will be
| Value name | Value description |
|:-------------------------|:--------------------------|
| Http Status | 426 - Upgrade required |
| `Upgrade` Header content | required software version |
| Value name | Value description |
|:-------------------------|:--------------------------|
| Http Status | 409 - Conflict |
| `Upgrade` Header content | required software version |
#### Worker heartbeat (internal)
```http request
GET /orchestra/heartbeat
```
| Parameter | Description |
|:------------|:------------------------------------------|
| `name` | the ID used in registration |
| `workerID` | the workerID assigned by the orchestrator |
HTTP status code responses:
| Status | Description |
|:-------|:----------------------------|
| `200` | Ok |
| `409` | Conflict, please reregister |
## Maintainer
- [@B00tLoad_](https://www.github.com/B00tLoad)
@@ -129,5 +68,5 @@ HTTP status code responses:
## Support
For support, open a ticket or email me at alix (at) ja-lol-ey (dot) de.
For support, [open a ticket](https://github.com/B00tLoad/SnowflakeService/issues) or email me at alix (at) ja-lol-ey (dot) de.

26
pom.xml
View File

@@ -6,10 +6,10 @@
<groupId>space.b00tload.services</groupId>
<artifactId>SnowflakeService</artifactId>
<version>1.0-SNAPSHOT</version>
<version>1.0.0-rc1</version>
<name>SnowflakeService</name>
<description>A tool/microservice to centrally generate snowflake IDs. Can be run distributed.</description>
<description>A tool/microservice to centrally generate snowflake IDs.</description>
<inceptionYear>2024</inceptionYear>
<url>https://github.com/B00tLoad/SnowflakeService</url>
@@ -65,18 +65,28 @@
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.5.5</version>
<version>1.5.6</version>
</dependency>
<dependency>
<groupId>org.jetbrains</groupId>
<artifactId>annotations</artifactId>
<version>24.0.1</version>
<version>24.1.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>4.12.0</version>
<groupId>io.javalin</groupId>
<artifactId>javalin</artifactId>
<version>6.1.3</version>
</dependency>
<dependency>
<groupId>com.aayushatharva.brotli4j</groupId>
<artifactId>brotli4j</artifactId>
<version>1.16.0</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.10.1</version>
</dependency>
</dependencies>
@@ -148,7 +158,7 @@
</execution>
</executions>
<configuration>
<finalName>${artifactId}</finalName>
<finalName>${project.artifactId}</finalName>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>

21
src/main/java/space/b00tload/services/snowflake/ConfigurationValues.java
View File

@@ -2,12 +2,29 @@ package space.b00tload.services.snowflake;
import space.b00tload.utils.configuration.ConfigValues;
/**
* All values to be configures
* @author Alix von Schirp
* @since 1.0.0
* @version 1.0.0
*/
public enum ConfigurationValues implements ConfigValues {
/**
* Bits reserved for machine id
*/
MACHINE_ID_BITS("machinebits", "M", "MACHINE_ID_BITS", "bitcount.machine", "10"),
/**
* Bits reserved for sequence
*/
SEQUENCE_BITS("sequencebits", "S", "SEQUENCE_BITS", "bitcount.sequence", "12"),
/**
* Selected epoch
*/
EPOCH("epoch", "E", "EPOCH", "epoch", "1704067200000"),
MACHINE_ID("machineid", "I", "MACHINE_ID", "machineid", "-1"),
ORCHESTRATOR_IP("orchestrator", "O", "ORCHESTRATOR_IP", "orchestrator.ip", "http://disabled"),
/**
* this machines id
*/
MACHINE_ID("machineid", "I", "MACHINE_ID", "machineid", "0"),
;
private final String cliFlag;

72
src/main/java/space/b00tload/services/snowflake/SnowflakeIDGenerator.java
View File

@@ -20,6 +20,9 @@ import static java.lang.System.currentTimeMillis;
public class SnowflakeIDGenerator {
//Singleton instance
/**
* Singleton instance
*/
private static SnowflakeIDGenerator INSTANCE;
// Constants
@@ -84,10 +87,10 @@ public class SnowflakeIDGenerator {
/**
* Used for initiating constants and state and performing bound checks on {@code EPOCH}, {@code MAX_TIMESTAMP} and {@code machineId}.
*
* @param machineId The machineID used for generation // expected to be null if not initialized by orchestrator, loaded from config
* @param epoch The snowflake epoch used for generation // expected to be null if not initialized by orchestrator, loaded from config
* @param machineIdBits The amount of bits used for the machineID // expected to be null if not initialized by orchestrator, loaded from config
* @param sequenceBits The amount of bits used for sequence counter // expected to be null if not initialized by orchestrator, loaded from config
* @param machineId The machineID used for generation // expected to be null, loaded from config
* @param epoch The snowflake epoch used for generation // expected to be null, loaded from config
* @param machineIdBits The amount of bits used for the machineID // expected to be null, loaded from config
* @param sequenceBits The amount of bits used for sequence counter // expected to be null, loaded from config
* @throws IllegalArgumentException if any bound checks fail.
*/
private SnowflakeIDGenerator(@Nullable Long machineId, @Nullable Long epoch, @Nullable Long machineIdBits, @Nullable Long sequenceBits) throws IllegalArgumentException{
@@ -129,6 +132,7 @@ public class SnowflakeIDGenerator {
* Generates a new snowflake ID.<br>
* It uses a long (64-bit signed int) made up of (in default configuration):
* <table>
* <caption>Bit layout</caption>
* <tr>
* <th>bit(s)</th>
* <th>content</th>
@@ -152,7 +156,7 @@ public class SnowflakeIDGenerator {
* </table>
* Generation will be halted
* <ul>
* <li>if the clock moves backwards (e.g. fast clock after ntp synchronization) until {@code last generated milli}<{@code current milli}</li>
* <li>if the clock moves backwards (e.g. fast clock after ntp synchronization) until {@code last generated milli}{@literal <}{@code current milli}</li>
* <li>if {@code count(generated IDs this millisecond)} is greater than {@code MAX_SEQUENCE} until next millisecond</li>
* </ul>
* @return the generated ID
@@ -186,6 +190,13 @@ public class SnowflakeIDGenerator {
// Wait for the next timestamp
/**
* Blocks thread until next timestemp (=next milli)
*
* @param currentTimestamp the current milli
* @return the awaited milli
*/
private long waitForNextTimestamp(long currentTimestamp) {
long nextTimestamp = currentTimeMillis();
while (nextTimestamp <= currentTimestamp) {
@@ -194,58 +205,19 @@ public class SnowflakeIDGenerator {
return nextTimestamp;
}
public static void init(long machineId, long epoch, long machineIdBits, long sequenceBits){
INSTANCE = new SnowflakeIDGenerator(machineId, epoch, machineIdBits, sequenceBits);
}
public static void init(){
INSTANCE = new SnowflakeIDGenerator(null, null, null, null);
}
/**
* Getter for the singleton instance. Inits singleton if needed.
* @return the singleton instance.
*/
public static SnowflakeIDGenerator getInstance() {
if(Objects.isNull(INSTANCE)) throw new UnsupportedOperationException("SnowflakeIDGenerator is not initialized.");
// if(Objects.isNull(INSTANCE)) throw new UnsupportedOperationException("SnowflakeIDGenerator is not initialized.");
if(Objects.isNull(INSTANCE)) init();
return INSTANCE;
}
public long getEPOCH() {
return EPOCH;
}
public long getMACHINE_ID_BITS() {
return MACHINE_ID_BITS;
}
public long getSEQUENCE_BITS() {
return SEQUENCE_BITS;
}
public long getMAX_MACHINE_ID() {
return MAX_MACHINE_ID;
}
public long getMAX_SEQUENCE() {
return MAX_SEQUENCE;
}
public long getMACHINE_ID_SHIFT() {
return MACHINE_ID_SHIFT;
}
public long getTIMESTAMP_SHIFT() {
return TIMESTAMP_SHIFT;
}
public long getMachineId() {
return machineId;
}
public long getTIMESTAMP_BITS() {
return TIMESTAMP_BITS;
}
public long getMAX_TIMESTAMP() {
return MAX_TIMESTAMP;
}
}

87
src/main/java/space/b00tload/services/snowflake/SnowflakeService.java
View File

@@ -4,31 +4,45 @@ import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.joran.JoranConfigurator;
import ch.qos.logback.core.joran.spi.JoranException;
import ch.qos.logback.core.util.StatusPrinter2;
import okhttp3.*;
import com.google.gson.JsonObject;
import io.javalin.Javalin;
import io.javalin.http.ContentType;
import io.javalin.http.HandlerType;
import io.javalin.router.Endpoint;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import space.b00tload.utils.configuration.Configuration;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.List;
import java.util.Objects;
import java.util.Random;
/**
* Main class orchestrating between generator, webserver and config
*
* @author Alix von Schirp
* @version 1.0.0
* @since 1.0.0
*/
public class SnowflakeService {
/**
* This software's version
*/
private static String SOFTWARE_VERSION;
/**
* Base dir for config and tmp data
*/
private static String APPLICATION_BASE;
private static String USER_AGENT;
private static Logger LOGGER;
private static String INSTANCE_NAME;
/**
* Initializes tool, generator and webserver.
* @param args cli args
*/
public static void main(String[] args) {
//Set up constants
SOFTWARE_VERSION = Objects.requireNonNullElse(SnowflakeService.class.getPackage().getImplementationVersion(), "0.0.1-indev");
USER_AGENT = "SnowflakeService " + SOFTWARE_VERSION + "(" + System.getProperty("os.name") + "; " + System.getProperty("os.arch") + ") Java/" + System.getProperty("java.version");
APPLICATION_BASE = List.of(args).contains("--docker") ? Paths.get("data", "b00tload-tools", "snowflake").toString() : Paths.get(System.getProperty("user.home"), ".b00tload-tools", "snowflake").toString();
INSTANCE_NAME = getHostname() + "_" + getPid() + "_" + new Random().nextInt(9999);
//Set up logger
LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
@@ -45,52 +59,25 @@ public class SnowflakeService {
(new StatusPrinter2()).printInCaseOfErrorsOrWarnings(loggerContext);
//Set up logger
LOGGER = LoggerFactory.getLogger(SnowflakeService.class);
Logger LOGGER = LoggerFactory.getLogger(SnowflakeService.class);
//Init config
Configuration.init(args, SOFTWARE_VERSION, APPLICATION_BASE, ConfigurationValues.values());
long machineID;
long sequenceBits;
long machineBits;
long epoch;
if(!Configuration.getInstance().get(ConfigurationValues.ORCHESTRATOR_IP).equals(ConfigurationValues.ORCHESTRATOR_IP.getDefaultValue())){
OkHttpClient httpClient = new OkHttpClient();
try {
try (Response r = httpClient.newCall(
new Request.Builder()
.url(Configuration.getInstance().get(ConfigurationValues.ORCHESTRATOR_IP))
.post(
new FormBody.Builder().addEncoded("name", INSTANCE_NAME).build()
)
.build()
).execute()){
if(r.code() == 200){
//Init webserver
Javalin endpointServer = Javalin.create(config -> {
config.http.brotliAndGzipCompression();
config.http.prefer405over404 = true;
config.requestLogger.http((ctx, executionTimeMs) -> {
LoggerFactory.getLogger(config.getClass()).info("{} served in {}ms to {}(UA: \"{}\")", ctx.fullUrl(), executionTimeMs, ctx.req().getRemoteAddr(), ctx.userAgent());
});
});
endpointServer.addEndpoint(new Endpoint(HandlerType.GET, "generate", ctx -> {
JsonObject ret = new JsonObject();
ret.addProperty("id", SnowflakeIDGenerator.getInstance().generateID());
ctx.status(200).result(ret.toString()).contentType(ContentType.APPLICATION_JSON);
}));
endpointServer.start(95674);
}
}
} catch (IOException e) {
throw new RuntimeException(e);
}
} else if((Long.parseLong(Configuration.getInstance().get(ConfigurationValues.MACHINE_ID))) != -1L) {
} else {
}
}
private static String getHostname(){
String hostname = System.getenv("COMPUTERNAME"); // On Windows
if (hostname == null || hostname.isEmpty()) {
hostname = System.getenv("HOSTNAME"); // On Unix/Linux
}
return hostname;
}
private static long getPid(){
return ProcessHandle.current().pid();
}
}

36
src/main/resources/config/logback/logback-bare.xml
View File

@@ -14,6 +14,9 @@
<level>ERROR</level>
<onMatch>DENY</onMatch>
</filter>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
</appender>
<appender name="STDERR" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
@@ -26,16 +29,45 @@
<appender name="RFAOUT" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/snowflake.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_ARCHIVE}/rollingfile.log%d{yyy-MM-dd}.lol.gz</fileNamePattern>
<fileNamePattern>${LOG_ARCHIVE}/snowflake.rolling.%d{yyy-MM-dd}.log.gz</fileNamePattern>
</rollingPolicy>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -%kvp- %msg%n</pattern>
</encoder>
</appender>
<appender name="debugOUT" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/snowflake.debug.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_ARCHIVE}/snowflake.debug.rolling.%d{yyy-MM-dd}.log.gz</fileNamePattern>
</rollingPolicy>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>DEBUG</level>
</filter>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -%kvp- %msg%n</pattern>
</encoder>
</appender>
<appender name="traceOUT" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/snowflake.trace.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_ARCHIVE}/snowflake.trace.rolling.%d{yyy-MM-dd}.log.gz</fileNamePattern>
</rollingPolicy>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>TRACE</level>
</filter>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -%kvp- %msg%n</pattern>
</encoder>
</appender>
<root level="debug">
<root level="trace">
<appender-ref ref="STDOUT" />
<appender-ref ref="STDERR" />
<appender-ref ref="RFAOUT" />
<appender-ref ref="debugOUT" />
<appender-ref ref="traceOUT" />
</root>
</configuration>

36
src/main/resources/config/logback/logback-docker.xml
View File

@@ -14,6 +14,9 @@
<level>ERROR</level>
<onMatch>DENY</onMatch>
</filter>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
</appender>
<appender name="STDERR" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
@@ -26,16 +29,45 @@
<appender name="RFAOUT" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/snowflake.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_ARCHIVE}/rollingfile.log%d{yyy-MM-dd}.lol.gz</fileNamePattern>
<fileNamePattern>${LOG_ARCHIVE}/snowflake.rolling.%d{yyy-MM-dd}.log.gz</fileNamePattern>
</rollingPolicy>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>INFO</level>
</filter>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -%kvp- %msg%n</pattern>
</encoder>
</appender>
<appender name="debugOUT" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/snowflake.debug.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_ARCHIVE}/snowflake.debug.rolling.%d{yyy-MM-dd}.log.gz</fileNamePattern>
</rollingPolicy>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>DEBUG</level>
</filter>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -%kvp- %msg%n</pattern>
</encoder>
</appender>
<appender name="traceOUT" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/snowflake.trace.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_ARCHIVE}/snowflake.trace.rolling.%d{yyy-MM-dd}.log.gz</fileNamePattern>
</rollingPolicy>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>TRACE</level>
</filter>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -%kvp- %msg%n</pattern>
</encoder>
</appender>
<root level="debug">
<root level="trace">
<appender-ref ref="STDOUT" />
<appender-ref ref="STDERR" />
<appender-ref ref="RFAOUT" />
<appender-ref ref="debugOUT" />
<appender-ref ref="traceOUT" />
</root>
</configuration>