forked from airbytehq/airbyte
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Split airbyte-db and move db dev commands to gradle (airbytehq#5616)
# Summary - A follow-up PR for airbytehq#5543. - This PR separates the `airbyte-db` project to two modules: - `lib` is the original `airbyte-db`. - `jooq` is for jOOQ code generation. - This is necessary because the jOOQ generator requires a custom database implementation that can run Flyway migration. So the code generator logic needs to depend on the compilation of the original `airbyte-db` project. # Commits * Separate db to lib and jooq modules * Update dependencies * Add jobs db migrator test * Fix compose build * Add migration dev center * Add schema dump task * Update airbyte-db/lib/README.md * Co-authored-by: Davin Chia <[email protected]> * Update readme * Remove bom dependency * Update readme * Use jooq code in db config persistence * Remove AirbyteConfigsTable Co-authored-by: Davin Chia <[email protected]>
- Loading branch information
Showing
122 changed files
with
629 additions
and
236 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| # jOOQ Code Generation | ||
|
|
||
| ## How to Use | ||
| This module generates jOOQ code for the configs and jobs database. To use the generated code, add the following dependency: | ||
|
|
||
| ```gradle | ||
| dependencies { | ||
| implementation project(':airbyte-db:jooq') | ||
| } | ||
| ``` | ||
|
|
||
| The generated code exists in the package `io.airbyte.db.instance.<db-name>.jooq` under the directory `build/generated/<db-name>Database/src/main/java`. | ||
|
|
||
| ## Code Generation | ||
| Gradle plugin `nu.studer.jooq` is used for jOOQ code generation. See [here](https://github.com/etiennestuder/gradle-jooq-plugin) for details. | ||
|
|
||
| It is necessary to separate this module from the `lib` module, because we use a custom database (`FlywayMigrationDatabase`) that runs Flyway migration first for the code generator. This implementation needs to be compiled before it can be used. | ||
|
|
||
| The code will be automatically generated when this module is compiled. To manually update the generated code, run the `compileJava` task: | ||
|
|
||
| ```sh | ||
| SUB_BUILD=PLATFORM ./gradlew :airbyte-db:jooq:compileJava | ||
| ``` | ||
|
|
||
| Or run the following tasks for individual database: | ||
|
|
||
| ```sh | ||
| # for configs database | ||
| SUB_BUILD=PLATFORM ./gradlew :airbyte-db:jooq:generateConfigsDatabaseJooq | ||
|
|
||
| # for jobs database | ||
| SUB_BUILD=PLATFORM ./gradlew :airbyte-db:jooq:generateJobsDatabaseJooq | ||
| ``` | ||
|
|
||
| ## How to Setup Code Generation for New Database | ||
| - In `build.gradle`, do the following. | ||
| - Add a new jOOQ configuration under `jooq.configuration`. | ||
| - This step will automatically create a `generate<db-name>DatabaseJooq` task. | ||
| - Register the output of the code generation task in the main sourceSet. | ||
| - Setup caching for the code generation task. | ||
|
|
||
| Template: | ||
|
|
||
| ```build.gradle | ||
| // add jooq configuration | ||
| jooq { | ||
| configurations { | ||
| <db-name>Database { | ||
| generateSchemaSourceOnCompilation = true | ||
| generationTool { | ||
| generator { | ||
| name = 'org.jooq.codegen.DefaultGenerator' | ||
| database { | ||
| name = 'io.airbyte.db.instance.configs.ConfigsFlywayMigrationDatabase' | ||
| inputSchema = 'public' | ||
| excludes = 'airbyte_configs_migrations' | ||
| } | ||
| target { | ||
| packageName = 'io.airbyte.db.instance.configs.jooq' | ||
| directory = 'build/generated/configsDatabase/src/main/java' | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| // register output as source set | ||
| sourceSets.main.java.srcDirs ( | ||
| tasks.named('generate<db-name>DatabaseJooq').flatMap { it.outputDir } | ||
| ) | ||
| sourceSets { | ||
| main { | ||
| java { | ||
| srcDirs "$buildDir/generated/<db-name>Database/src/main/java" | ||
| } | ||
| } | ||
| } | ||
| // setup caching | ||
| tasks.named('generate<db-name>DatabaseJooq').configure { | ||
| allInputsDeclared = true | ||
| outputs.cacheIf { true } | ||
| } | ||
| ``` |
Oops, something went wrong.