Skip to content

Commit

Permalink
automatically figure out ARM-related variables (airbytehq#11450)
Browse files Browse the repository at this point in the history
* automatically figure out ARM-related variables

* clean up

* explicitly use amd64-prefixed versions of alpine and postgres
  • Loading branch information
jrhizor authored Mar 31, 2022
1 parent 03ed894 commit 526c5a0
Show file tree
Hide file tree
Showing 6 changed files with 18 additions and 119 deletions.
12 changes: 8 additions & 4 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -132,11 +132,15 @@ check.dependsOn 'spotlessApply'
def Task getDockerBuildTask(String artifactName, String projectDir) {
return task ("buildDockerImage-$artifactName"(type: DockerBuildImage) {
def buildTag = System.getenv('VERSION') ?: 'dev'
def buildPlatform = System.getenv('DOCKER_BUILD_PLATFORM') ?: 'linux/amd64'
def alpineImage = System.getenv('ALPINE_IMAGE') ?: 'alpine:3.4'
def postgresImage = System.getenv('POSTGRES_IMAGE') ?: 'postgres:13-alpine'
def jdkVersion = System.getenv('JDK_VERSION') ?: '17.0.1'
def buildArch = System.getenv('DOCKER_BUILD_ARCH') ?: 'amd64'

def arch = System.getProperty("os.arch").toLowerCase()
def isArm64 = arch == "aarch64" || arch == "arm64"

def buildPlatform = System.getenv('DOCKER_BUILD_PLATFORM') ?: isArm64 ? 'linux/arm64' : 'linux/amd64'
def alpineImage = System.getenv('ALPINE_IMAGE') ?: isArm64 ? 'arm64v8/alpine:3.14' : 'amd64/alpine:3.14'
def postgresImage = System.getenv('POSTGRES_IMAGE') ?: isArm64 ? 'arm64v8/postgres:13-alpine' : 'amd64/postgres:13-alpine'
def buildArch = System.getenv('DOCKER_BUILD_ARCH') ?: isArm64 ? 'arm64' : 'amd64'

inputDir = file("$projectDir/build/docker")
platform = buildPlatform
Expand Down
6 changes: 5 additions & 1 deletion buildSrc/src/main/groovy/airbyte-docker.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,12 @@ abstract class AirbyteDockerTask extends DefaultTask {
if (project.file(fileName).exists()) {
def tag = DockerHelpers.getDevTaggedImage(projectDir, dockerfileName)

def arch = System.getProperty("os.arch").toLowerCase()
def isArm64 = arch == "aarch64" || arch == "arm64"
def buildPlatform = System.getenv('DOCKER_BUILD_PLATFORM') ?: isArm64 ? 'linux/arm64' : 'amd64'

project.exec {
commandLine scriptPath, rootDir.absolutePath, projectDir.absolutePath, dockerfileName, tag, idFileOutput.absolutePath, followSymlinks
commandLine scriptPath, rootDir.absolutePath, projectDir.absolutePath, dockerfileName, tag, idFileOutput.absolutePath, followSymlinks, buildPlatform
}
}
}
Expand Down
74 changes: 0 additions & 74 deletions docker-compose.build-m1.yaml

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -51,15 +51,7 @@ You can build the destination by running:
./gradlew :airbyte-integrations:connectors:destination-<name>:build
```

On Mac M1\(Apple Silicon\) machines\(until openjdk images natively support ARM64 images\) set the platform variable as shown below and build

```bash
export DOCKER_BUILD_PLATFORM=linux/amd64
# Must be run from the Airbyte project root
./gradlew :airbyte-integrations:connectors:destination-<name>:build
```

this compiles the java code for your destination and builds a Docker image with the connector. At this point, we haven't implemented anything of value yet, but once we do, you'll use this command to compile your code and Docker image.
This compiles the Java code for your destination and builds a Docker image with the connector. At this point, we haven't implemented anything of value yet, but once we do, you'll use this command to compile your code and Docker image.

{% hint style="info" %}
Airbyte uses Gradle to manage Java dependencies. To add dependencies for your connector, manage them in the `build.gradle` file inside your connector's directory.
Expand Down
13 changes: 1 addition & 12 deletions docs/contributing-to-airbyte/developing-locally.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,7 @@ To start contributing:
## Build with `gradle`

{% hint style="info" %}
If you're using Mac M1 \(Apple Silicon\) machines, you can run Airbyte locally only in `dev` mode and you need to set these environment variables in order to build Airbyte:

```bash
export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export ALPINE_IMAGE=arm64v8/alpine:3.14
export POSTGRES_IMAGE=arm64v8/postgres:13-alpine
export JDK_VERSION=17
```

There are some known issues (Temporal failing during runs, and some connectors not working). See the [GitHub issue](https://github.com/airbytehq/airbyte/issues/2017) for more information.

If you're using Mac M1 \(Apple Silicon\) machines, you may run into some problems (Temporal failing during runs, and some connectors not working). See the [GitHub issue](https://github.com/airbytehq/airbyte/issues/2017) for more information.
{% endhint %}

To compile and build just the platform \(not all the connectors\):
Expand Down
22 changes: 3 additions & 19 deletions docs/deploying-airbyte/local-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,20 +16,10 @@ docker-compose up
```

{% hint style="info" %}
If you're using Mac M1 \(Apple Silicon\) machines, you can run Airbyte locally only in `dev` mode and you need to set these environment variables in order to build Airbyte:

```bash
export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export ALPINE_IMAGE=arm64v8/alpine:3.14
export POSTGRES_IMAGE=arm64v8/postgres:13-alpine
export JDK_VERSION=17
```

after setting up the environment variables you can start Airbyte locally by running:
```bash
VERSION=dev docker-compose up
```
If you're using Mac M1 \(Apple Silicon\) machines, you must [build and run Airbyte locally in dev mode](../contributing-to-airbyte/developing-locally.md).
Some users using Macs with an M1 chip are facing problems running Airbyte even with a locally built version of the platform.
You can subscribe to [Issue \#2017](https://github.com/airbytehq/airbyte/issues/2017) and get updates on M1-related issues.

{% endhint %}

Expand Down Expand Up @@ -67,11 +57,5 @@ docker-compose up

## Troubleshooting

**I have a Mac with the M1 chip. Is it possible to run Airbyte?**

Yes, you can run Airbyte in dev mode on Macs with M1 chip (see the note above). Some users using Macs with an M1 chip are facing some problems running Airbyte. The problem is related with the chip and Docker. [Issue \#2017](https://github.com/airbytehq/airbyte/issues/2017) was created to follow up the problem, you can subscribe to it and get updates about the resolution. If you can successfully run Airbyte using a MacBook with the M1 chip, let us know so that we can share the process with the community!

**Other issues**

If you encounter any issues, just connect to our [Slack](https://slack.airbyte.io). Our community will help! We also have a [troubleshooting](../troubleshooting/on-deploying.md) section in our docs for common problems.

0 comments on commit 526c5a0

Please sign in to comment.