Skip to content

Commit

Permalink
lib llvm submodule (ponylang#2748)
Browse files Browse the repository at this point in the history
* Add lib/llvm

This PR It supports both system installed LLVM binaries and vendored LLVM
sources. The default is the system installed LLVM binaries with the
vendored LLVM sources supported if make -f Makefile-lib-llvm is used.

The Makefile is a symlink to Makefile-ponyc so the system is built just
as it currently, no changes.

To build with the vendored LLVM, you may do "make -f Makefile-lib-llvm" or
change the Makefile symlink to point to Makefile-lib-llvm.

When using Makefile-lib-llvm lib/llvm is built first and then
Makefile-ponyc is invoked as Makefile-ponyc assumes llvm-config is
already installed.

The possible targets are those defined by Makefile-pony plus the following:

  clean         Clean only Makefile-ponyc does not touch lib/llvm/Makefile
  cleanall      Clean Makefile-ponyc and the build artifacts of
                lib/llvm/Makefile, does not touch lib/llvm/src which
                contains the source code for llvm
  distclean     Does a cleanall plus deletes lib/llvm/src
  some-tests    Runs ./build/debug/libponyc.tests with whatever is passed
                in the gtest_filter parameter.

These rebuild targets are used for development and will invoke the clean
target on Makefile-ponyc, then incrementally rebulid lib/llvm and then
build ponyc. Depending upon the

  rebuild       Rebuilds only does not run any tests
  rebuild-tests	Rebuilds and then invokes the "test" target on
                Makefile-ponyc
  rebuild-some-tests
                Rebuilds and then recursively invokes the some-tests
                target of Makefile-lib-llvm above. So whatever tests are
                specified in gtest_filter parameter will execute.

The .circleci/config.yml has been modified to tests the vendored LLVM on
various Docker images added in .ci-dockerfiles. Currently Alpine doesn't
work because the Alpine release requires changes to work with the musl
C library. What could happen is that Alpine patches could be applied to
a branch in ponylang/llvm and then be used by ponyc by adding a new
llvm_proj target.

Regarding whether to use submodule or include the source code directly,
I prefer this PR using submodule as opposed to PR ponylang#2663
(wip-lib-llvm) which hosts the source code directly. For the following
reasons:

 - Smaller ponyc repository.
 - All LLVM source code will be in a fork hosted at ponylang/llvm.
 - No changes in workflow is necessary to compile ponyc using there
   installed LLVM binaries.
 - For developers and regular users that wish to test an LLVM version
   not supported by there system this can now be easily done using
   make -f Makefile-lib-llvm llvm_proj=llvm_xxx. If they'd like they
   can create a symlink, ln -sf Makefile-lib-llvm Makefile, so the
   -f Makefile-lib-llvm is not needed.
 - Developers can modify lib/llvm/src and test using llvm_proj=llvm-current
 - Can more easily support multiple versions of LLVM without having the
   pony repository containing the source code LLVM directly.
 - The source code for LLVM is a fork of the LLVM that the resides in
   ponylang and would be managed as separate repo. This has the benefit
   that it will be easy to keep in sync with the upstream project. It will
   be easier to make any customization's such as the need to support
   patches for Alpine. And it will be easier to upstream changes since the
   fork can be kept in sync.
 - LLVM is a huge project compared to ponylang/ponyc project. There is
   almost 10x the lines of code as per cloc so the stats for ponyc will
   be artificially skewed. In addition, the binary size of the ponyc repo
   will significantly increase.

I would say the major downside to git submodule is that newer versions of
git are needed where as in PR ponylang#2663 "any" version of git could be used.
Specifically, in PR ponylang#2663 the git installed by apt-get in
ponylang/ponyc-ci:ubuntu is 1.8.2 and it worked fine, but I had to the
Dockerfile to install a newer version to work with this PR. I chose to
install 2.17.1 as that what was on my Arch Linux system, I suspect it
doesn't have to be that new, but it did work for me so that's what I
used.

# Conflicts:
#	.circleci/config.yml
#	Makefile

* Add lib-llvm to beginning of the vendor lib/llvm tests

As suggested by [Dipin Hora](ponylang#2748 (comment)).
Regarding using lib/llvm for the x86-64 cross compiled targets, I suggest
deciding that in the future so we get this in sooner rather than later.

Also, added "lib-llvm-ubuntu-default" which wasn't being run.

* Tweaks to README.md
  • Loading branch information
winksaville authored and sylvanc committed Oct 16, 2018
1 parent fc878e4 commit d6df15f
Show file tree
Hide file tree
Showing 20 changed files with 1,944 additions and 1,110 deletions.
17 changes: 17 additions & 0 deletions .ci-dockerfiles/alpine/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
FROM alpine

RUN apk add --update \
alpine-sdk \
libressl-dev \
binutils-gold \
pcre2-dev \
libexecinfo-dev \
coreutils \
linux-headers \
cmake \
python

# add user pony in order to not run tests as root
RUN adduser -D -s /bin/sh -h /home/pony -g root pony
USER pony
WORKDIR /home/pony
36 changes: 36 additions & 0 deletions .ci-dockerfiles/alpine/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Build image

```bash
docker build -t ponylang/ponyc-ci:alpine .
```

# Run image to test

Will get you a bash shell in the image to try cloning Pony into where you can test a build to make sure everything will work before pushing:

```bash
docker run --name ponyc-ci-alpine --user pony --rm -i -t ponylang/ponyc-ci:alpine /bin/sh
```

# Run CircleCI jobs locally

Use the [CircleCI CLI](https://circleci.com/docs/2.0/local-cli/) to run the CI jobs using this image
from the ponyc project root:

```bash
circleci build --job alpine-debug
circleci build --job alpine-release
```
Note: when building you might want to set CPUs environment
variable to speed up the build:
```bash
circleci build -e CPUs=10 --job alpine-debug
```

# Push to dockerhub

You'll need credentials for the ponylang dockerhub account. Talk to @jemc or @seantallen for access

```bash
docker push ponylang/ponyc-ci:alpine
```
32 changes: 32 additions & 0 deletions .ci-dockerfiles/centos7/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
FROM centos:7

RUN yum install -y \
yum-plugin-copr \
which \
git \
gcc-c++ \
make \
openssl-devel \
pcre2-devel \
zlib-devel \
ncurses-devel \
libatomic \
wget

# install a newer cmake
RUN wget https://cmake.org/files/v3.11/cmake-3.11.2-Linux-x86_64.sh \
&& sh cmake-3.11.2-Linux-x86_64.sh --prefix=/usr/local --exclude-subdir

# install newer git to support submodules
RUN yum install -y gettext libcurl-devel expat-devel zlib-devel \
&& git clone https://github.com/git/git git-src \
&& cd git-src \
&& git checkout v2.17.1 \
&& make -j$(nproc) prefix=/usr all \
&& make prefix=/usr install \
&& rm -rf git-src

# add user pony in order to not run tests as root
RUN useradd -ms /bin/bash -d /home/pony -g root pony
USER pony
WORKDIR /home/pony
37 changes: 37 additions & 0 deletions .ci-dockerfiles/centos7/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Build image

```bash
docker build -t ponylang/ponyc-ci:centos7 .
```

# Run image to test

Will get you a bash shell in the image to try cloning Pony into where you can test a build to make sure everything will work before pushing:

```bash
docker run --name ponyc-ci-centos7 --user pony --rm -i -t ponylang/ponyc-ci:centos7 bash
```

## Run CircleCI jobs locally

Use the [CircleCI CLI](https://circleci.com/docs/2.0/local-cli/) to run the CI job using this image
from the ponyc project root:

```bash
circleci build --job centos7-debug
circleci build --job centos7-release
circleci build --job centos7-llvm-600-release
```
Note: when building you might want to set CPUs environment
variable to speed up the build:
```bash
circleci build -e CPUs=10 --job centos7-debug
```

# Push to dockerhub

You'll need credentials for the ponylang dockerhub account. Talk to @jemc or @seantallen for access

```bash
docker push ponylang/ponyc-ci:centos7
```
23 changes: 23 additions & 0 deletions .ci-dockerfiles/ubuntu-18.04/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
FROM ubuntu:18.04

RUN apt-get update \
&& apt-get install -y \
apt-transport-https \
g++ \
git \
clang \
llvm \
libncurses5-dev \
libpcre2-dev \
libssl-dev \
cmake \
make \
wget \
xz-utils \
zlib1g-dev \
python

# add user pony in order to not run tests as root
RUN useradd -ms /bin/bash -d /home/pony -g root pony
USER pony
WORKDIR /home/pony
36 changes: 36 additions & 0 deletions .ci-dockerfiles/ubuntu-18.04/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Build image

```bash
docker build -t ponylang/ponyc-ci:ubuntu-18.04 .
```

# Run image to test

Will get you a bash shell in the image to try cloning Pony into where you can test a build to make sure everything will work before pushing:

```bash
docker run --name ponyc-ci-ubuntu-18.04 --user pony --rm -i -t ponylang/ponyc-ci:ubuntu-18.04 bash
```

# Run CircleCI jobs locally

Use the [CircleCI CLI](https://circleci.com/docs/2.0/local-cli/) to run the CI job using this image
from the ponyc project root:

```bash
circleci build --job ubuntu-18.04-debug
circleci build --job ubuntu-18.04-release
```
Note: when building you might want to set CPUs environment
variable to speed up the build:
```bash
circleci build -e CPUs=10 --job ubuntu-ubuntu-18.04-debug
```

# Push to dockerhub

You'll need credentials for the ponylang dockerhub account. Talk to @jemc or @seantallen for access

```bash
docker push ponylang/ponyc-ci:ubuntu-18.04
```
38 changes: 38 additions & 0 deletions .ci-dockerfiles/ubuntu-openssl-110/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
FROM ubuntu:16.04

RUN apt-get update \
&& apt-get install -y \
apt-transport-https \
g++ \
git \
libncurses5-dev \
libpcre2-dev \
cmake \
make \
wget \
xz-utils \
zlib1g-dev \
python

RUN wget https://www.openssl.org/source/openssl-1.1.0.tar.gz \
&& tar xf openssl-1.1.0.tar.gz \
&& cd openssl-1.1.0 \
&& ./config \
&& make \
&& make install \
&& cd .. \
&& rm -rf openssl-1.1.0*

# install newer git to support submodules
RUN apt-get install -y gettext libcurl3-dev libexpat-dev zlib1g-dev \
&& git clone https://github.com/git/git git-src \
&& cd git-src \
&& git checkout v2.17.1 \
&& make -j$(nproc) prefix=/usr all \
&& make prefix=/usr install \
&& rm -rf git-src

# add user pony in order to not run tests as root
RUN useradd -ms /bin/bash -d /home/pony -g root pony
USER pony
WORKDIR /home/pony
36 changes: 36 additions & 0 deletions .ci-dockerfiles/ubuntu-openssl-110/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Build image

```bash
docker build -t ponylang/ponyc-ci:ubuntu-openssl-110 .
```

# Run image to test

Will get you a bash shell in the image to try cloning Pony into where you can test a build to make sure everything will work before pushing:

```bash
docker run --name ponyc-ci-ubuntu-openssl-110 --user pony --rm -i -t ponylang/ponyc-ci:ubuntu-openssl-110 bash
```

# Run CircleCI jobs locally

Use the [CircleCI CLI](https://circleci.com/docs/2.0/local-cli/) to run the CI job using this image
from the ponyc project root:

```bash
circleci build --job ubuntu-openssl-110-debug
circleci build --job ubuntu-openssl-110-release
```
Note: when building you might want to set CPUs environment
variable to speed up the build:
```bash
circleci build -e CPUs=10 --job ubuntu-openssl-110-debug
```

# Push to dockerhub

You'll need credentials for the ponylang dockerhub account. Talk to @jemc or @seantallen for access

```bash
docker push ponylang/ponyc-ci:ubuntu-openssl-110
```
45 changes: 45 additions & 0 deletions .ci-dockerfiles/ubuntu/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
FROM ubuntu:14.04

RUN apt-get update \
&& apt-get install -y software-properties-common \
&& add-apt-repository -y ppa:ubuntu-toolchain-r/test \
&& apt-get update \
&& apt-get install -y \
apt-transport-https \
build-essential \
g++-6 \
git \
libncurses5-dev \
libssl-dev \
make \
wget \
xz-utils \
zlib1g-dev \
python

# install pcre2
RUN wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre2-10.21.tar.bz2 \
&& tar xvf pcre2-10.21.tar.bz2 \
&& cd pcre2-10.21 \
&& ./configure --prefix=/usr \
&& make install \
&& cd .. \
&& rm -rf pcre2-10.21*

# install a newer cmake
RUN wget https://cmake.org/files/v3.11/cmake-3.11.2-Linux-x86_64.sh \
&& sh cmake-3.11.2-Linux-x86_64.sh --prefix=/usr/local --exclude-subdir

# install newer git to support submodules
RUN apt-get install -y gettext libcurl3-dev libexpat-dev zlib1g-dev \
&& git clone https://github.com/git/git git-src \
&& cd git-src \
&& git checkout v2.17.1 \
&& make -j$(nproc) prefix=/usr all \
&& make prefix=/usr install \
&& rm -rf git-src

# add user pony in order to not run tests as root
RUN useradd -ms /bin/bash -d /home/pony -g root pony
USER pony
WORKDIR /home/pony
38 changes: 38 additions & 0 deletions .ci-dockerfiles/ubuntu/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Build image

```bash
docker build -t ponylang/ponyc-ci:ubuntu .
```

# Run image to test

Will get you a bash shell in the image to try cloning Pony into where you can test a build to make sure everything will work before pushing:

```bash
docker run --name ponyc-ci-ubuntu --user pony --rm -i -t ponylang/ponyc-ci:ubuntu bash
```

# Run CircleCI jobs locally

Use the [CircleCI CLI](https://circleci.com/docs/2.0/local-cli/) to run the CI job using this image
from the ponyc project root:

```bash
circleci build --job ubuntu-debug
circleci build --job ubuntu-release
circleci build --job ubuntu-llvm-600-debug
circleci build --job ubuntu-llvm-600-release
```
Note: when building you might want to set CPUs environment
variable to speed up the build:
```bash
circleci build -e CPUs=10 --job ubuntu-debug
```

# Push to dockerhub

You'll need credentials for the ponylang dockerhub account. Talk to @jemc or @seantallen for access

```bash
docker push ponylang/ponyc-ci:ubuntu
```
Loading

0 comments on commit d6df15f

Please sign in to comment.