CMake support (#37)

* Add Dockerfile Useful for knowing what needs installed to build * Initial CMake build support Used cmake-init v0.24.1 * Had to remove '-Werror=float-equal' due to error in sleigh source code * Haven't yet integrated/tested anything but the build of the shared library Lots of todos including removal of third-party dependencies, bindings, testing, docs, spelling, linting, etc. * Add back '-Wfloat-equal' warning but don't error This warning exists in the codebase, so setting it to error will fail. Delaying fixing this until later. * Remove sleigh submodule in favor of just find_package Was nice for initial testing, but we shouldn't rely on a dependency submodule * Build and add tests Doesn't work yet because the compiled sla file paths are hard-coded * Working tests and sleighspec compile Fixed spot where I forgot to set compile definition with LIEF loader Still need to fix a few things to make this more robust. Using sleigh branch https://github.com/lifting-bits/sleigh/tree/cmake-module-sleigh-compile * Add back setting CMAKE_INSTALL_INCLUDEDIR to include/maat It duplicates maat/maat directories in the installed include directory, but this is to prevent including headers from other projects when using maat. This doesn't affect anyone using CMake, they can still #include <maat/header.h> but anyone not using CMake will either need to set -I<prefix>/include/maat when compiling or use #include <maat/maat/header.h> in their code * Python bindings Need to set environment variable MAAT_INSTALL_DIR to an already installed maat directory * Remove git submodule file * Add missing Python source file. Sort source listing * Fix include installation path * Fixup Dockerfile for CMake The Dockerfile is more like living documentation than a deployment method * Fix CMake version in presets * Let CMake generate data paths used in code Other small fixes and cleanups * Revert endline change to keep diff small * Remove CMake linting Will probably want to add this back in the future * Add PYTHONPATH to Dockerfile Now you can import maat from anywhere * More consistent whitespace in CMake files * Some documentation building fixes and edits I don't know Doxygen very well, but this seems to produce something reasonable * Remove unsupported sleigh architecture files * Revert removal of sleigh code Needs a workaround in CMake to also support usage of lifting-bits sleigh project. This makes the diff smaller * Remove CMake documentation building Might want to add this back later * Create find_package test * Fix errors with CMake when using after installed * Remove clang tidy file We can add it back later in another commit/PR * Fixup removed source code after merge * Use a macro for compiling sleigh targets * CMake options to use build with Z3 and LIEF backends * Remove CONTRIBUTING file in favor of HACKING file * Fix Z3 dependency handling Many distributions don't include Z3's CMake config file, so we need to write our own, unfortunately * Optionally use submodule'd sleigh Also try to fix CI and fix some other things * Debug sanitize fail * Update sleigh submodule * Use released version of sleigh * Sanitizer fixes (#41) * Some UB fixes * Run the CI on pull requests for all base branches * Fix leak warning for NewDefaultExprSimplifier * Fix UB on unaligned memory accesses * More UB fixes * Remove vestigial unit-tests for reading signed values from MemConcreteBuffer * Add missing virtual destructors * Use newer Z3 to fix tests with sanitizers Not entirely sure why the tests failed with the older Z3, but using a newer version seems to work with sanitizers Co-authored-by: Eric Kilmer <[email protected]> * Update cmake/install-rules.cmake * Remove C++ visibility settings This should be added back when we get a chance to annotate a real public API that can control visibility settings. Controlling visibility is best practice when creating shared libraries * Remove static analyzers from CI They're slow and we have many other compiler warnings to care about right now. We are leaving them in the presets file so that users may include them in their own CMakeUserPresets.json file * Move sleigh_interface.hpp to public include directory * Enable coverage collection but ony print summary We can decide later where we want to upload the report * Fixup some comments * Fixes for moved sleigh_interface.hpp * Disable coverage CI until open source Takes too long to run and also fails right now... * Update building and hacking documentation * Add more docs * Mention submodule for using vendored sleigh * Remove Makefile No longer supported after this branch is merged * Update BUILDING.md to mention Dockerfile Co-authored-by: Boyan MILANOV <[email protected]> * Instructions for installing Python module * Fix python build/installation docs * Download prebuilt dependencies except sleigh We support sleigh submodule, so let's test it here * Update building instructions for python bindings * Update command to get site-packages location in virtualenv * Use getsitepackages() * Fix site-packages location for virtualenvs * Fix getting site-packages location in bindings.cmake * Update documentation for Python module * Update BUILDING.md Co-authored-by: Alex Cameron <[email protected]> Co-authored-by: Boyan MILANOV <[email protected]>
trailofbits · Feb 14, 2022 · f01e678 · f01e678
1 parent a7cf644
commit f01e678
Show file tree

Hide file tree

Showing 44 changed files with 1,511 additions and 360 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,158 @@
+name: Continuous Integration
+
+on:
+  push:
+    branches:
+    - master
+
+  pull_request:
+    branches:
+    - '**'
+
+jobs:
+  coverage:
+    runs-on: ubuntu-latest
+
+    if: github.repository_owner == 'trailofbits'
+      && false
+
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Install Dependencies
+      run: |
+        sudo apt-get install libgmp-dev python3-dev libz3-dev
+        wget -O - -c https://github.com/lief-project/LIEF/releases/download/0.11.5/LIEF-0.11.5-Linux-x86_64.tar.gz | sudo tar xz -C /usr/local --strip-components=1
+        wget -O - -c https://github.com/lifting-bits/sleigh/releases/download/v10.1.2/Linux-sleigh-10.1.2-1.x86_64.tar.gz | sudo tar xz -C /usr/local --strip-components=1
+
+    - name: Install LCov
+      run: sudo apt-get update -q
+        && sudo apt-get install lcov -q -y
+
+    - name: Configure
+      run: cmake --preset=ci-coverage
+
+    - name: Build
+      run: cmake --build build/coverage -j 2
+
+    - name: Test
+      working-directory: build/coverage
+      run: ctest --output-on-failure -j 2
+
+    - name: Process coverage info
+      run: cmake --build build/coverage -t coverage
+
+    - name: Print coverage info
+      run: lcov --summary build/coverage/coverage.info
+
+    #- name: Submit to codecov.io
+    #  uses: codecov/codecov-action@v1
+    #  with:
+    #    file: build/coverage/coverage.info
+
+  sanitize:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Install Dependencies
+      run: |
+        sudo apt-get install libgmp-dev python3-dev
+        # LIEF
+        wget -O- https://github.com/lief-project/LIEF/releases/download/0.11.5/LIEF-0.11.5-Linux-x86_64.tar.gz | sudo tar xz -C /usr/local --strip-components=1
+        # Sleigh
+        wget -O- https://github.com/lifting-bits/sleigh/releases/download/v10.1.2/Linux-sleigh-10.1.2-1.x86_64.tar.gz | sudo tar xz -C /usr/local --strip-components=1
+        # Z3 Ubuntu 20.04 package doesn't play nice with sanitizers
+        # (also remove top-level directory from zip)
+        wget -O z3.zip https://github.com/Z3Prover/z3/releases/download/z3-4.8.14/z3-4.8.14-x64-glibc-2.31.zip
+        dest=/opt/z3; zip=./z3.zip; \
+          sudo mkdir -p "${dest}" && \
+          sudo unzip -d "${dest}" "${zip}" && \
+            f=("${dest}"/*) && sudo mv "${dest}"/*/* "$dest" && sudo rmdir "${f[@]}"
+
+    - name: Configure
+      env: { CC: clang-11, CXX: clang++-11 }
+      run: cmake --preset=ci-sanitize -DZ3_ROOT=/opt/z3
+
+    - name: Build
+      run: |
+        cmake --build build/sanitize -j 2
+
+    - name: Test
+      working-directory: build/sanitize
+      env:
+        ASAN_OPTIONS: "strict_string_checks=1:\
+          detect_stack_use_after_return=1:\
+          check_initialization_order=1:\
+          strict_init_order=1:\
+          detect_leaks=1"
+        UBSAN_OPTIONS: print_stacktrace=1
+      run: ctest --output-on-failure -j 2
+
+  test:
+    strategy:
+      matrix:
+        os: [
+          # TODO: windows,
+          # TODO: macos,
+          ubuntu
+        ]
+
+        type: [shared, static]
+
+        include:
+        - { type: shared, shared: YES }
+        - { type: static, shared: NO }
+
+    runs-on: ${{ matrix.os }}-latest
+
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Setup Git User for Sleigh
+      # See this thread for more details https://github.community/t/github-actions-bot-email-address/17204/5
+      run: |
+        git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
+        git config --global user.name "github-actions[bot]"
+
+    - name: Install Dependencies
+      if: matrix.os == 'ubuntu'
+      run: |
+        sudo apt-get install libgmp-dev python3-dev libz3-dev
+        wget -O - -c https://github.com/lief-project/LIEF/releases/download/0.11.5/LIEF-0.11.5-Linux-x86_64.tar.gz | sudo tar xz -C /usr/local --strip-components=1
+        wget -O - -c https://github.com/lifting-bits/sleigh/releases/download/v10.1.2/Linux-sleigh-10.1.2-1.x86_64.tar.gz | sudo tar xz -C /usr/local --strip-components=1
+
+    - name: Install Dependencies
+      if: matrix.os == 'macos'
+      run: |
+        brew install gmp python3 z3
+        wget -O - -c https://github.com/lief-project/LIEF/releases/download/0.11.5/LIEF-0.11.5-Darwin-x86_64.tar.gz | sudo tar xz -C /usr/local --strip-components=1
+        wget -O - -c https://github.com/lifting-bits/sleigh/releases/download/v10.1.2/macOS-sleigh-10.1.2-1.x86_64.tar.gz | sudo tar xz -C /usr/local --strip-components=1
+
+    - name: Configure
+      run: cmake --preset=ci-${{ matrix.os }}
+        -D BUILD_SHARED_LIBS=${{ matrix.shared }}
+
+    - name: Build
+      run: cmake --build build --config Release -j 2 || true
+
+    - name: Install
+      run: cmake --install build --config Release --prefix prefix
+
+    - name: Test install directory
+      working-directory: tests/find-package
+      run: |
+        cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -Dmaat_DIR=${{ github.workspace }}/prefix/share/maat
+        cmake --build build
+        ./build/maat_find_package_test
+
+    - name: Test
+      working-directory: build
+      run: ctest --output-on-failure -C Release -j 2
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,9 @@
+.idea/
+.vs/
+.vscode/
+build/
+cmake/open-cpp-coverage.cmake
+cmake-build-*/
+prefix/
+CMakeLists.txt.user
+CMakeUserPresets.json
diff --git a/.gitmodules b/.gitmodules
@@ -0,0 +1,4 @@
+[submodule "src/third-party/sleigh/sleigh-cmake"]
+	path = src/third-party/sleigh/sleigh-cmake
+	url = https://github.com/lifting-bits/sleigh
+	branch = cmake-module-sleigh-compile
diff --git a/BUILDING.md b/BUILDING.md
@@ -0,0 +1,83 @@
+# Building & installing with CMake
+
+## Build
+
+This project requires a few dependencies:
+
+* [GMP](https://gmplib.org/) - `libgmp` for big-sized arithmetic
+* [Z3](https://github.com/Z3Prover/z3) - for constraint solving
+  * Can be disabled with CMake option `-Dmaat_USE_Z3=OFF` during configuration
+* [LIEF](https://github.com/lief-project/LIEF) - for automatic binary loading
+  * Can be disabled with CMake option `-Dmaat_USE_LIEF=OFF` during configuration
+* [sleigh](https://github.com/lifting-bits/sleigh) - for Ghidra sleigh pcode handling
+  * Vendored version can be built with CMake option `-Dmaat_USE_EXTERNAL_SLEIGH=OFF` during configuration after you pull the submodule (`git submodule update --init --recursive`)
+* Python3 development headers and library - for Python bindings
+  * Bindings can be skipped with CMake option `-Dmaat_BUILD_PYTHON_BINDINGS=OFF` during configuration
+
+You can use our [Dockerfile](Dockerfile) as a reference for how to download and install dependencies.
+
+Assuming the above are installed into a system location (where CMake will automatically find them), here are the steps for building in release mode with a single-configuration generator, like the Unix Makefiles one:
+
+```sh
+cmake -S . -B build -D CMAKE_BUILD_TYPE=Release
+cmake --build build
+```
+
+Here are the steps for building in release mode with a multi-configuration
+generator, like the Visual Studio ones:
+
+```sh
+cmake -S . -B build
+cmake --build build --config Release
+```
+
+If the dependencies are installed into a non-system/default directory, then you can [pass hints to CMake][1] on where to find the respective packages (something like `-D<PackageName>_DIR` or `-D<PackageName>_ROOT` depending on what/how the package was installed; see the package's documentation on how to correctly use it with CMake).
+
+## Install
+
+This project doesn't require any special command-line flags to install to keep
+things simple. As a prerequisite, the project has to be built with the above
+commands already.
+
+The below commands require at least CMake 3.15 to run, because that is the
+version in which [Install a Project][2] was added.
+
+Here is the command for installing the release mode artifacts with a
+single-configuration generator, like the Unix Makefiles one:
+
+```sh
+cmake --install build
+```
+
+Here is the command for installing the release mode artifacts with a
+multi-configuration generator, like the Visual Studio ones:
+
+```sh
+cmake --install build --config Release
+```
+
+### Python bindings
+
+Python bindings should be built and installed by default. If not, make sure the project is configured with the option `-Dmaat_BUILD_PYTHON_BINDINGS=ON`.
+
+CMake will install the Python module to the location specified by `maat_INSTALL_PYTHONMODULEDIR`; if it is an absolute path, it will be installed to that location, but if it is a relative path, then it will be located relative to the installation prefix.
+
+If `maat_INSTALL_PYTHONMODULEDIR` isn't specified, CMake uses a default location that matches the most common use-case for regular users, in a way that is equivalent to:
+
+```sh
+# If you're configuring outside a virtualenv
+prefix="$(python3 -m site --user-site)"
+# If you're configuring inside a virtualenv
+prefix="$(python3 -c 'import sysconfig as sc; print(sc.get_path("platlib"))')"
+
+cmake -S . -B build "-Dmaat_INSTALL_PYTHONMODULEDIR=${prefix}" -Dmaat_BUILD_PYTHON_BINDINGS=ON
+cmake --build build
+cmake --install build
+```
+
+NOTE: CMake configuration and installation should both take place either inside or outside of the virtual environment or else the install path for the Python module could be incorrect (especially on macOS).
+
+If you are packaging this project, you will likely want to change the default value of `maat_INSTALL_PYTHONMODULEDIR`.
+
+[1]: https://cmake.org/cmake/help/latest/command/find_package.html#config-mode-search-procedure
+[2]: https://cmake.org/cmake/help/latest/manual/cmake.1.html#install-a-project