This release adds some much needed features that increase the accuracy of the results Tern produces. This release also adds functionality targeted towards the usability of the results with other tools. The project itself added some documentation changes and Continuous Integration with Travis CI. We hope that these changes will make the project easier to contribute to.
Tern was one of the projects in 2018's Hacktoberfest. During this time several contributors merged changes which enabled these new features. We are so grateful for their time and efforts!
- We added a filesystem identifier (fs_hash) to replace the diff_id of the layer tarballs that make up a container image (if you want to read up on container image layers, take a look at this presentation). diff_id is the hash of the filesystem tarball. This is a timestamped tarball and hence it changes for every new build, regardless of whether the files added, removed or modified stay the same. Although this helps with checking artifacts after download, it doesn't refer to the filesystem contents. The hashing we do walks through the files and hashes their contents as well as collect some file stats such as file permissions, size and extended attributes. As a result, the hashes change only when the file contents themselves change. This keeps the hashes constant from build to build. diff_id is still maintained as one of the properties of an image layer.
- We now have an alternative method of finding the list of packages that were installed in an image layer. We used to rely on the package manager's dependency tree to list a package's dependencies exclusively. This made it hard to have a consistent way of interpreting what the package manager outputs. We also had an instance where there was no CLI option for the package manager to list dependent packages. We now mount the filesystem layer, list the packages using the package manager, and then diff this list with the list we created from mounting the previous layer.
- One of the big goals for Tern this release was to be able to output structured data so the output can be used in a gui or be consumed by a downstream tool. To this effect, we have enabled Tern to output JSON and YAML structured data by using the
-Jand-Yflags respectively. To try this out run./tern report -J -i <your local docker image>or./tern report -Y -i <your local docker image>. Locally, the YAML output is the most readable as structured data. The JSON output is for the future goal of running Tern in a container as a service. Stay tuned for this in Release 0.3.0 - We added a new option
-r, --redowhich you can use when you want to repopulate the cache when running your image. This is useful when a new command is added to the command library that will get you more accurate results. When this happens and you need to update your cache, rather than invalidating the whole cache, you can just replace the existing data with new data for the images that will be affected. To try it out, run./tern -r report -i <your local docker image>. This is a feature added to address this bug. - There are some commits referring to enabling builds using Travis CI. It took a good bit of work to get it running but this is reflected in a number of strange looking commits:
97156d8 Abandon docker images command and python 3.7 env
599588c Attempt to export result of docker images command
43c198e Attempt rearranging commands to use env
0e0cf11 Replace back ticks with shell eval
7d1b598 Add travis-ci build status in README.md
f0ae47d Tag docker image that was pulled by digest
99b6eb1 Fixed export of env variable in .travis.yml file
c94759a Fixed docker command in .travis.yml
516789e Added .travis.yml file for CI
We finally managed to get Travis CI working in a ubuntu xenial environment using Python 3.6 and Docker.
- We added a Vagrantfile and a bootstrap script so you can easily try out Tern in an isolated environment using Vagrant. This is especially useful for users of Windows and Mac OSX who do not have a Linux machine.
Tern's command library for base container images (base.yml) will no longer reference image names and tags. This in response to the way container images are currently built and distributed. We haven't come across any metadata that references the container image repositories that are used to make complex images after they are distributed. As a result, a container downloaded from an image registry only has metadata about individual layers. So Tern tries to "guess" what the base OS of the container is by looking for a package manager executable like 'dpkg' or 'rpm' in the first layer. These are the keys used in the command library instead.
- Doesn't crash when working on large Docker images
- Doesn't crash on a Docker image with unknown files in it
- Can repopulate cache with new results
Release 0.3.0 will be focused on:
- Containerizing Tern: this involves using the Docker API rather than the Docker CLI, writing a Dockerfile that will build the image, and then running Tern on this image itself to produce data that can be uploaded along with the resulting container image (as this is what Tern is meant to do)
- Daemonizing Tern: this is work towards running Tern as a web service. Tern should be able to run as a service in the background and respond to requests sent to it by another service. This would be a stretch goal.
Note: this changelog will not include these release notes
7cea842 Compare package objects rather than package names
a13c544 Rewrite cache when --redo is True
a8d5e9f Add command line argument to redo cache
ae4f1c0 Fixed error with skipping extraction of base layer
a4bea09 Load from cache only when redo=False
5107fbe Fixed error message for unknown filesystem
4405dc6 Check if working directory exists before removing
60dc5bf Improvements to CONTRIBUTING accessability
09eaf4d Added issue templates for different issues
a5011ac Added reference to Vagrant setup
68ee778 Added quick links and updated intro in README.md
c780350 Added an issue template
2940108 Update vagrant setup
5e817ba Remove existing_layers, check from save_to_cache
61f3a67 Added command line for JSON output file
9018439 JSON formatted report
e491c12 Changed cmdline option to -Y for yaml file output
0c4bf4a Add YAML generation command to README.md
1c1c6da Generate YAML report
9e82bf3 Added notices to dict representation of classes
c1c9c96 Updates for full functional testing
fdd88ed Add unit test for `notice_origin`
7e440b5 Added test for to_dict method in test_class_notice.py
09e2277 Improve unit test for classes/notice.py
8ab17e6 filled to_dict() method in image class
754c02d Completed to_dict() method in image_layer class
966b731 Add instance check for every add_* methods in classes folder
e13e170 90 Add to_dict to docker_image.py
97156d8 Abandon docker images command and python 3.7 env
599588c Attempt to export result of docker images command
43c198e Attempt rearranging commands to use env
0e0cf11 Replace back ticks with shell eval
7d1b598 Add travis-ci build status in README.md
f0ae47d Tag docker image that was pulled by digest
99b6eb1 Fixed export of env variable in .travis.yml file
c94759a Fixed docker command in .travis.yml
516789e Added .travis.yml file for CI
444a69c Add to_dict method to Origins class
6de7b74 Fix indent in to_dict method in image_layer class
9ab2200 Added stub method to_dict
686a7a9 Changes to CONTRIBUTING.md regarding access and VM
0950060 Added Deep Comparison to Package Object
f9ccdb0 Notice and NoticeOrigin object to dictionary
04de6f7 Add current git SHA to report
3605d5f Merge: Alternative method for finding dependencies
3189f6d Fixed error in parsing package listing
55b6b8b Retrieve layer packages with command object
fbb2bd6 Refactor: get shell commands from docker history
0d0a545 Made finding packages using base OS method generic
1512bba Support for package listing as a key in base.yml
dae5d73 Fixed fs hash reporting and PEP8 formatting
cd4ccbe Fix mounting multiple filesystem layers
917a153 Used subprocess check_output for docker save
ffcad6a Remove inode from filesystem hash script
811ba15 Replace diff_id with fs_hash as layer id
d74cd92 Replaced file stats and hash function with script
f4ea81f Merge pull request #76 from johnmm/74-generate-random-names
89d2f13 Create random Container, Tag, and Image values
daa214c Fix updating master list of packages
6f8d413 Merge pull request #75 from johnmm/issue_1_remove_TODO
d6c45c1 Fixes #1, removed TODO and created #74
39da221 Added utilities to find filesystem hash
5983b17 Fixed parsing of dpkg installed packages
f6cb14e Updated README based on release v0.1.0
Laurie Roy [email protected]
Vikram Yadav [email protected]
Eric Cheatham [email protected]
Manjiri Tapaswi [email protected]
Abhi Gupta [email protected]
clooooode [email protected]
Kyle [email protected]
solaxun [email protected]
kdestasio [email protected]
John Miller [email protected]
Email: [email protected]
Twitter: @nishakmr