Re-format introduction and overview to be more book-shaped

HACKING.md

@@ -0,0 +1,21 @@
+# Developing angr
+
+These are some guidelines so that we can keep the codebase in good shape!
+
+We try to get as close as the [PEP8 code convention](http://legacy.python.org/dev/peps/pep-0008/) as is reasonable without being dumb.
+If you use Vim, the [python-mode](https://github.com/klen/python-mode) plugin does all you need. You can also [manually configure](https://wiki.python.org/moin/Vim) vim to adopt this behavior.
+
+Most importantly, please consider the following when writing code as part of angr:
+
+- Try to use attribute access (see the `@property` decorator) instead of getters and setters wherever you can. This isn't Java, and attributes enable tab completion in iPython. That being said, be reasonable: attributes should be fast. A rule of thumb is that if something could require a constraint solve, it should not be an attribute.
+- Use our `.pylintrc`. It's fairly permissive, but our CI server will fail your builds if pylint complains under those settings.
+- DO NOT, under ANY circumstances, `raise Exception` or `assert False`. **Use the right exception type**. If there isn't a correct exception type, subclass the core exception of the module that you're working in (i.e. `AngrError` in angr, `SimError` in SimuVEX, etc) and raise that. We catch, and properly handle, the right types of errors in the right places, but `AssertionError` and `Exception` are not handled anywhere and force-terminate analyses.
+- Avoid tabs; use space indentation instead. Even though it's wrong, the de facto standard is 4 spaces. It is a good idea to adopt this from the beginning, as merging code that mixes both tab and space indentation is awful.
+- Avoid super long lines. It's okay to have longer lines, but keep in mind that long lines are harder to read and should be avoided. Let's try to stick to **120 characters**.
+- Avoid extremely long functions, it is often better to break them up into smaller functions.
+- Prefer `_` to `__` for private members (so that we can access them when debugging). *You* might not think that anyone has a need to call a given function, but trust us, you're wrong.
+- **Document** your code. Every *class definition* and *public function definition* should have some description of:
+ - What it does.
+ - What are the type and the meaning of the parameters.
+ - What it returns.
+- If you're pushing a new feature, and it is not accompanied by a test case, it **will be broken** in very short order. Please write test cases for your stuff.

INSTALL.md

@@ -0,0 +1,120 @@
+# Installing angr
+
+Angr is a python library, so it must be installed into your python environment before it can be used.
+It is built for Python 2: Py3k support is feasable somewhere out in the future, but we are a little hesitant to make that commitment right now (pull requests welcome!).
+
+We highly recommend using a [python virtual environment](https://virtualenvwrapper.readthedocs.org/en/latest/) to install and use angr.
+Several of angr's dependencies (z3, pyvex) require libraries of native code that are forked from their originals, and if you already have libz3 or libVEX installed, you definitely don't want to overwrite the official shared objects with ours.
+In general, don't expect support for problems arising from installing angr outside of a virtualenv.
+
+## Dependencies
+
+All of the python dependencies should be handled by pip and/or the setup.py scripts.
+You will, however, need to build some C to get from here to the end, so you'll need a good build environment as well as the python development headers.
+At some point in the dependency install process, you'll install the python library cffi, but (on linux, at least) it won't run unless you install your operating system's libffi package.
+
+On Ubuntu, you will want: `sudo apt-get install python-dev libffi-dev build-essential virtualenvwrapper`
+
+## Most Operating systems, all \*nix systems
+
+`mkvirtualenv angr && pip install angr` should usually be sufficient to install angr in most cases, since angr is published on the Python Package Index.
+
+Failing that, you can install angr by installing the following repositories (and the dependencies listed in their requirements.txt files), in order, from https://github.com/angr:
+
+- [claripy](https://github.com/angr/claripy)
+- [archinfo](https://github.com/angr/archinfo)
+- [pyvex](https://github.com/angr/pyvex)
+- [cle](https://github.com/angr/cle)
+- [simuvex](https://github.com/angr/simuvex)
+- [angr](https://github.com/angr/angr)
+
+## Mac OS X
+
+Before you say `pip install angr`, you need to rebuild our fork of z3 with `pip install -I --no-use-wheel angr-only-z3-custom`.
+
+## Windows
+
+You cannot install angr from pip on windows.
+You must install all of its components individually.
+
+Capstone is difficult to install on windows.
+You might need to manually specify a wheel to install, but sometimes it installs under a name different from "capstone", so if that happens you want to just remove capstone from the requirements.txt files in angr and archinfo.
+
+Z3 might compile on windows if you have a l33t enough build environment.
+If this isn't the case for you, you should download a wheel from somewhere on the internet.
+One location for pre-built Windows wheel files is <https://github.com/Owlz/angr-Windows>.
+
+If you build z3 from source, make sure you're using the unstable branch of z3, which includes floating point support.
+In addition, make sure to have `Z3PATH=path/to/libz3.dll` in your environment.
+
+## Development install
+
+We created a repo with scripts to make life easier for angr developers.
+You can set up angr in development mode by doing:
+
+```bash
+git clone https://github.com/angr/angr-dev
+cd angr-dev
+mkvirtualenv angr
+./setup.sh
+```
+
+This clones all of the repositories and installs them in editable mode.
+`setup.sh` can even create a PyPy virtualenv for you, resulting in significantly faster performance and lower memory usage.
+
+You can branch/edit/recompile the various modules in-place, and it will automatically reflect in your virtual environment.
+
+## Docker install
+
+For convenience, we ship a Dockerfile that is 99% guaranteed to work.
+You can install via docker by doing:
+
+```bash
+# install docker
+curl -sSL https://get.docker.com/ | sudo sh
+
+# clone the repo
+git clone https://github.com/angr/angr-dev
+
+# create the docker image
+sudo docker build -t angr - < angr-dev/Dockerfile
+
+# run it
+sudo docker run -it angr
+```
+
+Synchronization of files in and out of docker is left as an excercize to the user (hint: check out `docker -v`).
+
+# Troubleshooting
+
+## libgomp.so.1: version `GOMP_4.0' not found
+This error represents an incompatibility between the pre-compiled version of `angr-only-z3-custom` and the installed version of `libgomp`. A Z3 recompile is required. You can do this by executing:
+
+```bash
+pip install -I --no-use-wheel angr-only-z3-custom
+```
+
+## Can't import mulpyplexer
+There are sometimes issues with installing mulpyplexer. Doing `pip install --upgrade 'git+https://github.com/zardus/mulpyplexer'` should fix this.
+
+## Can't import angr because of capstone
+Sometimes capstone isn't installed correctly for use by angr. There's a good chance just reinstalling capstone will solve this issue:
+
+```bash
+pip install -I --no-use-wheel capstone
+```
+
+## Claripy and z3
+Z3 is a bit weird to compile. Sometimes it just completely fails to build for
+no reason, saying that it can't create some object file because some file or
+directory doesn't exist. Just retry the build:
+
+```bash
+pip install -I --no-use-wheel angr-only-z3-custom
+```
+
+## No such file or directory: 'pyvex_c'
+
+Are you running 12.04? If so, please upgrade!
+
+You can also try upgrading pip (`pip install -U pip`), which might solve the issue.

README.md

@@ -2,129 +2,34 @@
 
 This is a collection of documentation for angr. By reading this, you'll become an angr pro and will be able to fold binaries to your whim.
 
-To dive right into angr's capabilities, start with the [top level methods](./docs/toplevel.md).
-
-# What is angr?
-
-angr is a multi-architecture binary analysis platform, with the capability to perform dynamic symbolic execution (like Mayhem, KLEE, etc) and various static analyses on binaries. Several challenges must be overcome to do this. They are, roughly:
-
-- Loading a binary into the analysis program.
-- Translating a binary into an intermediate representation (IR).
-- Translating that IR into a semantic representation (i.e., what it *does*, not just what it *is*).
-- Performing the actual analysis. This could be:
- - A partial or full-program static analysis (i.e., dependency analysis, program slicing).
- - A symbolic exploration of the program's state space (i.e., "Can we execute it until we find an overflow?").
- - Some combination of the above (i.e., "Let's execute only program slices that lead to a memory write, to find an overflow.")
-
-angr has components that meet all of these challenges. This document will explain how each one works, and how they can all be used to accomplish your evil goals.
-
-# Using angr
-
-We've tried to make using angr as pain-free as possible.
-Our goal is to create the most user-friendly binary analysis platform, allowing one to simply start up iPython and easily perform extremely complex binary analyses with a couple of commands.
+We've tried to make using angr as pain-free as possible - our goal is to create a user-friendly binary analysis suite, allowing a user to simply start up iPython and easily perform intensive binary analyses with a couple of commands.
 That being said, binary analysis is complex, which makes angr complex.
-We've tried to make life easier by providing this documentation, split into several sections for easier consumption.
-
-## Installing angr
-
-### All operating systems
-
-Before angr can be used, it must be installed.
-Ideally, you should just be able to say `pip install angr`, but failing that, you should be
-able to install the following repositories (and the dependencies listed in their requirements.txt files)
-from http://github.com/angr:
-- angr
-- simuvex
-- claripy
-- cle
-- pyvex
-- archinfo
-
-### Mac OS X
-
-To install angr on Mac OS X, you also need to install angr-z3 by `pip install -I --no-use-wheel angr-only-z3-custom`.
-
-### Windows
-
-For Windows, please check more information at [https://github.com/angr/angr#windows-and-capstone](https://github.com/angr/angr#windows-and-capstone).
-
-
-## Loading a Binary
-
-After angr is installed, you can load a binary for analysis.
-This process, and the angr component that powers it (called CLE) is described [here](./docs/loading.md).
-
-## Intermediate Representation
-
-angr uses an intermediate representation (specifically, VEX) to enable it to run analyses on binaries of different architectures.
-This IR is described [here](./docs/ir.md).
-
-## Solver Engine
-
-Constraint solving and other computational needs are provided by an angr sub-module called Claripy.
-Most users of angr will not need to know anything about Claripy, but documentation is provided in case it is needed.
-Claripy is detailed [here](./docs/claripy.md).
-
-## Program States
-
-angr provides an interface to the emulated machine state.
-Understanding this is critical to successfully using angr.
-It is detailed [here](./docs/states.md).
-
-## Program Paths
-
-Programs can be analyzed in terms of the possible *path* that execution takes through them.
-angr exposes information about what the paths execute and *do*.
-[This section](./docs/paths.md) gives an overview of how to use this capability of angr.
-
-## Semantic Representation
-
-A powerful feature of angr is the ability to represent basic blocks in terms of their effects on a program state.
-In other words, angr can reason about what basic blocks *do*, not just what they *are*.
-This is accomplished by a module named SimuVEX, further described [here](./docs/simuvex.md).
-
-## Symbolic Execution
-
-angr provides a capable symbolic execution engine.
-The interface to this engine, and how to use it, is described [here](./docs/surveyors.md).
-
-## Full-program Analysis
-
-All of the above components come together to enable complex, full-program analyses to be easily run with angr.
-The mechanism for running and writing these analyses is detailed [here](./docs/analyses.md).
-
-# Examples
+This documentation is an attempt to help out with that, providing narritive explanation and exploration of angr and its design.
 
-We've written some examples for using angr!
-You can read more [here](./docs/examples.md).
+## Get Started
 
-# Developing angr
+Installation instructions can be found [here](./INSTALL.md).
 
-We've put together some guidelines so that we can keep the codebase in good shape!
+To dive right into angr's capabilities, start with the [top level methods](./docs/toplevel.md), or read over the [overview](./docs/overview.md).
 
-We try to get as close as the [PEP8 code convention](http://legacy.python.org/dev/peps/pep-0008/) as is reasonable without being dumb.
-If you use Vim, the [python-mode](https://github.com/klen/python-mode) plugin does all you need. You can also [manually configure](https://wiki.python.org/moin/Vim) vim to adopt this behavior.
+A searchable HTML version of this documentation is hosted at [docs.angr.io](http://docs.angr.io/), and an HTML API reference can be found at [angr.io/api-doc](http://angr.io/api-doc/).
 
-Most importantly, please consider the following when writing code as part of angr:
+## Citing angr
 
-- Try to use attribute access (see the `@property` decorator) instead of getters and setters wherever you can. This isn't Java, and attributes enable tab completion in iPython. That being said, be reasonable: attributes should be fast. A rule of thumb is that if something could require a constraint solve, it should not be an attribute.
-- Use our `.pylintrc`. It's fairly permissive, but our CI server will fail your builds if pylint complains under those settings.
-- DO NOT, under ANY circumstances, `raise Exception` or `assert False`. **Use the right exception type**. If there isn't a correct exception type, subclass the core exception of the module that you're working in (i.e. `AngrError` in angr, `SimError` in SimuVEX, etc) and raise that. We catch, and properly handle, the right types of errors in the right places, but `AssertionError` and `Exception` are not handled anywhere and force-terminate analyses.
-- Avoid tabs; use space indentation instead. Even though it's wrong, the de facto standard is 4 spaces. It is a good idea to adopt this from the beginning, as merging code that mixes both tab and space indentation is awful.
-- Avoid super long lines. It's okay to have longer lines, but keep in mind that long lines are harder to read and should be avoided. Let's try to stick to 120 characters.
-- Avoid extremely long functions, it is often better to break them up into smaller functions.
-- Prefer `_` to `__` for private members (so that we can access them when debugging). *You* might not think that anyone has a need to call a given function, but trust us, you're wrong.
-- **Document** your code. Every *class definition* and *public function definition* should have some description of:
- - What it does.
- - What are the type and the meaning of the parameters.
- - What it returns.
-- If you're pushing a new feature, and it is not accompanied by a test case, it **will be broken** in very short order. Please write test cases for your stuff.
+If you use angr in an academic work, please cite the paper for which it was developed:
 
-# FAQ
+```bibtex
+@article{shoshitaishvili2015firmalice,
+  title={Firmalice - Automatic Detection of Authentication Bypass Vulnerabilities in Binary Firmware},
+  author={Shoshitaishvili, Yan and Wang, Ruoyu and Hauser, Christophe and Kruegel, Christopher and Vigna, Giovanni},
+  year={2015}
+}
+```
 
-We've collected miscellaneous questions about angr, and answers to them, in a [FAQ](./docs/faq.md).
+## Support
 
-# Change log!
+To get help with angr, you can ask via:
 
-To track major changes in angr, we maintain a changelog.
-You can view it [here](./CHANGELOG.md).
+- the mailing list: [email protected]
+- the IRC channel: **#angr** on [freenode](https://freenode.net/)
+- opening an issue on the appropriate github repository

SUMMARY.md

@@ -1,6 +1,9 @@
 # Summary
 
 * [Introduction](README.md)
+   * [Overview](docs/overview.md)
+   * [Installing](INSTALL.md)
+   * [Developing](HACKING.md)
 * [Top-level interfaces](docs/toplevel.md)
 * [Loading a binary](docs/loading.md)
 * [Intermediate Representation](docs/ir.md)

docs/overview.md

@@ -0,0 +1,63 @@
+# What is angr?
+
+angr is a multi-architecture binary analysis platform, with the capability to perform dynamic symbolic execution (like Mayhem, KLEE, etc) and various static analyses on binaries. Several challenges must be overcome to do this. They are, roughly:
+
+- Loading a binary into the analysis program.
+- Translating a binary into an intermediate representation (IR).
+- Translating that IR into a semantic representation (i.e., what it *does*, not just what it *is*).
+- Performing the actual analysis. This could be:
+ - A partial or full-program static analysis (i.e., dependency analysis, program slicing).
+ - A symbolic exploration of the program's state space (i.e., "Can we execute it until we find an overflow?").
+ - Some combination of the above (i.e., "Let's execute only program slices that lead to a memory write, to find an overflow.")
+
+angr has components that meet all of these challenges. This book will explain how each one works, and how they can all be used to accomplish your evil goals.
+
+
+## Loading a Binary
+
+After angr is installed, you can load a binary for analysis.
+This process, and the angr component that powers it (called CLE) is described [here](./docs/loading.md).
+
+## Intermediate Representation
+
+angr uses an intermediate representation (specifically, VEX) to enable it to run analyses on binaries of different architectures.
+This IR is described [here](./docs/ir.md).
+
+## Solver Engine
+
+Constraint solving and other computational needs are provided by an angr sub-module called Claripy.
+Most users of angr will not need to know anything about Claripy, but documentation is provided in case it is needed.
+Claripy is detailed [here](./docs/claripy.md).
+
+## Program States
+
+angr provides an interface to the emulated machine state.
+Understanding this is critical to successfully using angr.
+It is detailed [here](./docs/states.md).
+
+## Program Paths
+
+Programs can be analyzed in terms of the possible *path* that execution takes through them.
+angr exposes information about what the paths execute and *do*.
+[This section](./docs/paths.md) gives an overview of how to use this capability of angr.
+
+## Semantic Representation
+
+A powerful feature of angr is the ability to represent basic blocks in terms of their effects on a program state.
+In other words, angr can reason about what basic blocks *do*, not just what they *are*.
+This is accomplished by a module named SimuVEX, further described [here](./docs/simuvex.md).
+
+## Symbolic Execution
+
+angr provides a capable symbolic execution engine.
+The interface to this engine, and how to use it, is described [here](./docs/surveyors.md).
+
+## Full-program Analysis
+
+All of the above components come together to enable complex, full-program analyses to be easily run with angr.
+The mechanism for running and writing these analyses is detailed [here](./docs/analyses.md).
+
+# Examples
+
+We've written some examples for using angr!
+You can read more [here](./docs/examples.md).