Initial push to Github

.gitignore

@@ -0,0 +1,5 @@
+*.pyc
+__pycache__/
+config.py
+build/
+.~lock.*

README.md

@@ -0,0 +1,229 @@
+# QCSuper
+
+**QCSuper** is a tool communicating with Qualcomm-based phones and modems, allowing to **capture raw 2G/3G/4G** radio frames, among other things.
+
+It will allow you to **generate PCAP** captures of it using either a rooted Android phone, an USB dongle or an existing capture in another format.
+
+![Screenshot of using QCSuper along with Wireshark](docs/sample_pcaps/Wireshark%20screenshot.png?raw=true)
+
+After having [installed](#installation) it, you can plug your rooted phone in USB and using it is as simple as:
+
+```
+./qcsuper.py --adb --wireshark-live
+```
+
+It uses the Qualcomm Diag protocol, also called QCDM or DM (Diagnostic Monitor) in order to communicate with your phone's baseband.
+
+**You want support, to report that you device works or does not work or you'd like to join development doing research on the Diag protocol? You can [come talk on IRC (`#qcsuper` at Freenode)](http://webchat.freenode.net/?channels=#qcsuper) or open a [Github issue](https://github.com/P1sec/QCSuper/issues/new).**
+
+## Table of contents
+
+* **[Installation](#installation)**
+  * [Ubuntu and Debian installation](#ubuntu-and-debian-installation)
+  * [Windows installation](#ubuntu-and-debian-installation)
+* [Supported protocols](#supported-protocols)
+* **[Usage notice](#usage-notice)**
+
+**Annexes:**
+
+* [Using QCSuper with an USB modem](#using-qcsuper-with-an-usb-modem)
+* [Related tools using the Diag protocol](#related-tools-using-the-diag-protocol)
+
+**Blog post/demo:** [...]()
+
+**More documentation:**
+
+* [The Diag protocol](docs/The%20Diag%20protocol.md)
+* [QCSuper architecture](docs/QCSuper%20architecture.md)
+
+## Installation
+
+QCSuper was tested and developed on Ubuntu 16.04 and Windows 7. It depends on a few Python modules.
+
+Your phone must be rooted to use it. In order to check for compatibility with your phone, look up the phone's model on a site like [GSMArena](https://www.gsmarena.com/) and check whether it has a Qualcomm processor.
+
+In order to open PCAP files produces by QCSuper, you can use any Wireshark 2.x for 2G/3G frames, but you need at least Wireshark 2.5.x for 4G frames (and 2.6.x for individual NAS messages decrypted out of 4G frames).
+
+### Ubuntu and Debian installation
+
+Open a terminal and type the following:
+
+```bash
+# Download QCSuper
+git clone [email protected]:P1sec/QCSuper.git qcsuper
+cd qcsuper
+
+# Install dependencies
+sudo apt install python3-pip wireshark
+sudo pip3 install --upgrade pyserial crcmod https://github.com/ANSSI-FR/pycrate/archive/master.zip
+
+# Upgrade to a recent snapshot of Wireshark (needed for decoding 4G frames)
+sudo add-apt-repository ppa:dreibh/ppa
+sudo apt-get update
+sudo apt-get upgrade wireshark
+```
+
+### Windows installation
+
+On Windows, you will need to download and install your phone's USB drivers from your phone model. There is no generic way, search for your phone's model + "USB driver" or "ADB drive" on Google for instructions.
+
+Then, you need to ensure that your you can read your device using `adb`. You can find a tutorial on how to download and setup `adb` [here](https://www.xda-developers.com/install-adb-windows-macos-linux/). The `adb shell` command mush display a prompt to continue.
+
+Then, follow these links on order to:
+
+* [Install Python 3.6](https://www.python.org/ftp/python/3.6.5/python-3.6.5.exe) (be sure to check options to include it into PATH, install it for all users and install pip)
+* [Install Wireshark 2.6](https://1.eu.dl.wireshark.org/win32/Wireshark-win32-2.6.0.exe)
+* [Download and extract QCSuper](https://github.com/P1sec/QCSuper/archive/master.zip)
+
+To install the required Python modules, open your command prompt and type:
+
+```bash
+pip3 install --upgrade pyserial crcmod https://github.com/ANSSI-FR/pycrate/archive/master.zip
+```
+
+Still in your command prompt, move to the directory containing QCSuper using the `cd` command. You can then execute commands (which should start with `qcsuper.py` instead of `./qcsuper.py`).
+
+## Supported protocols
+
+QCSuper supports capturing a handful of mobile radio protocols. These protocols are put after a [GSMTAP header](http://osmocom.org/projects/baseband/wiki/GSMTAP), a standard header (encapsulated into UDP/IP) permitting to identify the protocol, and GSMTAP packets are put into a [PCAP file](https://wiki.wireshark.org/Development/LibpcapFileFormat) that is fully analyzable using Wireshark.
+
+2G/3G/4G protocols can be broken into a few "layers": layer 1 is the radio waves, layer 2 handles stuff like fragmentation, layer 3 is the proper signalling or data.
+
+QCSuper allows you most often to capture the layer 3, as it is the most pratical to analyze using Wireshark, and is what the Diag protocol provides natively (and all the interesting information is here).
+
+* 2G (GSM): Layer 3 and upwards (RR/...)
+* 2.5G (GPRS): Layer 2 and upwards (MAC-RLC/...) for data acknowledgements
+* 3G (UMTS): Layer 3 and upwards (RRC/...)
+  * Additionally, it supports reassembling SIBs (System Information Blocks, the data broadcast to all users) in separate GSMTAP frames, as Wireshark currently can't do it itself: flag `--reassemble-sibs`
+* 4G (LTE): Layer 3 and upwards (RRC/...)
+  * Additionally, it supports putting decrypted NAS message, which are embedded encrypted embedded into RRC packet, in additional frames: flag `--decrypt-nas`
+
+By default, the IP traffic sent by your device is not included, you see only the signalling frames. You can include the IP traffic you generate using the `--include-ip-traffic` option (IP being barely the layer 3 for your data traffic in 2G/3G/4G, at the detail that its headers may be compressed (ROHC) and a tiny PPP header may be included).
+
+The data traffic you send uses a channel different from the signalling traffic, this channed being is setup through the signalling traffic; QCSuper should thus show you all details releant to how this channel is initiated.
+
+## Usage notice
+
+In order to use QCSuper, you specify one input (e.g: `--adb` (Android phone), `--usb-modem`) and one or more modules (`--wireshark-live` for opening Wireshark, `--pcap-dump` for writing traffic to a PCAP file, `--info` for generic information about the device...).
+
+A few commands you can type are:
+
+```bash
+# Open Wireshark directly, using a rooted Android phone as an input
+./qcsuper.py --adb --wireshark-live
+
+# Same, but dump to a PCAP file instead of opening Wireshark directly
+./qcsuper.py --adb --pcap-dump /tmp/my_pcap.pcap
+
+# Same, but using an USB modem exposing a Diag serial port
+./qcsuper.py --usb-modem /dev/ttyHS2 --wireshark-live
+```
+
+Here is the current usage notice for QCSuper:
+
+```
+usage: qcsuper.py [-h] [--cli] [-v]
+                  (--adb | --usb-modem TTY_DEV | --dlf-read DLF_FILE | --json-geo-read JSON_FILE)
+                  [--info] [--pcap-dump PCAP_FILE] [--wireshark-live]
+                  [--memory-dump OUTPUT_DIR] [--dlf-dump DLF_FILE]
+                  [--json-geo-dump JSON_FILE] [--decoded-sibs-dump]
+                  [--reassemble-sibs] [--decrypt-nas] [--include-ip-traffic]
+                  [--start MEMORY_START] [--stop MEMORY_STOP]
+
+A tool for communicating with the Qualcomm DIAG protocol (also called QCDM or
+DM).
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --cli                 Use a command prompt, allowing for interactive
+                        completion of commands.
+  -v, --verbose         Add output for each received or sent Diag packet.
+
+Input mode:
+  Choose an one least input mode for DIAG data.
+
+  --adb                 Use a rooted Android phone with USB debugging enabled
+                        as input (requires adb).
+  --usb-modem TTY_DEV   Use an USB modem exposing a DIAG pseudo-serial port
+                        through USB.
+  --dlf-read DLF_FILE   Read a DLF file generated by QCSuper or QXDM, enabling
+                        interoperability with vendor software.
+  --json-geo-read JSON_FILE
+                        Read a JSON file generated using --json-geo-dump.
+
+Modules:
+  Modules writing to a file will append when it already exists, and consider
+  it Gzipped if their name contains ".gz".
+
+  --info                Read generic information about the baseband device.
+  --pcap-dump PCAP_FILE
+                        Generate a PCAP file containing GSMTAP frames for
+                        2G/3G/4G, to be loaded using Wireshark.
+  --wireshark-live      Same as --pcap-dump, but directly spawn a Wireshark
+                        instance.
+  --memory-dump OUTPUT_DIR
+                        Dump the memory of the device (may not or partially
+                        work with recent devices).
+  --dlf-dump DLF_FILE   Generate a DLF file to be loaded using QCSuper or
+                        QXDM, with network protocols logging.
+  --json-geo-dump JSON_FILE
+                        Generate a JSON file containing both raw log frames
+                        and GPS coordinates, for further reprocessing. To be
+                        used in combination with --adb.
+  --decoded-sibs-dump   Print decoded SIBs to stdout (experimental, requires
+                        pycrate).
+
+PCAP generation options:
+  To be used along with --pcap-dump or --wireshark-live.
+
+  --reassemble-sibs     Include reassembled UMTS SIBs as supplementary frames,
+                        also embedded fragmented in RRC frames.
+  --decrypt-nas         Include unencrypted LTE NAS as supplementary frames,
+                        also embedded ciphered in RRC frames.
+  --include-ip-traffic  Include unframed IP traffic from the UE.
+
+Memory dumping options:
+  To be used along with --memory-dump.
+
+  --start MEMORY_START  Offset at which to start to dump memory (hex number),
+                        by default 00000000.
+  --stop MEMORY_STOP    Offset at which to stop to dump memory (hex number),
+                        by default ffffffff.
+```
+
+## Using QCSuper with an USB modem
+
+You can use QCSuper with an USB modem exposing a Diag port using the `--usb-modem <device>` option, where `<device>` is the name of the pseudo-serial device on Linux (such as `/dev/ttyUSB0`, `/dev/ttyHS2` and other possibilites) or of the COM port on Windows (such as `COM3`).
+
+Please note that you need to use QCSuper as root in order to be able to talk with a serial port on Linux, except if you have changed your filesystem privileges.
+
+If you don't know which devices under `/dev` exposes the Diag port, you may have to try multiple of these. You can try to auto-detect it by stopping the ModemManager daemon, and using the following command: `sudo ModemManager --debug 2>&1 | grep -i 'port is QCDM-capable'` then Ctrl-C.
+
+Please note that if you're not able to use your device with for example ModemManager in first place, it is likely that it is not totally setup and that it will not work neither with QCSuper. A few possible gotchas are:
+
+  * You didn't apply the proper [mode switching](https://wiki.archlinux.org/index.php/USB_3G_Modem#Mode_switching) command for your device.
+
+  * If you bought a device that previously had a SIM from a different operator, your device may be sim-locked. You may have to use the unlock code from the former operator and submit it to the device like you would do for `sudo mmcli -i 0 --pin=<your_unlock_code>`
+
+If your Qualcomm-based USB device doesn't expose a Diag port by default, you may need to type the following through the AT port in order to enable the Diag port:
+
+```
+AT$QCDMG
+```
+
+Please note that only one client can communicate with the Diag port at the same time. Thus, QCSuper will not work if ModemManager is connected to your Diag port at the same time, even though there's no straigtforward way to prevent ModemManager to find your Diag device and connect to it.
+
+So, when using QCSuper with your modem on Linux, you must either stop ModemManager, either play with permissions/groups in order to prevent it to open the corresponding device, either try to close the descriptor open by ModemManager using gdb (this will only work if ModemManager didn't manage to exchange valid Diag data since it was started, otherwise it will hang).
+
+By default, QCSuper will detect that ModemManager is running and has opened the descriptor for your modem's device when it's the case, and will offer to stop its process.
+
+## Related tools using the Diag protocol
+
+There are a few other open tools implementing bits of the Diag protocol, serving various purposes:
+
+* [ModemManager](https://github.com/endlessm/ModemManager): the principal daemon enabling to use USB modems on Linux, implements bits of the Diag protocol (labelled as QCDM) in order to retrieve basic information about USB modem devices. (That's why it may interfere when using --usb-modem and QCSuper will propose to shut it down)
+* [SnoopSnitch](https://play.google.com/store/apps/details?id=de.srlabs.snoopsnitch&hl=fr) (specifically [gsm-parser](https://github.com/E3V3A/gsm-parser)): chiefly an Android application whose purpose is to detect potential attacks on the radio layer (IMSI catcher, fake BTS...). It also have a secondary feature to capture some signalling traffic to PCAP, which does not provide exactly the same thing as QCSuper (LTE traffic isn't encapsulated in GSMTAP for example, device support may be different).
+  * [diag-parser](https://github.com/moiji-mobile/diag-parser): A Linux tool derivate from the PCAP generation feature from SnoopSnitch, somewhat improved, designed to work with USB modems.
+* [MobileInsight](http://www.mobileinsight.net/): this Android application intends to parse all kinds of logs output by Qualcomm and Mediatek devices (not only those containing signalling information, but also proprietary debugging structures), and dumping these to a specific XML representation format. Does not provide user-facing PCAPs (but formerly used Wireshark as a backend for converting certain protocol information to XML).
+* [qcombbdbg](https://github.com/yingted/qcombbdbg): A debugger for the Qualcomm baseband setting up itself by hooking a Diag command, through using the Diag command that allows to write to memory, for the Option Icon 225 USB modem.
+* [OpenPST](https://github.com/openpst/openpst): A set of tools related to Qualcomm devices, including a GUI utility allowing, for example, to read data on the tiny embedded filesystem accessible through Diag (EFS).

docs/QCSuper architecture.md

@@ -0,0 +1,59 @@
+QCSuper can be split up into two building blocks:
+
+* **Inputs** are Python classes exposing interfaces to read and optionally send Diag protocol data. We can distinguish:
+  * Inputs for communicating live with devices (the `--adb` input for talking with rooted Android phones, the `--usb-modem` for talking with USB modems)
+  * Inputs for processing saved data (`--dlf-read` is able to read data in a format providing interoperability with the vendor QXDM tool)
+
+* **Modules** are Python classes using inputs to perform specific tasks using the Diag protocol. For example:
+  * Capturing raw 2G/3G/4G signalling traffic (`--pcap-dump` will dump to a PCAP file, `--wireshark-live` will open directly Wireshark)
+  * Gathering generation info about the device (`--info`)
+  * Capturing raw Diag logs information into reusable formats (`--json-geo-dump`, `--dlf-dump`)
+
+## Internal architecture
+
+QCSuper needs to deal to multiple sources of input:
+
+* The connected Diag device
+  * Which can be different kinds of descriptor:
+    * A pseudo-serial USB device
+    * A TCP socket communicating with a remote Android device
+    * A file to replay from
+  * Which can deliver different kinds of received data:
+    * Synchronous responses to requests
+    * Asynchronous logs or messages (see [The Diag protocol.md](The%20Diag%20protocol.md))
+  * When the source is a real device, the device can accept only one Diag client at once
+
+* An optional interactive prompt (`--cli` module): needed to provide a handy way to continue capturing on the Diag client while executing other tasks
+
+All this requires a form of concurrency to be acheived: either threading, or a way to poll on descriptors through an event loop.
+
+The design ease/simplicity tradeoff I have chosen was to use threading (but I'm open to rework the architecture if someone has something else to propose). Polling on both a serial port and a featureful command prompt or thread queue (for example) is not doable easily in a multi-platform way, and using asyncio seemed to add some design and syntaxing overhead/external libraries in the equation.
+
+### Threading model
+
+QCSuper makes uses of different threads:
+
+* The main thread contains the loop reading from the device, and is the only place where reading is performed (it will also dispatch asynchronous messages to modules, calling the `on_log`, `on_message` which may not write neither read, and call at teardown the `on_deinit` callback with may write)
+* A background thread is used for initializing the modules selected through command line arguments (calling the `on_init` callback which may write)
+* A background thread may be used for the optional interactive prompt (`--cli`) and initializing the modules called from it (calling the `on_init` callback which may write)
+
+### Modules API
+
+A module is a Python class which may expose different methods:
+
+* `__init__`: will receive the input object as its first argument, and optionally other arguments from the command line or interactive prompt (passed in order from the entry point `qcsuper.py`).
+* `on_init`: called when the connection to the Diag device was established. Not called when the input is not a device but a file containing recorded log data.
+* Callbacks triggered by a read on the input source:
+  * `on_log`: called when an asynchronous response Diag protocol raw "log" was received.
+  * `on_message`: called when an asynchronous response Diag protocol text "message" structure was received.
+* `on_deinit`: called when the connection to the Diag device ceased establishment, or the user hit Ctrl+C.
+
+The methods composing these callback may perform request-response operations (using `self.diag_reader.send_recv(opcode, payload)`, where `self.diag_reader` is the input object).
+
+When a request-response operation is performed, the thread for the callback is paused using the response is received, using a thread synchronization primitive shared with the main thread.
+
+When using the interactive prompt (`--cli`), the moment where the `on_init` callbacks ends is the moment where the used is informed that the task continued to background (or is finished, in the case where there is no further callbacks).
+
+### Inputs API
+
+To be documented