threat_model.tex

\section{Threat Model}\label{sec:threat-model}
In our threat model, we assume an attacker owns a copy of the target application and can run it in a malicious host environment \cite{collberg2002watermarking}, aka the white-box attack context \cite{chow2003white,liem2008compiler}.
In such an environment, the adversary has full privileged accesses to the system. We also assume the adversary can use static and dynamic analysis tools (such as, \texttt{IDA}~\cite{ida}, \texttt{OllyDbg}~\cite{od} and \texttt{Sysinternals Suite}~\cite{sysinternals}) to trace and analyze instructions, monitor registers and process memory, and modify instruction bytes and control flows at runtime, etc.
Prior work has demonstrated that these are reason assumptions~\cite{falliere2009inside}.
At present, there are two preliminarily used methods to attack VM-based protection systems, which are presented as follows.

The first is attack based on the virtual execution analysis proposed by Rolles et al. \cite{rolles2009unpacking}, which requires an analyst to have a certain understanding of the principle of code virtualization. By dynamically tracking the execution process of virtual interpreter to extract key bytecodes and handlers, and then through the analysis and simplify eventually recovering the original program's logic. Nicolas Falliere \cite{falliere2009inside} presented an example of the above analysis process which is used to analyze the Trojan.Clampi protected by VMProtect \cite{vmp}.
This type of attack method is closely related to the principle and structure of the code virtualization, and has the most realistic and comprehensive results.


The other one is attack based on the behavior and semantic analysis, this type of attack method can be used to attack not only code virtualization protection but also other confusion methods.
Coogan et al.~\cite{coogan2011deobfuscation} puts forward a behavior based analysis method, which aims to analyze the important behavior of code, but it does not pay attention to how to restore the original code. %This type of approach is usually used for malicious code analysis because the malicious code will interact with the system frequently in order to achieve a malicious purpose.
Yadegari et al.~\cite{Yadegari2015A} propose a method based on semantic analysis, which use taint propagation to track the flow of inputs values, and semantics-preserving code transformations to simplify the logic of the instructions.
%For the results of a run obtained, the function is equivalent to the original program, but it is only for one implementation, and does not cover all the execution branches. So the final control flow graph is only part of the original program. We need to perform analysis through multiple tracking and specify different input values each time, then comprehensive analysis to get a more complete control flow graph.
This type of method has wider applicability, but it is hard to get a comprehensive analysis results.
%So this paper mainly aims at the first kind of attack, but also will provide some measures to prevent the second kind of attack. And in our threat model, we assume that the analyst is familiar with the mechanism of code virtualized obfuscation and follows the above steps while reverse engineering a VM-obfuscated program. The ultimate goal of the analyst is to fully reverse engineer the VM-obfuscated application and automate the reverse analysis process.