Configurable system call tracer in QEMU emulator

Sometimes programmers face the task of analyzing the work of a compiled program to find its flaws, defects, and even search for malicious code in it. To analyze the work of such applications, we have to study their binary code or try to decompile the code, which is a laborious task. In order to simplify the analysis of applications, we can use the system calls of this application. System calls provide an essential interface between a program and the operating system. It is possible to track which system calls the application makes, and draw conclusions about the behavior of the program. This method allows us to debug the application without delving into the level of instructions and architecture features, thereby reducing the time required to find the problem.

Debugging applications using system tracing can be done inside the operating system, but still a number of problems arise:

• •    strong dependence of the debugger on the operating system;

• •    impossibility to run several debuggers at the same time;

• •    inaccessibility to the privileged execution;

• •    necessity to secure the operating system when analyzing programs that have harmful effects.

• 2.    Approach and uniqueness

To solve these problems, we can use the virtual machine tools. In this way, we can debug applications in a wide range of different operating systems running under different processor architectures.

To date, several debuggers allow us to trace an application using system calls. All these debuggers have a drawback - they do not provide enough portability of the debugger within different operating systems and processor architectures. We offer a new approach to implementing the debugger through system calls, by loading all the information necessary for tracing from the configuration file. The configuration files will allow us to easily configure and change the parameters needed for debugging, and to simplify the addition of support for new operating systems and architectures without recompiling the program and learning the debugger code.

It was decided to implement the debugger under the virtual machine QEMU [ 1 ], using the plugin mechanism. QEMU is an open source virtual machine that emulates the hardware of various platforms. This virtual machine supports the emulation of a large number of processor architectures such as x86, PowerPC, ARM, MIPS, SPARC, m68k. In addition, this simulator supports the launch of a large number of different operating systems.

Now, there is a plugin mechanism for QEMU implemented by ISP RAS [2], which allows us to connect developed plugins to a virtual machine during its both startup and operation. The implementation of the plugin mechanism enables each additional translation of the instruction to be substituted by an additional code for execution, when this instruction is called. This mechanism is suitable for debugging through system calls, so it was decided to use it.

In addition, various mechanisms of the system call play an important role. The classical way of implementation is the use of interrupts. With the help of interrupts, control is transferred to the kernel of the operating system, with the application having to enter the number of the system call and the necessary arguments into the corresponding registers of the processor.

For many RISC processors, this method is the only one; however, the CISC architecture has additional methods. The two mechanisms developed by AMD and Intel are independent of each other, but, in fact, perform the same functions. These are SYSCALL / SYSRET or SYSENTER / SYSEXIT statements. They allow us to transfer control to the operating system without interrupts.

Each operating system supports values returned from the system call, which are passed as reference types when the system call handler is called. Dining the execution of the system call, the service procedure records the required values if necessary by the available links, after which the system call is exited.

One of the main tasks that we had to face was the task of supporting the plugin of different operating systems and processor architectures. The solution to this problem was the interface with the configuration file. The configuration file makes the debugger more flexible and customizable. With its help, we can disconnect a certain mechanism of system calls from the trace or disable unnecessary system calls. In addition, such a mechanism makes it easier to add support for new operating systems and processor architectures.

To implement the interface with the configuration file, it was necessary to study a wide range of different operating systems and processor architectures. After gathering the necessary information, we can determine the information necessary for debugging: what type of system call is supported by SYSCALL / SYSRET or SYSENTER / SYSEXIT and their opcodes; location of system call arguments; a list of system calls, with the name of each system call, its code, and the list of arguments. Thus, by developing an interface for debugger and configuration file interfacing, we can add support for operating systems without going into the debugger code.

When implementing the debugger interaction interface with the configuration file, it became necessary to recognize the various expressions read from the file. For this task, we used the generator of the bison parser and developed the corresponding grammar [3].

• 3.    Background and related work

• 4.    Conclusion and discussion

Now, there are several debuggers to solve existing problems. Nitro [4] allows us to trace system calls, but it works only under Intel x86 architecture. Another debugger - Panda [5], can also trace system calls, supporting such operating systems as Linux, Windows 7, Windows XP and two architectures of the І386 processor and ARM. The description of all system calls is found in the code of this debugger, because of which this approach makes it difficult to add support for new operating systems and processor architectures, and worsens the flexibility in configuring the plugin, since the system debugger settings mechanism is not provided.

Based on the results of the work done, the plugin was developed in the QEMU virtual machine, with which we can trace and debug an application using system calls. As input to the plugin, the configuration file corresponding to the operating system running in the QEMU virtual machine and corresponding to the selected processor architecture is used.

The structure of the configuration file consists of 4 parts. The first part provides information about the operating system, its name and bit capacity. The second part is responsible for the supported mechanisms of system calls. The next part contains the location of the system call arguments. The last part includes a list of all available system calls and service information about the arguments of the system calls.

Because of the plugin’s work, a log file containing all the system calls that the plugin has intercepted is created. Each system call displays detailed information: the name and value of each system call argument, the number of the thread of execution from which this system call was made and the value that returned the system call after execution. Fig. 1 presents a small fragment of the output file that was created by the implementation of the plugin launched in the windows XP operating system and the І386 processor architecture.

ӨхЗе84ӨӨӨ entr:

ӨхЗе84ӨӨӨ exit: return: ӨхӨ

8x114: NtWriteRequestData 8x114: NtWriteRequestData

8x3e84888 entr: Өхс4: NtReplyWaitReceivePortEx 8x3e84888 entr: 8x112: NtWriteFile arg 8: Өх2а4( arg 1: 8x8( arg 2: 8x8( arg 3: 8x8( arg 4: 8x8ff6d8( arg 5: 8x9859f8( arg 6: 8xbc( arg 7: 8x8ff6e8( arg 8: 8x8(

8x3e84888 exit: 8x112:

HANDLE FileHandle )

HANDLE Event )

PIO APC ROUTINE ApcRoutine )

PVOID ApcContext )

PIO STATUS BLOCK loStatusBlock )

PVOID Buffer )

ULONG Length )

PLARGE INTEGER ByteOffset )

PULONG Key )

NtWriteFile return: 8x8

8x3e84888

arg 5: 8x3e84888

•ntr: 8x74: 8x8ff6c4 8x188188 8x8ff688 8x8ff6a4 8x7 8x284828

*xit: 8x74:

NtOpenFile

( PHANDLE FileHandle )

( ACCESS MASK DesiredAccess )

( POBJECT ATTRIBUTES ObjectAttributes

( PIO STATUS BLOCK loStatusBlock )

( ULONG ShareAccess )

( ULONG OpenOptions ) NtOpenFile return: 8x8

8x3e84888 entr: ӨхеӨ: NtSetlnformationFile arg 8: 8x31c      ( HANDLE FileHandle )

arg 1: 8x8ff6a4   ( PIO STATUS BLOCK loStatusBlock )

arg 2: 8x8ff658   ( PVOID Fileinformation )

arg 3: 8x28        ( ULONG Length )

arg 4: 8x4        ( FILE INFORMATION CLASS FilelnformationClass

8x3e84888 exit: ӨхеӨ: NtSetlnformationFile return: ӨхӨ ӨхЗе84ӨӨӨ entr: Өх19: NtClose arg Ө: ӨхЗІс ( HANDLE Handle ) ӨхЗе84ӨӨӨ exit: Өх19: NtClose return: ӨхӨ

Fig. 1. Part of the output file of the plugin

Upon the information gathered in the log fde, we can analyze the operation of the debugged application running inside the virtual machine. The operating system load time when using the developed plugin is increased 20% slowdown compared to the time of the operating system loading without this plugin.

Acknowledgments

The work was supported by the Russian Foundation of Basic Research (research grant 18-07-00900 A)