vltrace.1

.\" Automatically generated by Pandoc 1.19.1
.\"
.TH "vltrace" "1" "" "" "" ""
.hy
.\" Copyright 2016-2017, Intel Corporation
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\"
.\"     * Redistributions of source code must retain the above copyright
.\"       notice, this list of conditions and the following disclaimer.
.\"
.\"     * Redistributions in binary form must reproduce the above copyright
.\"       notice, this list of conditions and the following disclaimer in
.\"       the documentation and/or other materials provided with the
.\"       distribution.
.\"
.\"     * Neither the name of the copyright holder nor the names of its
.\"       contributors may be used to endorse or promote products derived
.\"       from this software without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.SH NAME
.PP
\f[B]vltrace\f[] \-\- extremely fast strace\-like tool built on top of
eBPF, Kprobe and Tracepoint technologies.
.SH SYNOPSIS
.PP
\f[C]vltrace\ [options]\ [command\ [arg\ ...]]\f[]
.SH DESCRIPTION
.PP
vltrace is a strace equivalent tool for Linux with limited functionality
based on eBPF, Kprobe and Tracepoint technologies and libbcc library.
.SH FEATURES
.IP \[bu] 2
Used combination of technologies allow tool to be about one order of
magnitude faster than regular system strace.
.IP \[bu] 2
This tool consumes fewer resources.
.IP \[bu] 2
Output of this tool is designed to be suitable for processing with
classical tools and technologies, like awk.
.IP \[bu] 2
Could trace syscalls system\-wide.
.IP \[bu] 2
Could trace init (process with \[aq]pid == 1\[aq]) \- finally we have a
proper tool for debugging systemd ;\-)
.PP
WARNING: System\-wide tracing can fill out your disk really fast.
.SH LIMITATIONS
.IP \[bu] 2
Limited functionality
.IP \[bu] 2
Slow attaching and detaching
.IP \[bu] 2
Depends on modern kernel features
.IP \[bu] 2
The underlying eBPF technology is still in active development, so we can
expect hangs and crashes, especially on low\-res systems.
.SH Outdated limitations
.PP
Could be useful with old libbcc versions.
.IP \[bu] 2
Limited possibility to run few instances simultaneously.
Details:
.RS 2
.IP \[bu] 2
https://github.com/iovisor/bcc/pull/918
.IP \[bu] 2
https://github.com/iovisor/bcc/issues/872
.RE
.SH SYSTEM REQUIREMENTS
.IP \[bu] 2
kernel v4.7 or later (to attach eBPF to tracepoints)
.IP \[bu] 2
kernel headers installed:
.RS 2
.IP \[bu] 2
\[aq]kernel\-devel\[aq] package on RHEL, Fedora and CentOS or
.IP \[bu] 2
\[aq]linux\-headers\[aq] package on Debian and Ubuntu
.RE
.IP \[bu] 2
libbcc v0.4.0
.IP \[bu] 2
CAP_SYS_ADMIN capability (required by the bpf() syscall)
.IP \[bu] 2
mounted debugfs and tracefs
.SH OPTIONS
.SH Output format:
.PP
\f[C]\-o,\ \-\-output\ <file>\f[]
.PP
filename
.PP
\f[C]\-t,\ \-\-timestamp\f[]
.PP
include timestamp in output
.PP
\f[C]\-l,\ \-\-format\ <fmt>\f[]
.PP
output logs format.
Possible values: \[aq]bin\[aq] or \[aq]text\[aq]:
.IP \[bu] 2
\[aq]bin\[aq] \- the binary format, allows the fastest operation.
It is described in generated trace.h.
If current directory is not writable, generating of trace.h is skipped.
.IP \[bu] 2
\[aq]text\[aq] \- the text log format.
.PP
Default: \[aq]text\[aq]
.PP
\f[C]\-\-hex\-separator\ <sep>\f[]
.PP
set field separator for hex logs.
Default is a single space \[aq] \[aq].
.SH Filtering:
.PP
\f[C]\-\-failed\f[]
.PP
only show failed syscalls
.PP
\f[C]\-\-expr\ <expr>\f[]
.PP
defines which syscalls should be traced: 1) Intercepting using both
Kprobes and Tracepoints (requires kernel >= 4.7): \- \[aq]all\[aq] all
syscalls provided by the kernel using: \- Kprobes on syscalls\[aq] entry
and \- Tracepoint (raw syscall sys_exit) on syscalls\[aq] exit.
This is the default and recommended option.
2) Intercepting using Kprobes only: \- \[aq]kp\-all\[aq] \- all syscalls
provided by kernel \- \[aq]kp\-fileio\[aq] \- all syscalls related to
file IO \- \[aq]kp\-file\[aq] \- all syscalls with path arguments \-
\[aq]kp\-desc\[aq] \- all syscalls with file descriptor arguments
.PP
Default: all
.SH Tracing:
.PP
\f[C]\-f,\ \-\-full\-follow\-fork\f[]
.PP
Follow new processes created with fork()/vfork()/clone() syscall as
regular strace does.
.PP
\f[C]\-s,\ \-\-string\-args\ <length>\f[]
.PP
defines the maximum possible length of string arguments read by vltrace.
eBPF virtual machine is extremely limited in available memory.
Also currently there is no way to calculate the length of a string
argument.
For this reason there are four modes of fetching such arguments chosen
depending on value of \[aq]length\[aq]:
.IP \[bu] 2
\[aq]fast\[aq] \- for \[aq]length\[aq] <= 126:
.RS 2
.IP
.nf
\f[C]
\ \ \ \ \ \ 1\ packet\ is\ generated\ per\ each\ syscall,\ maximum\ length\ of\ a\ string\ depends\ on\ number\ of\ string\ arguments\ in\ the\ syscall:
\ \ \ \ \ \ \-\ 1\ string\ argument\ \ =\ 382,
\ \ \ \ \ \ \-\ 2\ string\ arguments\ =\ 190,
\ \ \ \ \ \ \-\ 3\ string\ arguments\ =\ 126,
\ \ \ \ \ \ This\ is\ the\ fastest\ mode.
\f[]
.fi
.RE
.IP \[bu] 2
\[aq]packet\[aq] \- for \[aq]length\[aq] <= 382:
.RS 2
.IP
.nf
\f[C]
\ \ \ \ \ \ 1\ packet\ is\ generated\ per\ each\ string\ argument,\ maximum\ length\ of\ a\ string\ is\ 382.
\f[]
.fi
.RE
.IP \[bu] 2
\[aq]const\[aq] \- for \[aq]length\[aq] > 382 and kernel version < 4.11:
.RS 2
.IP
.nf
\f[C]
\ \ \ \ \ \ Constant\ number\ N\ of\ packets\ is\ generated\ per\ each\ string\ argument,\ counted\ depending\ on\ value\ of\ \[aq]length\[aq].
\ \ \ \ \ \ Maximum\ length\ of\ a\ string\ is\ the\ smallest\ value\ of\ (N\ *\ 383\ \-\ 1)\ that\ is\ greater\ or\ equal\ to\ \[aq]length\[aq].
\f[]
.fi
.RE
.IP \[bu] 2
\[aq]full\[aq] \- for \[aq]length\[aq] > 382 and kernel version >= 4.11:
.RS 2
.IP
.nf
\f[C]
\ \ \ \ \ \ Variable\ number\ N\ of\ packets\ is\ generated\ per\ each\ string\ argument,\ depending\ on\ the\ actual\ length\ of\ each\ string\ argument.
\ \ \ \ \ \ Maximum\ length\ of\ a\ string\ is\ the\ smallest\ value\ of\ (N\ *\ 383\ \-\ 1)\ that\ is\ greater\ or\ equal\ to\ \[aq]length\[aq].
\f[]
.fi
.RE
.PP
Default: fast
.SH Startup:
.PP
\f[C]\-p,\ \-\-pid\ <PID>\f[]
.PP
trace the process with this PID only.
It excludes the \f[C]command\f[] argument: the process to be traced can
be defined by exactly one of the options: \[aq]command\[aq] or this one.
Press (CTRL\-C) to send interrupt signal to exit.
Note:
.IP
.nf
\f[C]
\-p\ "`pidof\ PROG`"
\f[]
.fi
.PP
syntax.
.PP
\f[C]\-\-ebpf\-src\-dir\ <dir>\f[]
.PP
Enable checking of updated ebpf templates in directory <dir>.
.SH Miscellaneous:
.PP
\f[C]\-d,\ \-\-debug\f[]
.PP
enable debug output
.PP
\f[C]\-h,\ \-\-help\f[]
.PP
print help
.PP
\f[C]\-\-list\f[]
.PP
Print a list of all traceable syscalls of the running kernel.
.PP
\f[C]\-\-ll\-list\f[]
.PP
Print a list of all traceable low\-level funcs of the running kernel.
.PP
WARNING: really long.
~45000 functions for 4.4 kernel.
.PP
\f[C]\-\-builtin\-list\f[]
.PP
Print a list of all known syscalls.
.SH CONFIGURATION
.PP
** System configuration **
.IP "1." 3
You should provide permissions to access tracefs for final user
according to your distro documentation.
Some of possible options:
.RS 4
.IP \[bu] 2
In /etc/fstab add mode=755 option for debugfs AND tracefs.
.IP \[bu] 2
Use sudo
.RE
.IP "2." 3
It\[aq]s a good idea to put this command in init scripts such as
local.rc:
.RS 4
.PP
echo 1 > /proc/sys/net/core/bpf_jit_enable
.PP
It will significantly improve performance and avoid \[aq]Lost
events\[aq]
.RE
.IP "3." 3
You should increase "Open File Limit" according to your distro
documentation.
Few common ways you can find in this instruction:
.RS 4
.PP
https://easyengine.io/tutorials/linux/increase\-open\-files\-limit/
.RE
.IP "4." 3
Kernel headers for running kernel should be installed.
.IP "5." 3
CAP_SYS_ADMIN capability should be provided for user for bpf() syscall.
.SH EXAMPLES
.SH Example output:
.PP
# ./vltrace \-l hex
.PP
\&./vltrace \-l hex
.PP
PID ERR RES SYSCALL ARG1 ARG2 ARG3 AUX_DATA
.PP
0000000000000AFD 000000000000000B FFFFFFFFFFFFFFFF read 0000000000000005
.PP
0000000000000427 0000000000000000 0000000000000020 read 000000000000000A
.PP
0000000000000B3D 0000000000000000 0000000000000001 write
000000000000001C
.PP
0000000000000B11 0000000000000000 0000000000000001 read 000000000000001B
.PP
0000000000000427 0000000000000000 0000000000000020 read 000000000000000A
.PP
0000000000000B3D 0000000000000000 0000000000000001 write
000000000000001C
.PP
0000000000000B11 0000000000000000 0000000000000001 read 000000000000001B
.PP
0000000000000B3D 0000000000000000 0000000000000001 write
000000000000001C
.PP
0000000000000B11 0000000000000000 0000000000000001 read 000000000000001B
.PP
0000000000000B3D 0000000000000000 0000000000000001 write
000000000000001C
.PP
0000000000000B11 0000000000000000 0000000000000001 read 000000000000001B
.PP
\&...
.PP
^C
.PP
#
.SH The \-p option can be used to filter on a PID, which is filtered
in\-kernel.
.PP
Here \-t option is used to print timestamps:
.PP
# ./vltrace \-l hex \-tp 2833
.PP
\&./vltrace \-l hex \-tp 2833 PID TIME(usec) ERR RES SYSCALL ARG1 ARG2
ARG3 AUX_DATA
.PP
0000000000000B11 0000000000000000 0000000000000000 0000000000000001 read
000000000000001B
.PP
0000000000000B11 0000000000004047 0000000000000000 0000000000000001 read
000000000000001B
.PP
0000000000000B11 0000000000008347 0000000000000000 0000000000000001 read
000000000000001B
.PP
0000000000000B11 000000000000C120 0000000000000000 0000000000000001 read
000000000000001B
.PP
0000000000000B11 000000000000C287 0000000000000000 0000000000000001 read
000000000000001B
.PP
0000000000000B11 000000000000C508 0000000000000000 0000000000000001 read
000000000000001B
.PP
0000000000000B11 0000000000010548 0000000000000000 0000000000000001 read
000000000000001B
.PP
0000000000000B11 00000000000144A4 0000000000000000 0000000000000001 read
000000000000001B
.PP
\&...
.PP
^C
.PP
#
.SH The \-X option only prints failed syscalls:
.PP
# ./vltrace \-l hex \-X mkdir .
.PP
\&./vltrace \-l hex \-X mkdir .
.PP
PID ERR RES SYSCALL ARG1 ARG2 ARG3 AUX_DATA
.PP
000000000000441A 0000000000000002 FFFFFFFFFFFFFFFF open
/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo mkdir
.PP
000000000000441A 0000000000000002 FFFFFFFFFFFFFFFF open
/usr/share/locale/en/LC_MESSAGES/coreutils.mo mkdir
.PP
000000000000441A 0000000000000002 FFFFFFFFFFFFFFFF open
/usr/share/locale\-langpack/en_US/LC_MESSAGES/coreutils.mo mkdir
.PP
000000000000441A 0000000000000002 FFFFFFFFFFFFFFFF open
/usr/lib/x86_64\-linux\-gnu/charset.alias mkdir
.PP
000000000000441A 0000000000000002 FFFFFFFFFFFFFFFF open
/usr/share/locale/en_US/LC_MESSAGES/libc.mo mkdir
.PP
000000000000441A 0000000000000002 FFFFFFFFFFFFFFFF open
/usr/share/locale/en/LC_MESSAGES/libc.mo mkdir
.PP
000000000000441A 0000000000000002 FFFFFFFFFFFFFFFF open
/usr/share/locale\-langpack/en_US/LC_MESSAGES/libc.mo mkdir
.PP
000000000000441A 0000000000000002 FFFFFFFFFFFFFFFF open
/usr/share/locale\-langpack/en/LC_MESSAGES/libc.mo mkdir
.PP
#
.PP
The ERR column is the system error number.
Error number 2 is ENOENT: no such file or directory.
.SH SEE ALSO
.PP
\f[B]strace\f[](1), \f[B]bpf\f[](2),
\f[B]<https://github.com/pmem/vltrace>\f[].
.PP
Also Documentation/networking/filter.txt in kernel sources.