zq is a command-line tool for searching and analyzing logs,
particularly Zeek logs. If you are familiar with
zeek-cut,
you can think of zq as zeek-cut on steroids.
zq is comprised of:
- an execution engine for log pattern search and analytics,
- a query language that compiles into a program that runs on the execution engine, and
- an open specification for structured logs, called ZNG.
(Note: The ZNG format is in Alpha and subject to change.)
zq takes Zeek/ZNG logs as input and filters, transforms, and performs
analytics using the
zq query language,
producing a log stream as its output.
We don't yet distribute pre-built binaries, so to install zq, you must
clone the repo and compile the source.
If you don't have Go installed, download and install it from the Go downloads page.
If you're new to Go, remember to set GOPATH. A common convention is to create ~/go and point GOPATH at $HOME/go.
To install the binaries in $GOPATH/bin, clone this repo and
execute make install:
git clone https://github.com/brimsec/zq
cd zq
make install
For zq command usage, see the built-in help by running:
zq help
zq program syntax and semantics are documented in the
query language README.
Here are a few examples based on a very simple "conn" log from Zeek (conn.log), located in this directory. See the zq-sample-data repo for more test data, which is used in the examples in the query language documentation.
To cut the columns of a Zeek "conn" log like zeek-cut does, run:
zq "* | cut ts,id.orig_h,id.orig_p" conn.log
The "*" tells zq to match every line, which is sent to the cut processor
using the UNIX-like pipe syntax.
When looking over everything like this, you can omit the search pattern as a shorthand and simply type:
zq "cut ts,id.orig_h,id.orig_p" conn.log
The default output is a ZNG file. If you want just the tab-separated lines
like zeek-cut, you can specify text output:
zq -f text "cut ts,id.orig_h,id.orig_p" conn.log
If you want the old-style Zeek ASCII TSV
log format, run the command with the -f flag specifying zeek for the output
format:
zq -f zeek "cut ts,id.orig_h,id.orig_p" conn.log
You can use an aggregate function to summarize data over one or more fields, e.g., summing field values, counting, or computing an average.
zq "sum(orig_bytes)" conn.log
zq "orig_bytes > 10000 | count()" conn.log
zq "avg(orig_bytes)" conn.log
The ZNG specification describes the significance of the
_path field. By leveraging this, diverse Zeek logs can be combined into a single
file.
zq *.log > all.zng
Revisiting the cut example shown above:
zq -f text "cut ts,id.orig_h,id.orig_p" conn.log
This is functionally equivalent to the zeek-cut command-line:
zeek-cut ts id.orig_h id.orig_p < conn.log
If your Zeek events are stored as JSON and you are accustomed to querying with jq,
the equivalent would be:
jq -c '. | { ts, "id.orig_h", "id.orig_p" }' conn.ndjson
Comparisons of other simple operations and their relative performance are described at the performance page.
See the contributing guide on how you can help improve zq!
Join our Public Slack workspace for announcements, Q&A, and to trade tips!