daq.txt

The Data AcQuisition library (DAQ), provides pluggable packet I/O.  The DAQ
replaces direct calls to libraries like libpcap with an abstraction layer
that facilitates operation on a variety of hardware and software interfaces
without requiring changes to Snort.  It is possible to select the DAQ type
and mode when invoking Snort to perform pcap readback or inline operation,
etc.  The DAQ library may be useful for other packet processing
applications and the modular nature allows you to build new modules for
other platforms.

The DAQ library is provided as an external package on snort.org.  There are
a few additional modules provided with Snort++.  This section summarizes
the important things you need to know to use these DAQ modules.  There are
also 3rd DAQ modules available.


=== Building the DAQ Library and DAQ Modules

The DAQ is bundled with Snort but must be built first using these steps:

    ./configure
    make
    sudo make install

This will build and install both static and dynamic DAQ modules.

Note that pcap >= 1.0.0 is required.  pcap 1.1.1 is available at the time
of this writing and is recommended.

Also, libdnet is required for IPQ and NFQ DAQs.  If you get a relocation error
trying to build those DAQs, you may need to reinstall libdnet and configure it
with something like this:

    ./configure "CFLAGS=-fPIC -g -O2"

You may also experience problems trying to find the dynamic dnet library
because it isn't always named properly.  Try creating a link to the shared
library (identified by its .x or .x.y etc. extension) with the same name but
with ".so" inserted as follows:

    $ ln -s libdnet.1.1 libdnet.so.1.1
    $ ldconfig -Rv /usr/local/lib 2>&1 | grep dnet
      Adding /usr/local/lib/libdnet.so.1.1

Alternatively, you should be able to fix both issues as follows:

    libtoolize --copy --force
    aclocal -I config
    autoheader
    autoconf
    automake --foreign

When the DAQ library is built, both static and dynamic flavors will be
generated.  The various DAQ modules will be built if the requisite headers and
libraries are available.  You can disable individual modules, etc. with options
to configure.  For the complete list of configure options, run:

    ./configure --help


=== PCAP Module

pcap is the default DAQ.  If snort is run w/o any DAQ arguments, it will
operate as it always did using this module.  These are equivalent:

    ./snort -i <device>
    ./snort -r <file>

    ./snort --daq pcap --daq-mode passive -i <device>
    ./snort --daq pcap --daq-mode read-file -r <file>

You can specify the buffer size pcap uses with:

    ./snort --daq pcap --daq-var buffer_size=<#bytes>

* The pcap DAQ does not count filtered packets. *


=== AFPACKET Module

afpacket functions similar to the pcap DAQ but with better performance:

    ./snort --daq afpacket -i <device>
            [--daq-var buffer_size_mb=<#MB>]
            [--daq-var debug]

If you want to run afpacket in inline mode, you must craft the device string as
one or more interface pairs, where each member of a pair is separated by a
single colon and each pair is separated by a double colon like this:

    eth0:eth1

or this:

    eth0:eth1::eth2:eth3

By default, the afpacket DAQ allocates 128MB for packet memory.  You can change
this with:

    --daq-var buffer_size_mb=<#MB>

Note that the total allocated is actually higher, here's why.  Assuming the
default packet memory with a snaplen of 1518, the numbers break down like this:

* The frame size is 1518 (snaplen) + the size of the AFPacket header (66
  bytes) = 1584 bytes.

* The number of frames is 128 MB / 1518 = 84733.

* The smallest block size that can fit at least one frame is  4 KB = 4096 bytes
  @ 2 frames per block.

* As a result, we need 84733 / 2 = 42366 blocks.

* Actual memory allocated is 42366 * 4 KB = 165.5 MB.

NOTE: Linux kernel version 2.6.31 or higher is required for the AFPacket DAQ
module due to its dependency on both TPACKET v2 and PACKET_TX_RING support.


=== NFQ Module

NFQ is the new and improved way to process iptables packets:

    ./snort --daq nfq \
        [--daq-var device=<dev>] \
        [--daq-var proto=<proto>] \
        [--daq-var queue=<qid>]

    <dev> ::= ip | eth0, etc; default is IP injection
    <proto> ::= ip4 | ip6 |; default is ip4
    <qid> ::= 0..65535; default is 0

This module can not run unprivileged so ./snort -u -g will produce a warning
and won't change user or group.

Notes on iptables are given below.


=== IPQ Module

IPQ is the old way to process iptables packets.  It replaces the inline version
available in pre-2.9 versions built with this:

    ./configure --enable-inline

Note that layer 2 resets are not supported with the IPQ DAQ:

    config layer2resets[: <mac>]

Start the IPQ DAQ as follows:

    ./snort --daq ipq \
        [--daq-var device=<dev>] \
        [--daq-var proto=<proto>] \

    <dev> ::= ip | eth0, etc; default is IP injection
    <proto> ::= ip4 | ip6; default is ip4

This module can not run unprivileged so ./snort -u -g will produce a warning
and won't change user or group.

Notes on iptables are given below.


=== IPFW Module

IPFW is available for BSD systems.  It replaces the inline version available in
pre-2.9 versions built with this:

    ./configure --enable-ipfw

This command line argument is no longer supported:

    ./snort -J <port#>

Instead, start Snort like this:

    ./snort --daq ipfw [--daq-var port=<port>]

    <port> ::= 1..65535; default is 8000

* IPFW only supports ip4 traffic.

Notes on FreeBSD and OpenBSD are given below.


=== Dump Module

The dump DAQ allows you to test the various inline mode features available in
2.9 Snort like injection and normalization.

    ./snort -i <device> --daq dump
    ./snort -r <pcap> --daq dump

By default a file named inline-out.pcap will be created containing all packets
that passed through or were generated by snort.  You can optionally specify a
different name.

    ./snort --daq dump --daq-var file=<name>

dump uses the pcap daq for packet acquisition.  It therefore does not count
filtered packets (a pcap limitation).

Note that the dump DAQ inline mode is not an actual inline mode.  Furthermore,
you will probably want to have the pcap DAQ acquire in another mode like this:

    ./snort -r <pcap> -Q --daq dump --daq-var load-mode=read-file
    ./snort -i <device> -Q --daq dump --daq-var load-mode=passive


=== Netmap Module

The netmap project is a framework for very high speed packet I/O.  It is
available on both FreeBSD and Linux with varying amounts of preparatory
setup required.  Specific notes for each follow.

    ./snort --daq netmap -i <device>
            [--daq-var debug]

If you want to run netmap in inline mode, you must craft the device string as
one or more interface pairs, where each member of a pair is separated by a
single colon and each pair is separated by a double colon like this:

    em1:em2

or this:

    em1:em2::em3:em4

Inline operation performs Layer 2 forwarding with no MAC filtering, akin to the
AFPacket module's behavior.  All packets received on one interface in an inline
pair will be forwarded out the other interface unless dropped by the reader and
vice versa.

IMPORTANT: The interfaces will need to be up and in promiscuous mode in order to
function ('ifconfig em1 up promisc').  The DAQ module does not currently do
either of these configuration steps for itself.

==== FreeBSD

In FreeBSD 10.0, netmap has been integrated into the core OS.  In order to use
it, you must recompile your kernel with the line

    device netmap

added to your kernel config.

==== Linux

You will need to download the netmap source code from the project's repository:

    https://code.google.com/p/netmap/

Follow the instructions on the project's homepage for compiling and installing
the code:

    http://info.iet.unipi.it/~luigi/netmap/

It will involve a standalone kernel module (netmap_lin) as well as patching and
rebuilding the kernel module used to drive your network adapters. The following
drivers are supported under Linux at the time of writing (June 2014):

    e1000
    e1000e
    forcedeth
    igb
    ixgbe
    r8169
    virtio

TODO:

* Support for attaching to only a single ring (queue) on a network adapter.

* Support for VALE and netmap pipes.


=== Notes on iptables

These notes are just a quick reminder that you need to set up iptables to use
the IPQ or NFQ DAQs.  Doing so may cause problems with your network so tread
carefully.  The examples below are intentionally incomplete so please read the
related documentation first.

Here is a blog post by Marty for historical reference:

    http://archives.neohapsis.com/archives/snort/2000-11/0394.html

You can check this out for queue sizing tips:

    http://www.inliniac.net/blog/2008/01/23/improving-snort_inlines-nfq-performance.html

You might find useful IPQ info here:

    http://snort-inline.sourceforge.net/

Use this to examine your iptables:

    sudo /sbin/iptables -L

Use something like this to set up NFQ:

    sudo /sbin/iptables
        -I <table> [<protocol stuff>] [<state stuff>]
        -j NFQUEUE --queue-num 1

Use something like this to set up IPQ:

    sudo iptables -I FORWARD -j QUEUE

Use something like this to "disconnect" snort:

    sudo /sbin/iptables -D <table> <rule pos>

Be sure to start Snort prior to routing packets through NFQ with iptables.
Such packets will be dropped until Snort is started.

The queue-num is the number you must give Snort.

If you are running on a system with both NFQ and IPQ support, you may
experience some start-up failures of the sort:

The solution seems to be to remove both modules from the kernel like this:

    modprobe -r nfnetlink_queue
    modprobe -r ip_queue

and then install the module you want:

    modprobe ip_queue

or:

    modprobe nfnetlink_queue

These DAQs should be run with a snaplen of 65535 since the kernel defrags the
packets before queuing.  Also, no need to configure frag3.


=== Notes on FreeBSD::IPFW

Check the online manual at:

    http://www.freebsd.org/doc/handbook/firewalls-ipfw.html.

Here is a brief example to divert icmp packets to Snort at port 8000:

To enable support for divert sockets, place the following lines in the
kernel configuration file:

    options IPFIREWALL
    options IPDIVERT

(The file in this case was: /usr/src/sys/i386/conf/GENERIC; which is platform
dependent.)

You may need to also set these to use the loadable kernel modules:

    /etc/rc.conf:
    firewall_enable="YES"

    /boot/loader.conf:
    ipfw_load="YES"
    ipdivert_load="YES"

    $ dmesg | grep ipfw
    ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based
    forwarding disabled, default to deny, logging disabled

    $ kldload -v ipdivert
    Loaded ipdivert, id=4

    $ ipfw add 75 divert 8000 icmp from any to any
    00075 divert 8000 icmp from any to any

    $ ipfw list
    ...
    00075 divert 8000 icmp from any to any
    00080 allow icmp from any to any
    ...

* Note that on FreeBSD, divert sockets don't work with bridges!

Please refer to the following articles for more information:

* https://forums.snort.org/forums/support/topics/snort-inline-on-freebsd-ipfw

* http://freebsd.rogness.net/snort_inline/

NAT gateway can be used with divert sockets if the network environment is
conducive to using NAT.

The steps to set up NAT with ipfw are as follows:

1. Set up NAT with two interface em0 and em1 by adding the following to
/etc/rc.conf.  Here em0 is connected to external network and em1 to
host-only LAN.

    gateway_enable="YES"
    natd_program="/sbin/natd"   # path to natd
    natd_enable="YES"           # Enable natd (if firewall_enable == YES)
    natd_interface="em0"       # Public interface or IP Address
    natd_flags="-dynamic"       # Additional flags
    defaultrouter=""
    ifconfig_em0="DHCP"
    ifconfig_em1="inet 192.168.1.2 netmask 255.255.255.0"
    firewall_enable="YES"
    firewall_script="/etc/rc.firewall"
    firewall_type="simple"

2. Add the following divert rules to divert packets to Snort above and
below the NAT rule in the "Simple" section of /etc/rc.firewall.

   ...
   # Inspect outbound packets (those arriving on "inside" interface)
   # before NAT translation.
   ${fwcmd} add divert 8000 all from any to any in via ${iif}
   case ${natd_enable} in
   [Yy][Ee][Ss])
       if [ -n "${natd_interface}" ]; then
           ${fwcmd} add divert natd all from any to any via ${natd_interface}
       fi
       ;;
   esac
   ...
   # Inspect inbound packets (those arriving on "outside" interface)
   # after NAT translation that aren't blocked for other reasons,
   # after the TCP "established" rule.
   ${fwcmd} add divert 8000 all from any to any in via ${oif}


=== Notes on OpenBSD::IPFW

OpenBSD supports divert sockets as of 4.7, so we use the ipfw DAQ.

Here is one way to set things up:

1.  Configure the system to forward packets:

    $ sysctl net.inet.ip.forwarding=1
    $ sysctl net.inet6.ip6.forwarding=1

    (You can also put that in /etc/sysctl.conf to enable on boot.)

2.  Set up interfaces

    $ dhclient vic1
    $ dhclient vic2

3.  Set up packet filter rules:

    $ echo "pass out on vic1 divert-packet port 9000 keep-state" > rules.txt
    $ echo "pass out on vic2 divert-packet port 9000 keep-state" >> rules.txt

    $ pfctl -v -f rules.txt

4.  Analyze packets diverted to port 9000:

    $ ./snort --daq ipfw --daq-var port=9000

* Note that on OpenBSD, divert sockets don't work with bridges!


=== Socket Module

The socket module provides provides a stream socket server that will accept
up to 2 simultaneous connections and bridge them together while also
passing data to Snort++ for inspection.  The first connection accepted is
considered the client and the second connection accepted is considered the
server.  If there is only one connection, stream data can't be forwarded
but it is still inspected.

Each read from a socket of up to snaplen bytes is passed as a packet to
Snort++ along with a DAQ_SktHdr_t pointer in DAQ_PktHdr_t->priv_ptr.
DAQ_SktHdr_t conveys IP4 address, ports, protocol, and direction.  Socket
packets can be configured to be TCP or UDP.  The socket DAQ can be operated
in inline mode and is able to block packets.

The socket DAQ uses DLT_SOCKET and requires that Snort++ load the socket
codec which is included in the extra package.

To use the socket DAQ, start Snort++ like this:

    ./snort --plugin-path /path/to/lib/snort_extra \
        --daq socket [--daq-var port=<port>] [--daq-var proto=<proto>] [-Q]

    <port> ::= 1..65535; default is 8000
    <proto> ::= tcp | udp

* This module only supports ip4 traffic.

* This module is only supported by Snort++.  It is not compatible with
  Snort.

* This module is primarily for development and test.


=== File Module

The file module provides the ability to process files directly w/o having
to extract them from pcaps.  Use the file module with Snort's stream_file
to get file type identification and signature services.  The usual IPS
detection and logging etc. is available too.

You can process all the files in a directory recursively using 8 threads
with these Snort options:

    --pcap-dir path -z 8

* This module is only supported by Snort++.  It is not compatible with
  Snort.

* This module is primarily for development and test.


=== Hext Module

The hext module generates packets suitable for processing by Snort from
hex/plain text.  Raw packets include full headers and are processed
normally.  Otherwise the packets contain only payload and are accompanied
with flow information (4-tuple) suitable for processing by stream_user.

The first character of the line determines it's purpose:

    '$' command
    '#' comment
    '"' quoted string packet data
    'x' hex packet data
    ' ' empty line separates packets

The available commands are:

    $client <ip4> <port>
    $server <ip4> <port>

    $packet -> client
    $packet -> server

    $packet <addr> <port> -> <addr> <port>

Client and server are determined as follows.  $packet -> client indicates
to the client (from server) and $packet -> server indicates a packet to the
server (from client).  $packet followed by a 4-tuple uses the heuristic
that the client is the side with the lower port number.

The default client and server are 192.168.1.1 12345 and 10.1.2.3 80
respectively.  $packet commands with a 4-tuple do not change client and
server set with the other $packet commands.

$packet commands should be followed by packet data, which may contain any
combination of hex and strings.  Data for a packet ends with the next
command or a blank line.  Data after a blank line will start another packet
with the same tuple as the prior one.

Strings may contain the following escape sequences:

    \r = 0x0D = carriage return
    \n = 0x0A = new line
    \t = 0x09 = tab
    \\ = 0x5C = \

Format your input carefully; there is minimal error checking and little
tolerance for arbitrary whitespace.  You can use Snort's -L hext option to
generate hext input from a pcap.

* This module only supports ip4 traffic.

* This module is only supported by Snort++.  It is not compatible with
  Snort.

* This module is primarily for development and test.

The hext DAQ also supports a raw mode which is activated by setting the
data link type.  For example, you can input full ethernet packets with
--daq-var dlt=1 (Data link types are defined in the DAQ include
sfbpf_dlt.h.)  Combine that with the hext logger in raw mode for a quick
(and dirty) way to edit pcaps.  With --lua "log_hext = { raw = true }", the
hext logger will dump the full packet in a way that can be read by the hext
DAQ in raw mode.  Here is an example:

    # 3 [96]

    x02 09 08 07 06 05 02 01 02 03 04 05 08 00 45 00 00 52 00 03  # ..............E..R..
    x00 00 40 06 5C 90 0A 01 02 03 0A 09 08 07 BD EC 00 50 00 00  # ..@.\............P..
    x00 02 00 00 00 02 50 10 20 00 8A E1 00 00 47 45 54 20 2F 74  # ......P.  .....GET /t
    x72 69 67 67 65 72 2F 31 20 48 54 54 50 2F 31 2E 31 0D 0A 48  # rigger/1 HTTP/1.1..H
    x6F 73 74 3A 20 6C 6F 63 61 6C 68 6F 73 74 0D 0A              # ost: localhost..

A comment indicating packet number and size precedes each packet dump.
Note that the commands are not applicable in raw mode and have no effect.