diff --git a/Makefile b/Makefile index 55c23ed..16df46f 100644 --- a/Makefile +++ b/Makefile @@ -2,8 +2,8 @@ # # Makefile: Makefile to build the goProbe traffic monitor # -# Written by Fabian Kohn fko@open.ch -# and Lennart Elsen lel@open.ch, August 2014 +# Written by Lennart Elsen +# and Fabian Kohn, August 2014 # Copyright (c) 2014 Open Systems AG, Switzerland # All Rights Reserved. # diff --git a/README.md b/README.md index 3d42278..a8771b6 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,10 @@ goProbe This package comprises: -* goProbe - A lightweight, concurrent, network packet aggregator -* goDB - A small, columnar database -* goQuery - A query front-end used to read out data acquired by goProbe and stored by goDB +* goProbe - A lightweight, concurrent, network packet aggregator +* goDB - A small, high-performance, columnar database +* goQuery - Query front-end used to read out data acquired by goProbe and stored by goDB +* goConvert - Helper binary to convert goProbe-flow data stored in `csv` files As the name suggests, all components are written in Google [go](https://golang.org/). @@ -49,6 +50,8 @@ The capturing probe can be run as a daemon via /etc/init.d/goprobe.init {start|stop|status|restart|force-reload} ``` +By default, the interface `eth0` is specified. If you want to perform capturing on other interfaces, change the respective line in `goprobe.init` (the variable `DAEMON_ARGS` stores the interfaces). + goDB -------------------------- The flow records are stored block-wise on a five minute basis in their respective attribute files. The database is partitioned on a per day basis, which means that for each day, a new folder is created which holds the attribute files for all flow records written throughout the day. @@ -96,12 +99,25 @@ Query produced 779 hits and took 33.66236ms Overall packets: 79.07 M , Overall data volume: 32.44 GB ``` +### Converting data + +If you use `goConvert`, you need to make sure that the data which you are importing is _temporally ordered_ and provides a column which stores UNIX timestamps. An example `csv` file may look as follows: + +``` +# HEADER: bytes_rcvd,bytes_sent,dip,dport,l7_proto,packets_rcvd,packets_sent,proto,sip,tstamp +... +40,72,172.23.34.171,8080,158,1,1,6,10.11.72.28,1392997558 +40,72,172.23.34.171,49362,158,1,1,6,10.11.72.28,1392999058 +... +``` +You _must_ abide by this structure, otherwise the conversion will fail. Installation ------------ -This package was designed to work out of the box. Thus, you do not even need the `go` environment. All of the dependencies are downloaded during package configuration. To install the package, go to the directory into which you cloned this repository and run the following commands (as `root`): +This package was designed to work out of the box. Thus, you do not even need the `go` environment. All of the dependencies are downloaded during package configuration. To install the package, go to the directory into which you cloned this repository and run the following commands: ``` +sudo -s make all ``` @@ -112,6 +128,22 @@ Above command runs the following targets: * `make compile`: compiles dependencies, goProbe and goQuery * `make install`: set up package as a binary tree. The binaries and used libraries are placed in `/usr/local/goProbe` per default. The init script can be found under `/etc/init.d/goprobe.init`. It is also possible to install a cronjob used to clean up outdated database entries. It is not installed by default. Uncomment the line in the Makefile if you need this feature. The cronjob can be found in `/etc/cron.d/goprobe.cron` +By default, `goConvert` is not compiled. If you wish to do so, add the following line to the `install` target in the Makefile: + +``` +go build -a -o goConvert $(PWD)/addon/gocode/src/OSAG/convert/DBConvert.go +``` +The binary will reside in the directory specified in the above command. + +### Supported Operating Systems + +goProbe is currently set up to run on Linux based systems. Tested versions include: + +* Ubuntu 14.04 +* Debian 7 + +Support for Mac OS X will follow eventually. + Authors & Contributors ---------------------- diff --git a/addon/dpi/Makefile b/addon/dpi/Makefile index 8563527..801ce7f 100644 --- a/addon/dpi/Makefile +++ b/addon/dpi/Makefile @@ -1,3 +1,19 @@ +# This code has been developed by Open Systems AG +# +# goProbe is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# goProbe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with goProbe; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + PROGRAM = ProtoId SERIALIZER = serialize_prot_list diff --git a/addon/dpi/ProtoId.cxx b/addon/dpi/ProtoId.cxx index fcab2eb..b94ba7c 100644 --- a/addon/dpi/ProtoId.cxx +++ b/addon/dpi/ProtoId.cxx @@ -6,11 +6,29 @@ // approrpiate functions and exposing the API to C (which, in turn, can be used // by Google Go). // -// Written by Lennart Elsen lel@open.ch, July 2014 +// Written by Lennart Elsen +// and Fabian Kohn, July 2014 // Copyright (c) 2014 Open Systems AG, Switzerland // All Rights Reserved. // ///////////////////////////////////////////////////////////////////////////////// +/* This code has been developed by Open Systems AG +* +* goProbe is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* goProbe is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with goProbe; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + #include "ProtoId.h" #include diff --git a/addon/dpi/ProtoId.h b/addon/dpi/ProtoId.h index c008e2a..9c87749 100644 --- a/addon/dpi/ProtoId.h +++ b/addon/dpi/ProtoId.h @@ -6,11 +6,29 @@ // approrpiate functions and exposing the API to C (which, in turn, can be used // by Google Go). // -// Written by Lennart Elsen lel@open.ch, July 2014 +// Written by Lennart Elsen +// and Fabian Kohn, July 2014 // Copyright (c) 2014 Open Systems AG, Switzerland // All Rights Reserved. // ///////////////////////////////////////////////////////////////////////////////// +/* This code has been developed by Open Systems AG +* +* goProbe is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* goProbe is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with goProbe; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + #ifndef ProtoId_h #define ProtoId_h diff --git a/addon/dpi/serialize_ipprot_list.sh b/addon/dpi/serialize_ipprot_list.sh index b1e7db8..eb1e477 100755 --- a/addon/dpi/serialize_ipprot_list.sh +++ b/addon/dpi/serialize_ipprot_list.sh @@ -1,4 +1,19 @@ #!/bin/bash +# This code has been developed by Open Systems AG +# +# goProbe is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# goProbe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with goProbe; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA echo -e "\nvar IPProtocols = map[int] string {" egrep -v "^#" /etc/protocols | awk '{if($2 != "" && $1 != "ip"){ print " " $2 ": \"" $3 "\","} }' diff --git a/addon/dpi/serialize_prot_list.cxx b/addon/dpi/serialize_prot_list.cxx index 6fa3a42..f9a7717 100644 --- a/addon/dpi/serialize_prot_list.cxx +++ b/addon/dpi/serialize_prot_list.cxx @@ -5,11 +5,29 @@ // Helper binary to extract the protocol-category mappings directly from // libprotoident such that they are made available to goquery // -// Written by Lennart Elsen lel@open.ch, July 2014 +// Written by Lennart Elsen +// and Fabian Kohn, July 2014 // Copyright (c) 2014 Open Systems AG, Switzerland // All Rights Reserved. // ///////////////////////////////////////////////////////////////////////////////// +/* This code has been developed by Open Systems AG + * + * goProbe is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * goProbe is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with goProbe; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + */ + #include "libprotoident.h" int main(){ diff --git a/addon/gocode/src/OSAG/convert/DBConvert.go b/addon/gocode/src/OSAG/convert/DBConvert.go index 064324b..06c40c6 100644 --- a/addon/gocode/src/OSAG/convert/DBConvert.go +++ b/addon/gocode/src/OSAG/convert/DBConvert.go @@ -11,6 +11,22 @@ // All Rights Reserved. // ///////////////////////////////////////////////////////////////////////////////// +/* This code has been developed by Open Systems AG + * + * goProbe is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * goProbe is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with goProbe; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ package main import( diff --git a/addon/goprobe.cron b/addon/goprobe.cron index 76c900b..b12f24c 100644 --- a/addon/goprobe.cron +++ b/addon/goprobe.cron @@ -1,2 +1,18 @@ +# This code has been developed by Open Systems AG +# +# goProbe is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# goProbe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with goProbe; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + # Run goprobe database cleanup (retention time 30 days) 3 3 * * * root test -e /usr/local/goProbe/data/db && CURRENT_EPOCH=`date +\%s` && /usr/local/goProbe/shared/goquery -x -clean `expr $CURRENT_EPOCH - 2592000` > /dev/null 2>&1 diff --git a/addon/goprobe.init b/addon/goprobe.init index 65318a7..89ec734 100644 --- a/addon/goprobe.init +++ b/addon/goprobe.init @@ -1,11 +1,26 @@ -#! /bin/bash +#!/bin/bash +# This code has been developed by Open Systems AG +# +# goProbe is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# goProbe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with goProbe; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + ### BEGIN INIT INFO -# Provides: fko-usbwde # Required-Start: $remote_fs $syslog # Required-Stop: $remote_fs $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 -# Short-Description: Logs temperature data +# Short-Description: Starts and stops the goProbe daemon ### END INIT INFO # PATH should only include /usr/* if it runs after the mountnfs.sh script diff --git a/addon/goquery b/addon/goquery index cce74b1..ac00e5d 100755 --- a/addon/goquery +++ b/addon/goquery @@ -6,11 +6,27 @@ # Wrapper script used for querying the go database and nicing the process # appropriately # -# Written by Lennart Elsen lel@open.ch, August 2014 +# Written by Lennart Elsen +# and Fabian Kohn, August 2014 # Copyright (c) 2014 Open Systems AG, Switzerland # All Rights Reserved. # ################################################################################ +# This code has been developed by Open Systems AG +# +# goProbe is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# goProbe is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with goProbe; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA #------------------------------------------------------------------------------- # Global Constants