Skip to content

Commit

Permalink
Added info to README and licence banners to the remaining files
Browse files Browse the repository at this point in the history
  • Loading branch information
Lennart Elsen committed Sep 30, 2014
1 parent 952fec4 commit d8a62cf
Show file tree
Hide file tree
Showing 11 changed files with 193 additions and 13 deletions.
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
#
# Makefile: Makefile to build the goProbe traffic monitor
#
# Written by Fabian Kohn [email protected]
# and Lennart Elsen [email protected], August 2014
# Written by Lennart Elsen
# and Fabian Kohn, August 2014
# Copyright (c) 2014 Open Systems AG, Switzerland
# All Rights Reserved.
#
Expand Down
40 changes: 36 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@ goProbe

This package comprises:

* goProbe - A lightweight, concurrent, network packet aggregator
* goDB - A small, columnar database
* goQuery - A query front-end used to read out data acquired by goProbe and stored by goDB
* goProbe - A lightweight, concurrent, network packet aggregator
* goDB - A small, high-performance, columnar database
* goQuery - Query front-end used to read out data acquired by goProbe and stored by goDB
* goConvert - Helper binary to convert goProbe-flow data stored in `csv` files

As the name suggests, all components are written in Google [go](https://golang.org/).

Expand Down Expand Up @@ -49,6 +50,8 @@ The capturing probe can be run as a daemon via
/etc/init.d/goprobe.init {start|stop|status|restart|force-reload}
```

By default, the interface `eth0` is specified. If you want to perform capturing on other interfaces, change the respective line in `goprobe.init` (the variable `DAEMON_ARGS` stores the interfaces).

goDB
--------------------------
The flow records are stored block-wise on a five minute basis in their respective attribute files. The database is partitioned on a per day basis, which means that for each day, a new folder is created which holds the attribute files for all flow records written throughout the day.
Expand Down Expand Up @@ -96,12 +99,25 @@ Query produced 779 hits and took 33.66236ms
Overall packets: 79.07 M , Overall data volume: 32.44 GB
```

### Converting data

If you use `goConvert`, you need to make sure that the data which you are importing is _temporally ordered_ and provides a column which stores UNIX timestamps. An example `csv` file may look as follows:

```
# HEADER: bytes_rcvd,bytes_sent,dip,dport,l7_proto,packets_rcvd,packets_sent,proto,sip,tstamp
...
40,72,172.23.34.171,8080,158,1,1,6,10.11.72.28,1392997558
40,72,172.23.34.171,49362,158,1,1,6,10.11.72.28,1392999058
...
```
You _must_ abide by this structure, otherwise the conversion will fail.
Installation
------------

This package was designed to work out of the box. Thus, you do not even need the `go` environment. All of the dependencies are downloaded during package configuration. To install the package, go to the directory into which you cloned this repository and run the following commands (as `root`):
This package was designed to work out of the box. Thus, you do not even need the `go` environment. All of the dependencies are downloaded during package configuration. To install the package, go to the directory into which you cloned this repository and run the following commands:

```
sudo -s
make all
```

Expand All @@ -112,6 +128,22 @@ Above command runs the following targets:
* `make compile`: compiles dependencies, goProbe and goQuery
* `make install`: set up package as a binary tree. The binaries and used libraries are placed in `/usr/local/goProbe` per default. The init script can be found under `/etc/init.d/goprobe.init`. It is also possible to install a cronjob used to clean up outdated database entries. It is not installed by default. Uncomment the line in the Makefile if you need this feature. The cronjob can be found in `/etc/cron.d/goprobe.cron`

By default, `goConvert` is not compiled. If you wish to do so, add the following line to the `install` target in the Makefile:

```
go build -a -o goConvert $(PWD)/addon/gocode/src/OSAG/convert/DBConvert.go
```
The binary will reside in the directory specified in the above command.

### Supported Operating Systems

goProbe is currently set up to run on Linux based systems. Tested versions include:

* Ubuntu 14.04
* Debian 7

Support for Mac OS X will follow eventually.

Authors & Contributors
----------------------

Expand Down
16 changes: 16 additions & 0 deletions addon/dpi/Makefile
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
# This code has been developed by Open Systems AG
#
# goProbe is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# goProbe is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with goProbe; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

PROGRAM = ProtoId
SERIALIZER = serialize_prot_list

Expand Down
20 changes: 19 additions & 1 deletion addon/dpi/ProtoId.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,29 @@
// approrpiate functions and exposing the API to C (which, in turn, can be used
// by Google Go).
//
// Written by Lennart Elsen [email protected], July 2014
// Written by Lennart Elsen
// and Fabian Kohn, July 2014
// Copyright (c) 2014 Open Systems AG, Switzerland
// All Rights Reserved.
//
/////////////////////////////////////////////////////////////////////////////////
/* This code has been developed by Open Systems AG
*
* goProbe is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* goProbe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with goProbe; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/

#include "ProtoId.h"

#include <stdio.h>
Expand Down
20 changes: 19 additions & 1 deletion addon/dpi/ProtoId.h
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,29 @@
// approrpiate functions and exposing the API to C (which, in turn, can be used
// by Google Go).
//
// Written by Lennart Elsen [email protected], July 2014
// Written by Lennart Elsen
// and Fabian Kohn, July 2014
// Copyright (c) 2014 Open Systems AG, Switzerland
// All Rights Reserved.
//
/////////////////////////////////////////////////////////////////////////////////
/* This code has been developed by Open Systems AG
*
* goProbe is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* goProbe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with goProbe; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/

#ifndef ProtoId_h
#define ProtoId_h

Expand Down
15 changes: 15 additions & 0 deletions addon/dpi/serialize_ipprot_list.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,19 @@
#!/bin/bash
# This code has been developed by Open Systems AG
#
# goProbe is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# goProbe is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with goProbe; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

echo -e "\nvar IPProtocols = map[int] string {"
egrep -v "^#" /etc/protocols | awk '{if($2 != "" && $1 != "ip"){ print " " $2 ": \"" $3 "\","} }'
Expand Down
20 changes: 19 additions & 1 deletion addon/dpi/serialize_prot_list.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,29 @@
// Helper binary to extract the protocol-category mappings directly from
// libprotoident such that they are made available to goquery
//
// Written by Lennart Elsen [email protected], July 2014
// Written by Lennart Elsen
// and Fabian Kohn, July 2014
// Copyright (c) 2014 Open Systems AG, Switzerland
// All Rights Reserved.
//
/////////////////////////////////////////////////////////////////////////////////
/* This code has been developed by Open Systems AG
*
* goProbe is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* goProbe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with goProbe; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/

#include "libprotoident.h"

int main(){
Expand Down
16 changes: 16 additions & 0 deletions addon/gocode/src/OSAG/convert/DBConvert.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,22 @@
// All Rights Reserved.
//
/////////////////////////////////////////////////////////////////////////////////
/* This code has been developed by Open Systems AG
*
* goProbe is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* goProbe is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with goProbe; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/
package main

import(
Expand Down
16 changes: 16 additions & 0 deletions addon/goprobe.cron
Original file line number Diff line number Diff line change
@@ -1,2 +1,18 @@
# This code has been developed by Open Systems AG
#
# goProbe is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# goProbe is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with goProbe; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

# Run goprobe database cleanup (retention time 30 days)
3 3 * * * root test -e /usr/local/goProbe/data/db && CURRENT_EPOCH=`date +\%s` && /usr/local/goProbe/shared/goquery -x -clean `expr $CURRENT_EPOCH - 2592000` > /dev/null 2>&1
21 changes: 18 additions & 3 deletions addon/goprobe.init
Original file line number Diff line number Diff line change
@@ -1,11 +1,26 @@
#! /bin/bash
#!/bin/bash
# This code has been developed by Open Systems AG
#
# goProbe is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# goProbe is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with goProbe; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

### BEGIN INIT INFO
# Provides: fko-usbwde
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Logs temperature data
# Short-Description: Starts and stops the goProbe daemon
### END INIT INFO

# PATH should only include /usr/* if it runs after the mountnfs.sh script
Expand Down
18 changes: 17 additions & 1 deletion addon/goquery
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,27 @@
# Wrapper script used for querying the go database and nicing the process
# appropriately
#
# Written by Lennart Elsen [email protected], August 2014
# Written by Lennart Elsen
# and Fabian Kohn, August 2014
# Copyright (c) 2014 Open Systems AG, Switzerland
# All Rights Reserved.
#
################################################################################
# This code has been developed by Open Systems AG
#
# goProbe is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# goProbe is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with goProbe; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

#-------------------------------------------------------------------------------
# Global Constants
Expand Down

0 comments on commit d8a62cf

Please sign in to comment.