Go to file

Dominik Pataky 85e6af4bd2 Add buffering of exports with unknown template Until now, exports which were received, but their template was not known, resulted in KeyError exceptions due to a missing key in the template dict. With this release, these exports are buffered until a template export updates this dict, and all buffered exports are again examined. Release v0.7.0 Fixes #4 Fixes #5		2019-03-31 20:51:34 +02:00
src/netflow	Add buffering of exports with unknown template	2019-03-31 20:51:34 +02:00
.gitignore	Add JSON export and analyzing example script	2017-10-28 19:00:18 +02:00
analyze_json.py	Add buffering of exports with unknown template	2019-03-31 20:51:34 +02:00
LICENSE	Bump to 0.6; expand analyzer	2017-10-29 11:53:32 +01:00
main.py	Add buffering of exports with unknown template	2019-03-31 20:51:34 +02:00
README.md	Fix missing IP_PROTOCOL_VERSION field in analyzer	2018-02-20 12:09:54 +01:00
setup.py	Add buffering of exports with unknown template	2019-03-31 20:51:34 +02:00

README.md

Python NetFlow v9 parser and UDP collector

This script is able to parse incoming UDP NetFlow packets of NetFlow version 9.

Version 9 is the first NetFlow version using templates. Templates make dynamically sized and configured NetFlow data flowsets possible, which makes the collector's job harder.

Licensed under MIT License. See LICENSE.

Using the collector and analyzer

In this repo you also find main.py and analyze_json.py.

To start an example collector run python3 main.py -p 9000 -D. This will run a collector at port 9000 in debug mode. Point your flow exporter to this port on your host and after some time the first ExportPackets should appear (the flows need to expire first).

After you collected some data, main.py exports them into JSON files, simply named <timestamp>.json.

To analyze the saved traffic, run analyze_json.py <json file>. In my example script this will look like the following, with resolved hostnames and services, transfered bytes and connection duration:

2017-10-28 23:17.01: SSH     | 4.25M    | 15:27 min | localmachine-2 (<IPv4>) to localmachine-1 (<IPv4>)
2017-10-28 23:17.01: SSH     | 4.29M    | 16:22 min | remotemachine (<IPv4>) to localmachine-2 (<IPv4>)
2017-10-28 23:19.01: HTTP    | 22.79M   | 47:32 min | uwstream3.somafm.com (173.239.76.148) to localmachine-1 (<IPv4>)
2017-10-28 23:22.01: HTTPS   | 1.21M    | 3 sec     | fra16s12-in-x0e.1e100.net (2a00:1450:4001:818::200e) to localmachine-1 (<IPv6>)
2017-10-28 23:23.01: SSH     | 93.79M   | 21 sec    | remotemachine (<IPv4>) to localmachine-2 (<IPv4>)
2017-10-28 23:51.01: SSH     | 14.08M   | 1:23.09 hours | remotemachine (<IPv4>) to localmachine-2 (<IPv4>)

Feel free to customize the analyzing script, e.g. make it print some nice graphs or calculate broader statistics.

Resources

Development environment

I have specifically written this script in combination with NetFlow exports from softflowd v0.9.9 - it should work with every correct NetFlow v9 implementation though.