Commit graph

10 commits

Author SHA1 Message Date
Dominik Pataky 6b9d20c8a6 Refactor storing data and writing to disk - using gzip and lines
In previous versions, collected flows (parsed data) were stored in
memory by the collector. In regular intervals, or at shutdown, this one
single dict was dumped as JSON onto disk.

With this commit, the behaviour is changed to line-based JSON dumps for
each flow, gzipped onto disk for storage efficiency. The analyze_json is
updated as well to handle the new gzipped files in the new format.

See the comments in main.py for more details.

Fixes #10
2019-11-03 12:02:05 +01:00
Dominik Pataky 9f16d246a5 Add v1, v5 to README; change fallback; add timeout parameter
Updated the README to reference NetFlow v1 and v5 as well.

The fallback(key, dict) method used an exception-based testing of the
keys existence. Switched to 'if x in'.

The NetFlowListener is based on threading.Thread, which uses the
'timeout' parameter in .join(). Added.
2019-10-31 17:55:48 +01:00
Dominik Pataky bfec3953e6 Bump version, fix small errors, decrease packet num in tests 2019-10-31 17:35:15 +01:00
Carey Metcalfe 96817f1f8d Add support for v1 and v5 NetFlow packets
Thanks to @alieissa for the initial v1 and v5 code
2019-10-16 23:46:32 -04:00
Carey Metcalfe 8e6d0c54e8 Allow analyze_json.py to accept input via stdin
This will make testing much cleaner in the future (no temp files needed)

Also increase performance by memoizing the hostname lookup
2019-10-16 23:44:19 -04:00
Dominik Pataky 85e6af4bd2 Add buffering of exports with unknown template
Until now, exports which were received, but their template was not known,
resulted in KeyError exceptions due to a missing key in the template dict.
With this release, these exports are buffered until a template export
updates this dict, and all buffered exports are again examined.

Release v0.7.0

Fixes #4
Fixes #5
2019-03-31 20:51:34 +02:00
Dominik Pataky 9395aafa71 Fix missing IP_PROTOCOL_VERSION field in analyzer
Checks for the key first and handles non-existence.
Update to Copyright notices.

Fixes #3
2018-02-20 12:09:54 +01:00
Dominik Pataky 691a3480fd Add duration to Connection 2017-10-29 19:38:33 +01:00
Dominik Pataky 6c267c8c77 Bump to 0.6; expand analyzer 2017-10-29 11:53:32 +01:00
Dominik Pataky 898d220a91 Add JSON export and analyzing example script 2017-10-28 19:00:18 +02:00