Provide parse_packet as API; fix parse_packet input handling; README
To get closer to a stable package, netflow now offers the parse_packet function in its top-level __init__ file. This function was also enhanced to handle multiple input formats (str, bytes, hex bytes). Updated README accordingly.
This commit is contained in:
parent
7ae179cb33
commit
d4d6d59713
42
README.md
42
README.md
|
@ -1,13 +1,41 @@
|
||||||
# Python NetFlow library
|
# Python NetFlow library
|
||||||
This package contains libraries and tools for **NetFlow versions 1, 5 and 9**.
|
This package contains libraries and tools for **NetFlow versions 1, 5 and 9**.
|
||||||
|
|
||||||
Version 9 is the first NetFlow version using templates. Templates make dynamically sized and configured NetFlow data flowsets possible, which makes the collector's job harder. By importing `netflow.v1`, `netflow.v5` or `netflow.v9` you have direct access to the respective parsing objects, but at the beginning you probably will have more success by running the reference collector.
|
Version 9 is the first NetFlow version using templates. Templates make dynamically sized and configured NetFlow data flowsets possible, which makes the collector's job harder. The library provides the `netflow.parse_packet()` function as the main API point (see below). By importing `netflow.v1`, `netflow.v5` or `netflow.v9` you have direct access to the respective parsing objects, but at the beginning you probably will have more success by running the reference collector (example below) and look into its code.
|
||||||
|
|
||||||
Copyright 2016-2020 Dominik Pataky <dev@bitkeks.eu>
|
Copyright 2016-2020 Dominik Pataky <dev@bitkeks.eu>
|
||||||
|
|
||||||
Licensed under MIT License. See LICENSE.
|
Licensed under MIT License. See LICENSE.
|
||||||
|
|
||||||
|
|
||||||
|
## Using the library
|
||||||
|
If you chose to use the classes provided by this library directly, here's an example for a NetFlow v5 export packet:
|
||||||
|
|
||||||
|
1. Create a collector which listens for exported packets on some UDP port. It should then receive UDP packets from exporters.
|
||||||
|
2. Inside the UDP packets, the NetFlow payload is contained. For NetFlow v5 it should begin with bytes `0005` for example.
|
||||||
|
3. Call the `netflow.parse_packet()` function with the payload as first argument (takes string, bytes string and hex'd bytes).
|
||||||
|
|
||||||
|
Example UDP collector server (receiving exports on port 2055):
|
||||||
|
```python
|
||||||
|
import netflow
|
||||||
|
import socket
|
||||||
|
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||||
|
sock.bind(("0.0.0.0", 2055))
|
||||||
|
payload, client = sock.recvfrom(4096) # experimental, tested with 1464 bytes
|
||||||
|
p = netflow.parse_packet(payload) # Test result: <ExportPacket v5 with 30 records>
|
||||||
|
print(p.header.version) # Test result: 5
|
||||||
|
```
|
||||||
|
|
||||||
|
Or from hex dump:
|
||||||
|
```python
|
||||||
|
import netflow
|
||||||
|
p = netflow.parse_packet("00050003000379a35e80c58622a...") # see test_netflow.py
|
||||||
|
assert p.header.version == 5 # NetFlow v5 packet
|
||||||
|
assert p.flows[0].PROTO == 1 # ICMP flow
|
||||||
|
```
|
||||||
|
|
||||||
|
In NetFlow v9, templates are used instead of a fixed set of fields (like PROTO). See `collector.py` on how to handle these.
|
||||||
|
|
||||||
## Using the collector and analyzer
|
## Using the collector and analyzer
|
||||||
Since v0.9.0 the `netflow` library also includes reference implementations of a collector and an analyzer as CLI tools.
|
Since v0.9.0 the `netflow` library also includes reference implementations of a collector and an analyzer as CLI tools.
|
||||||
These can be used on the CLI with `python3 -m netflow.collector` and `python3 -m netflow.analyzer`. Use the `-h` flag to receive the respective help output with all provided CLI flags.
|
These can be used on the CLI with `python3 -m netflow.collector` and `python3 -m netflow.analyzer`. Use the `-h` flag to receive the respective help output with all provided CLI flags.
|
||||||
|
@ -16,12 +44,12 @@ Example: to start the collector run `python3 -m netflow.collector -p 9000 -D`. T
|
||||||
|
|
||||||
To analyze the saved traffic, run `python3 -m netflow.analyzer -f <gzip file>`. The output will look similar to the following snippet, with resolved hostnames and services, transferred bytes and connection duration:
|
To analyze the saved traffic, run `python3 -m netflow.analyzer -f <gzip file>`. The output will look similar to the following snippet, with resolved hostnames and services, transferred bytes and connection duration:
|
||||||
|
|
||||||
2017-10-28 23:17.01: SSH | 4.25M | 15:27 min | localmachine-2 (<IPv4>) to localmachine-1 (<IPv4>)
|
2017-10-28 23:17.01: SSH | 4.25M | 15:27 min | local-2 (<IPv4>) to local-1 (<IPv4>)
|
||||||
2017-10-28 23:17.01: SSH | 4.29M | 16:22 min | remotemachine (<IPv4>) to localmachine-2 (<IPv4>)
|
2017-10-28 23:17.01: SSH | 4.29M | 16:22 min | remote-1 (<IPv4>) to local-2 (<IPv4>)
|
||||||
2017-10-28 23:19.01: HTTP | 22.79M | 47:32 min | uwstream3.somafm.com (173.239.76.148) to localmachine-1 (<IPv4>)
|
2017-10-28 23:19.01: HTTP | 22.79M | 47:32 min | uwstream3.somafm.com (173...) to local-1 (<IPv4>)
|
||||||
2017-10-28 23:22.01: HTTPS | 1.21M | 3 sec | fra16s12-in-x0e.1e100.net (2a00:1450:4001:818::200e) to localmachine-1 (<IPv6>)
|
2017-10-28 23:22.01: HTTPS | 1.21M | 3 sec | fra16s12-in-x0e.1e100.net (2a00:..) to local-1 (<IPv6>)
|
||||||
2017-10-28 23:23.01: SSH | 93.79M | 21 sec | remotemachine (<IPv4>) to localmachine-2 (<IPv4>)
|
2017-10-28 23:23.01: SSH | 93.79M | 21 sec | remote-1 (<IPv4>) to local-2 (<IPv4>)
|
||||||
2017-10-28 23:51.01: SSH | 14.08M | 1:23.09 hours | remotemachine (<IPv4>) to localmachine-2 (<IPv4>)
|
2017-10-28 23:51.01: SSH | 14.08M | 1:23.09 hours | remote-1 (<IPv4>) to local-2 (<IPv4>)
|
||||||
|
|
||||||
**Please note that the collector and analyzer are experimental reference implementations. Do not rely on them in production monitoring use cases!** In any case I recommend looking into the `netflow/collector.py` and `netflow/analyzer.py` scripts for customization. Feel free to use the code and extend it in your own tool set - that's what the MIT license is for!
|
**Please note that the collector and analyzer are experimental reference implementations. Do not rely on them in production monitoring use cases!** In any case I recommend looking into the `netflow/collector.py` and `netflow/analyzer.py` scripts for customization. Feel free to use the code and extend it in your own tool set - that's what the MIT license is for!
|
||||||
|
|
||||||
|
|
|
@ -6,3 +6,5 @@ This file belongs to https://github.com/bitkeks/python-netflow-v9-softflowd.
|
||||||
Copyright 2017-2020 Dominik Pataky <dev@bitkeks.eu>
|
Copyright 2017-2020 Dominik Pataky <dev@bitkeks.eu>
|
||||||
Licensed under MIT License. See LICENSE.
|
Licensed under MIT License. See LICENSE.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
from .utils import parse_packet
|
||||||
|
|
|
@ -29,7 +29,23 @@ def get_netflow_version(data):
|
||||||
return struct.unpack('!H', data[:2])[0]
|
return struct.unpack('!H', data[:2])[0]
|
||||||
|
|
||||||
|
|
||||||
def parse_packet(data, templates):
|
def parse_packet(data, templates=None):
|
||||||
|
if templates is None: # compatibility for v1 and v5
|
||||||
|
templates = {}
|
||||||
|
|
||||||
|
if type(data) == str:
|
||||||
|
# hex dump as string
|
||||||
|
data = bytes.fromhex(data)
|
||||||
|
elif type(data) == bytes:
|
||||||
|
# check representation based on utf-8 decoding result
|
||||||
|
try:
|
||||||
|
# hex dump as bytes, but not hex
|
||||||
|
dec = data.decode()
|
||||||
|
data = bytes.fromhex(dec)
|
||||||
|
except UnicodeDecodeError:
|
||||||
|
# use data as given, assuming hex-formatted bytes
|
||||||
|
pass
|
||||||
|
|
||||||
version = get_netflow_version(data)
|
version = get_netflow_version(data)
|
||||||
if version == 1:
|
if version == 1:
|
||||||
return V1ExportPacket(data)
|
return V1ExportPacket(data)
|
||||||
|
|
2
setup.py
2
setup.py
|
@ -16,7 +16,7 @@ setup(
|
||||||
url='https://github.com/bitkeks/python-netflow-v9-softflowd',
|
url='https://github.com/bitkeks/python-netflow-v9-softflowd',
|
||||||
packages=["netflow"],
|
packages=["netflow"],
|
||||||
license='MIT',
|
license='MIT',
|
||||||
python_requires='>=3',
|
python_requires='>=3.5',
|
||||||
keywords='netflow collector parser',
|
keywords='netflow collector parser',
|
||||||
classifiers=[
|
classifiers=[
|
||||||
"Programming Language :: Python :: 3",
|
"Programming Language :: Python :: 3",
|
||||||
|
|
Loading…
Reference in a new issue