In IPFIX, template fields can be signed or unsigned, or even be pure
bytes or unicode string. This differentiation was extended in this
commit.
Additionally, the IPFIX_FIELD_TYPES dict mapping from int->str was
replaced by a more verbose version, which also includes the standardized
IANA data types. The class' methods provides access to the fixed data
set. This is then used in the IPFIXDataRecord parser.
Refs #25
The function send_recv_packets in tests stored all processed
ExportPackets by default in a list. Memory usage tests were therefore
based on this high amount of stored objects, since no instance of any
ExportPacket was deleted until exit.
With the new parameter store_packets the caller can define how many
packets should be stored during receiving, as to test multiple
scenarios.
Three such scenarios are implemented: don't store any packet, store
maximum of 500 at a time and store all packets. This comes much closer
to the real world scenario of the collector, which uses a "for export in
listener.get" loop, dumping any new ExportPacket to file immediatelly
and then deleting the object.
Yet, the case where all packets are stored must still be covered as
well, because the collector might not be the only implementation which
uses listener.get, so finding memory leaks should be covered.
A new test file is added which contains memory and CPU tests. For now,
only the memory usage tests work (threading!). They print out tables of
memory usage based on file path and on function. Additionally, they check
some basic measurements: if all packets were processed and if a
collection of version 9/10 called any functions in 10/9.
Refs #24
Analyzer test was missing imports.
IPFIX templates with 16 bytes fields were processed extra, since struct
does not natively support conversion to int. The new implementation
still handles it extra, but uses struct's "s" unpack format descriptor
now.
The previously introduced tests/lib.py contained the NetFlow v9 packets
and then the IPFIX packets, those were split and put into their
respective test files again. The lib now contains shared objects only.
For IPFIX tests were added. Two new packets were added, one with
templates and one without (again, real exports from softflowd).
Different cases are checked: no template, template and later template.
Fields of flows are also checked, especially IPv6 addresses.
Note: exports made with softflowd were created by softflowd 1.0.0,
compiled from https://github.com/irino/softflowd
The collector should catch both v9 and IPFIX template errors - syntax
error corrected. The v9 ExportPacket.templates attribute is now
@property and read-only.
The tests are now located in tests/. They are also split into multiple
files, beginning with test_netflow and test_analyzer. The tests for
IPFIX will be added to test_ipfix.
Python struct does not natively support 16 byte fields. But since IPFIX
uses fields of length 16 bytes for at least IPv6 addresses, they must be
processed in the IPFIX parser. This commit adds support for 16 byte
fields by handling them as special struct.unpack cases.
At differnt points in the tool set, NetFlow (v9) is set as the default
case. Now that IPFIX is on its way to be supported as well, adapt all
occurences where a differentiation must be done.
Second half of the IPFIX implementation now adds the support for data
records. The templates are also extracted, allowing the collector to use
them across exports.
The field types were extracted from the IANA assignment list at
https://www.iana.org/assignments/ipfix/ipfix-information-elements.csv
Please note that the IPFIX implementation was made from scratch and
differs from the NetFlow v9 implementation, as there was little
copy/paste.
Adds a new module, IPFIX. The collector already recognizes version 10 in
the header, meaning IPFIX. The parser is able to dissect the export
package and all sets with their headers.
Missing is the handling of the templates in the data sets - a feature
needed for the whole parsing process to complete.
The collector is able to parse templates in an export and then use these
templates to parse dataflows inside the same export packet. But the test
implementation was based on the assumption, that the templates always
arrive first in the packet. Now, a mixed order is also processed
successfully. Test included.
To get closer to a stable package, netflow now offers the parse_packet
function in its top-level __init__ file. This function was also enhanced
to handle multiple input formats (str, bytes, hex bytes).
Updated README accordingly.
The V1DataFlow and V5DataFlow classes used a verbose way of unpacking
the hex byte stream to the specific fields until now. With this commit,
both use a list of field names, one struct.unpack call and then a
mapping for-loop for each field.
Additionally the upper boundary of the passed data slice was added.
With the self.__dict__.update() call all fields are now also accessible
as direct attributes of the corresponding instance, e.g. flow.PROTO to
access flow.data["PROTO"]. This works for flows of all three versions.
The tests were adapted to reflect this new implementation.
Beginning with this commit, the reference implementations of the
collector and analyzer are now included in the package. They are
callable by running `python3 -m netflow.collector` or `.analyzer`, with
the same flags as before. Use `-h` to list them.
Additional fixes are contained in this commit as well, e.g. adding more
version prefixes and moving parts of code from __init__ to utils, to fix
circular imports.
The tests are now also parsing export packets for version 1 and 5.
Version 9 received an additional test, inspecting the data inside the
export.
All new packet hex dumps were created by using a Docker container with
alpine Linux, running a softflowd daemon inside and then pinging the
Docker host IP. After review with "softflowctl dump-flows" issueing
"softflowctl expire-all" sends the packets away to the collector (should
be an IP address outside of the Docker bridge). The export network
packets are then collected with Wireshark running in the host namespace,
capturing on the Docker bridge.
Bump version to v0.8.3
Resolves#13Resolves#14
Refs #18
Until now, every NetFlow version file used similar names for their
classes, e.g. "Header". These are now prefixed with their respective
version, e.g. "V1Header", to avoid confusion in imports etc.
The README and setup.py were adapted to the current state, preparing for
PyPI upload and package info.
In v9, the header received an additional .json property, which exports
the header as a dict to allow JSON serialization in the export file.
This export is used in main.py
This commit extends the analyzer script with two new flags:
* Adding --no-dns disables hostname DNS resolution, improving speed
* Adding --match-host <IP address> filters all flows not matching the IP
Additional small things were changed, the script is still work in
progress. Especially the "pairing" of two flows will be removed in
future versions.
Before, the output queue of the collector received unnamed tuples with
three fields. This broke the tests and was less understandable. The new
version uses a named tuple for clarity.
The tests were adapted to the new type in the queue and are fixed.
For backwards compatibility a check of the Python version is added and
the subprocess stdout/stderr arguments are passed depending on this
version. See #18.
With the '--host' flag, a local interface IP address can be set on which
the collector listens for incoming flows. Since now, this only worked
with IPv4 addresses (using the default 0.0.0.0 interface). The commit
adds handling of passed-in IPv6 addresses by identifying ":" and then
switching to the AF_INET6 socket family.
Adds a new flag, '-v' or '--verbose', to the analyzer.py script. It uses
a new print method and also skips some parts of the script if not passed
on the CLI.
The analyzer is now found in analyzer.py and uses the '-f' flag for
GZIPed input files. Bundled with the previous PR commit, this update
should now be clearer.
Previously, the analyzer assumed that two consecutive flows would be a
pair. This proved unreliable, therefore a new comparison algorithm is
ussed. It utilizes the IP addresses and the 'first_switched' parameter
to identify two flows of the same connection.
More improvements can be done, especially filtering and in the
identification of the initiating peer.
Tests still fail, have to be adapted to the new dicts and gzip.
Until now, packets arriving at the collector's interface were stored by
timestamp, with the exported flows in the payload. This format is now
extended to also store the client's IP address and port, allowing
multiple clients to export flows to the same collector instance.
As mentioned by @pR0Ps in 6b9d20c8a6/analyze_json.py (L83)
IP addresses, especially in IPv6, should better be stored as parsed
strings instead of their raw integer values. Implemented.
In previous versions, collected flows (parsed data) were stored in
memory by the collector. In regular intervals, or at shutdown, this one
single dict was dumped as JSON onto disk.
With this commit, the behaviour is changed to line-based JSON dumps for
each flow, gzipped onto disk for storage efficiency. The analyze_json is
updated as well to handle the new gzipped files in the new format.
See the comments in main.py for more details.
Fixes#10
Updated the README to reference NetFlow v1 and v5 as well.
The fallback(key, dict) method used an exception-based testing of the
keys existence. Switched to 'if x in'.
The NetFlowListener is based on threading.Thread, which uses the
'timeout' parameter in .join(). Added.
Uses the analyzer's new stdin-reading capabilities to test the analysis
without having to write temporary files. Also removes most of the delays
because the listener can keep up now.
This commit splits the packet collecting and processing out into a
thread that provides a queue-like `get(block=True, timeout=None)`
function for getting processed `ExportPackets`.
This makes it much easier to use rather than starting a generator and
sending a value to it when you want to stop. The `get_export_packets`
generator is an example of using it - it just starts the thread and
yields values from it.
