There's no 3.9.3, only 3.9.2, which is also the correct Debian version.
3.11.1 is available in pyenv 2.3.8, but 2.3.7 is currently in use by the
Github action. So 3.11 is used, which uses the latest patch version
Previously, option templates and their data records were not correctly
recognized. This is fixed now. Collectors can now use the
V9ExportPacket.options field to get a list of V9OptionsDataRecord, with
scopes and data fields.
Templates are mixed in the templates dict. They will have both data
templates and option templates. Let's hope exporters do not mix them
(re-use the same IDs for both template types).
During development, the search for the correct template was refactored.
The templates are not pased into the V9DataFlowSet any more. Only the
one single matching template is passed into V9DataFlowSet and
V9OptionsDataFlowset, as should be.
Refs #30
This is a hacky workaround to handle V9 options templates, without
implementing the full corresponding spec. This solves missing templates
which raise a V9TemplateNotRecognized exception, even though an exporter
might do everything correctly.
Refs #29
Refs #30
The parse_packet function is one of the main functions for usage of this
library in other scripts. It works, but was under-documented until now.
Especially the 'templates' parameter might lead to confusions for new
users who have not yet worked with templates. This commit should make
things clearer.
Refs #28
Signals INT and TERM were not correctly handled in the 'while True' loop
of the yielding listener function. Now, the loop breaks as expected,
terminating the listener thread and the application.
This commit replaces multiple occurences of new features which were not
yet implemented with Python 3.5.3, which is the reference backwards
compatibility version for this package. The version is based on the
current Python version in Debian Stretch (oldstable). According to
pkgs.org, all other distros use 3.6+, so 3.5.3 is the lower boundary.
Changes:
* Add maxsize argument to functools.lru_cache decorator
* Replace f"" with .format()
* Replace variable type hints "var: type = val" with "# type:" comments
* Replace pstats.SortKey enum with strings in performance tests
Additionally, various styling fixes were applied.
The version compatibility was tested with tox, pyenv and Python 3.5.3,
but there is no tox.ini yet which automates this test.
Bump patch version number to 0.10.3
Update author's email address.
Resolves#27
Templates may be withdrawn as per RFC7011. Receiving a template with an
existing template_id and a field_count of 0 now triggers deletion of
this template.
Parts of the IPFIXFieldTypes class were extracted into the new
IPFIXDataTypes class, to increase readability and stability.
The IPFIXDataRecord class and its field parser is now more in tune with
the specifications, handling signed and unsigned, as well as float,
boolean and UTF8 strings etc.
Corresponding tests were extended with softflowd packets (level
"ethernet") and value checks (e.g. MAC address).
Resolves#25
In IPFIX, template fields can be signed or unsigned, or even be pure
bytes or unicode string. This differentiation was extended in this
commit.
Additionally, the IPFIX_FIELD_TYPES dict mapping from int->str was
replaced by a more verbose version, which also includes the standardized
IANA data types. The class' methods provides access to the fixed data
set. This is then used in the IPFIXDataRecord parser.
Refs #25
The function send_recv_packets in tests stored all processed
ExportPackets by default in a list. Memory usage tests were therefore
based on this high amount of stored objects, since no instance of any
ExportPacket was deleted until exit.
With the new parameter store_packets the caller can define how many
packets should be stored during receiving, as to test multiple
scenarios.
Three such scenarios are implemented: don't store any packet, store
maximum of 500 at a time and store all packets. This comes much closer
to the real world scenario of the collector, which uses a "for export in
listener.get" loop, dumping any new ExportPacket to file immediatelly
and then deleting the object.
Yet, the case where all packets are stored must still be covered as
well, because the collector might not be the only implementation which
uses listener.get, so finding memory leaks should be covered.
A new test file is added which contains memory and CPU tests. For now,
only the memory usage tests work (threading!). They print out tables of
memory usage based on file path and on function. Additionally, they check
some basic measurements: if all packets were processed and if a
collection of version 9/10 called any functions in 10/9.
Refs #24
Analyzer test was missing imports.
IPFIX templates with 16 bytes fields were processed extra, since struct
does not natively support conversion to int. The new implementation
still handles it extra, but uses struct's "s" unpack format descriptor
now.
The previously introduced tests/lib.py contained the NetFlow v9 packets
and then the IPFIX packets, those were split and put into their
respective test files again. The lib now contains shared objects only.
For IPFIX tests were added. Two new packets were added, one with
templates and one without (again, real exports from softflowd).
Different cases are checked: no template, template and later template.
Fields of flows are also checked, especially IPv6 addresses.
Note: exports made with softflowd were created by softflowd 1.0.0,
compiled from https://github.com/irino/softflowd