Monitoring for Large-Scale Networks

fixbuf

Documentation

NAME

ipfix2json - Print contents of an IPFIX file as json text

SYNOPSIS

 ipfix2json [--in FILE_NAME] [--out FILE_NAME]
            [--rfc5610] [--element-file FILE_NAME]
            [--cert-element-path DIRECTORY_NAME]
            [--templates] [--data]
            [--full-structure] [--allow-duplicates]
            [--octet-format FORMAT]

 ipfix2json [--version]

 ipfix2json [--help]

DESCRIPTION

ipfix2json is a tool to read an IPFIX file and print its contents as json. ipfix2json prints templates, data records, options templates, and options records to the output.

ipfix2json supports IPFIX structured data in the form of basicLists, subTemplateLists, and subTemplateMultiLists. By default some of these container structures are not represented as they are in IPFIX, favoring a simpler format more conducive to the json format. For more information read about --full-structure below.

By default, ipfix2json reads the IPFIX file from the standard input and writes the text to the standard output. To specify the input or output file's location, use the --in or --out option, respectively.

Information elements whose data type is a time stamp are printed as a string in ISO-8601 format with no T between the date and time and fractional seconds if needed, for example, "2005-09-21 13:03:56.748Z".

IP addresses are printed as a string in the dotted-quad format for IPv4 and :-separated hexadecimal for IPv6.

Information elements of data type string are printed as string and use the escape sequence \u0000 for control characters and non-ASCII characters.

Elements of type octetArray are printed using base64 encoding by default, and the --octet-format option may be used to change this as described in the "OPTIONS" section below. The user is required to know which elements are string and which are octetArray.

ipfix2json requires the input file to contain the IPFIX templates that describe the data records within the file, and the template must appear before the records that use it. Any records that do not have a corresponding template are ignored.

The default information model used by ipfix2json includes the standard information elements defined by IANA provided by libfixbuf, and the CERT private enterprise information elements used by the NetSA tools yaf(1), pipeline(8), super_mediator(1), and rwsilk2ipfix(1). The CERT information element definitions are provided by the file named cert_ipfix.xml. ipfix2json checks several directories to attempt to find this file, stopping once it finds the first file. An additional path can be added to the search path by using the --cert-element-path option. The list of directories, in search order, is

the directory ../share/libfixbuf relative to the directory containing the application
the libfixbuf subdirectory of the datadir directory specified when ipfix2json was configured (defaults to $prefix/share)
the share/libfixbuf subdirectory installation folder for the GLib-2 library
the libfixbuf subdirectory of the directories specified by the $XDG_DATA_DIRS environment variable, or /usr/local/share and /usr/share/libfixbuf when that variable is empty

ipfix2json exits with an error if it is unable to find the cert_ipfix.xml file. See https://tools.netsa.cert.org/cert-ipfix-registry/ for additional information about this file.

There are two ways to augment the set of elements in ipfix2json's information model:

The --rfc5610 option instructs ipfix2json to watch the input for options records that define private enterprise information elements (as defined by RFC5610) and to add those elements to the information model.
The --element-file=FILE_NAME option tells ipfix2json to parse the contents of FILE_NAME and add those information elements to the information model. The argument is an XML file whose schema is that used by IANA's XML Information Element Registry, with the following additions:

cert:enterpriseId

A number representing the Private Enterprise Number of the element

cert:reversible

A boolean value (true, yes, or 1 for true; false, no, or 0 for false) that specifies whether the element may have a separate identity in a reverse flow.

The --element-file option may be used multiple times to load multiple files, and the loaded elements replace existing elements with the same identifier.

OPTIONS

The following options are available for ipfix2json:

--in FILE_NAME

Sets the input file name to FILE_NAME. When the option is not specified, ipfix2json reads from the standard input or exits with an error when the standard input is a terminal. ipfix2json reads from the standard input if FILE_NAME is '-'.

--out FILE_NAME

Sets the output file name to FILE_NAME. If FILE_NAME exists, it is overwritten. The string '-' may be used to write to standard output (the default). Short option: -o.

--rfc5610

Tells ipfix2json to scan the IPFIX input file for options records that define private enterprise information elements and to add those elements to the information model.

--element-file FILE_NAME

Loads the XML file FILE_NAME and incorporates information element information found in it. The format of the file is described above. The option may be used multiple times to load multiple files, and later elements replace existing elements when they have the same identifier. Short option: -e.

--cert-element-path DIRECTORY_NAME

Adds DIRECTORY_NAME to the search path for the CERT information element XML file. This file must be located to properly process YAF and Super Mediator output. Short option: -c.

--templates

Suppresses the printing of records, causing the output to contain only template records. Short option: -t.

--data

Suppresses the printing of templates, causing the output to contain only data records. Short option: -d.

--full-structure

Produces json output that does not collapse any intermediate containers such as basicLists, subTemplateLists, and subTemplateMultiLists. The result of which resembles the original IPFIX structures more closely. A comparison of the outputs is given in "EXAMPLES" below.

--allow-duplicates

Potentially creates invalid JSON by allowing an object to contain duplicate keys. By default, when a template contains repeated information elements, ipfix2json prints the first instance using the element's name (for example templateId), the second instance using the name and the suffix "-2" (templateId-2), the third instance with the suffix "-3", et cetera. A similar suffix is added when record a contains multiple subTemplateLists that use the same template. Using this option causes ipfix2json to disable use of these suffixes, which may make the output more human-readable.

--octet-format FORMAT

Determines the output format of data record elements whose type is octetArray. The default format is base64.

base64: Displays the value using base-64 encoding, e.g., "Zm9vCg==".
string: Displays octetArray values in the same form as string values. Specifically, displays octets from 32 (space) to 126 (tilde) inclusive as ASCII except for octet value 34 (displayed as \") and 92 (\\). All other values are displayed as escaped utf-8. For example, "foo\u000A" where \u000A represents a newline.
hexadecimal: Outputs the hexadecimal value for each octet in the octetArray, e.g., "666f6f0a".
empty: Outputs an empty string as the value, "".

--version

Prints version and copyright information to standard error and exits. Short option: -V.

--help

Prints a brief usage message to the standard output and exits. Short option: -h.

EXAMPLES

In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line.

 $ ipfix2json --in - --out -

 $ ipfix2json --in /data/ipfix.ipfix --out /data/text.txt

Condensed vs Full Structure output

By default (condensed) output resembles the following:

 {
     "template:0x123()": {
         "informationElementText": "value",
         "informationElementInt": 123,
         "template:0x456()": [
             {
                 "informationElementText": "value",
                 "informationElementInt": 123
             }
         ],
         "template:0x789()": [
             {
                 "informationElementInt": [789,456],
                 "informationElementText": ["abc","def"]
             }
         ]
     }
 }

When --full-structure is used, ipfix2json does not collapse any intermediate containers such as basicLists, subTemplateLists, and subTemplateMultiLists. The result of which resembles the original IPFIX structures more closely:

 {
     "template:0x123()": {
         "informationElementText": "value",
         "informationElementInt": 123,
         "subTemplateMultiList": {
             "template:0x456()": [
                 {
                     "informationElementText": "value",
                     "informationElementInt": 123
                 }
             ],
             "template:0x789()": [
                 {
                     "basicList": {
                         "informationElementInt": [789,456]
                     },
                     "basicList": {
                         "informationElementText": ["abc","def"]
                     }
                 }
             ]
         }
     }
 }

NOTE: No content other than container information elements is added or removed with either format.

AUTHORS

Emily Sarneso, Matt Coates and the CERT Network Situational Awareness Group Engineering Team, <http://www.cert.org/netsa>.

Bug reports may be sent directly to the Network Situational Awareness team at <netsa-help@cert.org>.

fixbuf

fixbuf

NAME

SYNOPSIS

DESCRIPTION

OPTIONS

EXAMPLES

Condensed vs Full Structure output

AUTHORS

SEE ALSO