NAME

yaf - Yet Another Flowmeter

SYNOPSIS

    yaf     [--in INPUT_SPECIFIER] [--out OUTPUT_SPECIFIER]
            [--live LIVE_TYPE] [--ipfix TRANSPORT_PROTOCOL] 
            [--filter BPF_FILTER]
            [--rotate ROTATE_DELAY] [--lock] [--caplist]
            [--group SPREAD_GROUP_NAME(s)]
            [--groupby GROUPBY_TYPE]
            [--stats INTERVAL][--no-stats] [--noerror] 
            [--dag-interface]
            [--napatech-interface]
            [--gre-decode] [--no-frag] 
            [--max-frags FRAG_TABLE_MAX]
            [--ip4-only] [--ip6-only]
            [--idle-timeout IDLE_TIMEOUT] 
            [--active-timeout ACTIVE_TIMEOUT] 
            [--udp-temp-timeout TEMPLATE_TIMEOUT]
            [--force-read-all]
            [--max-payload PAYLOAD_OCTETS] [--udp-payload]
            [--max-flows FLOW_TABLE_MAX]
            [--export-payload] [--silk] [--udp-uniflow PORT]
            [--uniflow] [--mac] [--force-ip6-export]
            [--observation-domain DOMAIN_ID] [--entropy]
            [--applabel] [--applabel-rules RULES_FILE]
            [--ipfix-port PORT] [--tls] [--tls-ca CA_PEM_FILE]
            [--tls-cert CERT_PEM_FILE] [--tls-key KEY_PEM_FILE]
            [--become-user UNPRIVILEGED_USER]
            [--become-group UNPRIVILEGED_GROUP]
            [--log LOG_SPECIFIER] [--loglevel LOG_LEVEL]
            [--verbose] [--version]
            [--p0fprint] [--p0f-fingerprints FILENAME] 
            [--fpexport]
            [--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]]
            [--plugin-opts "OPTIONS[,OPTIONS...]"]
            [--plugin-conf CONF_FILE_PATH[,CONF_FILE_PATH...]]
            [--pcap PCAP_OUTPUT_DIRECTORY] [--pcap-per-flow]
            [--max-pcap MAX_FILE_MB] [--pcap-timer PCAP_ROTATE_DELAY]
            [--pcap-meta-file META_FILENAME]

DESCRIPTION

yaf is Yet Another Flowmeter and yaf is a suite of tools to do flow metering. yaf is used as a sensor to capture flow information on a network and export that information in IPFIX format. It reads packet data from pcap(3) dumpfiles as generated by tcpdump(1), from live capture from an interface using pcap(3), an Endace DAG capture device, or a Napatech adapter, aggregates these packets into flows, and exports flow records via IPFIX over SCTP, TCP or UDP, Spread, or into serialized IPFIX message streams (IPFIX files) on the local file system.

Since yaf is designed to be deployed on white-box sensors attached to local network segments or span ports at symmetric routing points, it supports bidirectional flow assembly natively. Biflow export is done via the export method specified in RFC 5103 Bidirectional Flow Export using IPFIX. See the OUTPUT section below for information on this format.

yaf also supports experimental partial payload capture, specifically for banner-grabbing applications and protocol verification purposes.

The output of yaf is designed to be collected and manipulated by flow processing toolchains supporting IPFIX. The yafscii(1) tool, which is installed as part of YAF, can also be used to print yaf output in a human-readable format somewhat reminiscient of tcpdump(1). yaf output can also be analyzed using the SiLK suite, and the nafalize(1) tool, both available from the CERT NetSA group.

OPTIONS

Input Options

These options control where yaf will take its input from. yaf can read packets from a pcap dumpfile (as generated by tcpdump -w) or live from an interface via libpcap, libdag, or libpcapexpress (Napatech Adapter library). By default, if no input options are given, yaf reads a pcap dumpfile on standard input.

--in INPUT_SPECIFIER

INPUT_SPECIFIER is an input specifier. If --live is given, this is the name of an interface (e.g. eth0, en0, dag0) to capture packets from. Otherwise, it is a filename; the string - may be used to read from standard input (the default).

--caplist

If present, treat the filename in INPUT_SPECIFIER as an ordered newline-delimited list of pathnames to pcap(3) dumpfiles. Blank lines and lines beginning with the character '#' within this are ignored. All pathnames are evaluated with respect to the working directory yaf is run in. These dumpfiles are processed in order using the same flow table, so they must be listed in ascending time order. This option is intended to ease the use of yaf with rotated or otherwise split tcpdump(1) output.

--noerror

Used with the --caplist option. When present, this prevents yaf from exiting when processing a list of dumpfiles in the middle due to an error in a file. yaf will continue to process all files given in the INPUT_SPECIFIER despite errors within those files.

--live LIVE_TYPE

If present, capture packets from an interface named in the INPUT_SPECIFIER. LIVE_TYPE is one of pcap for packet capture via libpcap, or dag for packet capture via an Endace DAG interface using libdag, or napatech for packet capture via a Napatech Adapter using libpcapexpress. dag is only available if yaf was built with Endace DAG support. napatech is only available if yaf was built with libpcapexpress, a library maintained by nPulse Technologies for use with a Napatech adapter.

--dag-interface

If present, the interface on which a packet was received will be noted internally within yaf. When flow records are exported from yaf, an ingressinterface and an egressinterface set of fields will be added to the output. The ingressinterface field will be the physical interface which captured the packet while the egressinterface will be the physical interface | 0x100. This can be used to separate traffic based on DAG physical ports. This option requires building yaf with the --enable-daginterface switch.

--napatech-interface

Similar to --dag-interface, if present, the interface on which a packet was received will be noted internally within yaf. When flow records are exported from yaf, an ingressinterface and an egressinterface set of fields will be added to the output. The ingressinterface field will be the physical interface which captured the packet while the egressinterface will be the the physical interface | 0x100. This can be used to separate traffic based on Napatech physical ports. This option requires building yaf with the --enable-napatechinterface switch.

--filter BPF_FILTER

If present, enable Berkeley Packet Filtering (BPF) in yaf with FILTER_EXPRESSION as the incoming traffic filter. Syntax of FILTER_EXPRESSION follows the expression format described in the tcpdump(1) man page. This option is not currently supported if --live is set to dag or napatech as BPF filtering is implemented with libpcap. This option is also not supported on a Bivio due to the Bivio supplied version of libpcap that supports the zero-copy driver.

Output Options

These options control where yaf will send its output. yaf can write flows to an IPFIX file or export flows to an IPFIX collector over SCTP, TCP, UDP, or Spread. By default, if no output options are given, yaf writes an IPFIX file to standard output.

--out OUTPUT_SPECIFIER

OUTPUT_SPECIFIER is an output specifier. If --ipfix is present, the OUTPUT_SPECIFIER specifies the hostname or IP address of the collector to which the flows will be exported. Otherwise, if --rotate is present, OUTPUT_SPECIFIER is the prefix name of each output file to write to. If --ipfix is present and set to spread, then OUTPUT_SPECIFIER should be set to the name of the Spread daemon to connect to (See below examples of spread daemon names). Otherwise, OUTPUT_SPECIFIER is a filename in which the flows will be written; the string - may be used to write to standard output (the default).

Examples
Output to file

--out flows.yaf

Output to collector on port 18000 at IP address 1.2.3.4

--out 1.2.3.4 --ipfix-port 18000 --ipfix tcp

Connect to the Spread daemon named "4803" on the local machine

--out 4803 or --out 4803@localhost

Connect to the machine identified by the domain name "host.domain.edu" on port 4803.

--out 4803@host.domain.edu

Connect to the machine identified by the IP address "x.y.123.45" on port 18000.

--out x.y.123.45 --ipfix-port 18000

--ipfix TRANSPORT_PROTOCOL

If present, causes yaf to operate as an IPFIX exporter, sending IPFIX Messages via the specified transport protocol to the collector (e.g., SiLK's rwflowpack or flowcap facilities) named in the OUTPUT_SPECIFIER. Valid TRANSPORT_PROTOCOL values are tcp, udp, sctp, and spread; sctp is only available if yaf was built with SCTP support; spread is only available if yaf was built with Spread support. UDP is not recommended, as it is not a reliable transport protocol, and cannot guarantee delivery of messages. As per the recommendations in RFC 5101, yaf will retransmit templates three times within the template timeout period (configurable using --udp-temp-timeout or by default, 10 minutes). Use the --ipfix-port, --tls, --tls-ca, --tls-cert, --tls-key, --tls-pass, and --group options to further configure the connection to the IPFIX collector.

--rotate ROTATE_DELAY

If present, causes yaf to write output to multiple files, opening a new output file every ROTATE_DELAY seconds in the input data. Rotated files are named using the prefix given in the OUTPUT_SPECIFIER, followed by a suffix containing a timestamp in YYYYMMDDhhmmss format, a decimal serial number, and the file extension .yaf.

--lock

Use lockfiles for concurrent file access protection on output files. This is recommended for interoperating with the Airframe filedaemon facility.

--stats INTERVAL

If present, causes yaf to export process statistics every INTERVAL seconds. The default value for INTERVAL is 300 seconds or every 5 minutes. yaf uses IPFIX Options Templates and Records to export flow, fragment, and decoding statistics. If INTERVAL is set to zero, stats will not be exported.

--no-stats

If present, yaf will not export process statistics. yaf uses IPFIX Options Templates and Records to export flow, fragment, and decoding statistics. --no-stats takes precedence over --stats.

Decoder Options

These options are used to modify the yaf packet decoder's behavior. None of these options are required; the default behavior for each option when not present is noted.

--no-frag

If present, ignore all fragmented packets. By default, yaf will reassemble fragments with a 30 second fragment timeout.

--max-frags FRAG_TABLE_MAX

If present, limit the number of outstanding, not-yet reassembled fragments in the fragment table to FRAG_TABLE_MAX by prematurely expiring fragments from the table. This option is provided to limit yaf resource usage when operating on data from very large networks or networks with abnormal fragmentation. The fragment table may exceed this limit slightly due to limits on how often yaf prunes the fragment table (every 5 seconds). By default, there is no fragment table limit, and the fragment table can grow to resource exhaustion.

--ip4-only

If present, ignore all IPv6 packets and export IPv4 flows only. The default is to process both IPv4 and IPv6 packets.

--ip6-only

If present, ignore all IPv4 packets and export IPv6 flows only. The default is to process both IPv4 and IPv6 packets.

--gre-decode

If present, attempt to decode GRE version 0 encapsulated packets. Flows will be created from packets within the GRE tunnels. Undecodeable GRE packets will be dropped. Without this option, GRE traffic is exported as IP protocol 47 flows. This option is presently experimental.

Flow Table Options

These options are used to modify the flow table behavior within yaf. None of these options are required; the default behavior for each option when not present is noted.

--idle-timeout IDLE_TIMEOUT

Set flow idle timeout in seconds. Flows are considered idle and flushed from the flow table if no packets are received for IDLE_TIMEOUT seconds. The default flow idle timeout is 300 seconds (5 minutes).

--active-timeout ACTIVE_TIMEOUT

Set flow active timeout in seconds. Any flow lasting longer than ACTIVE_TIMEOUT seconds will be flushed from the flow table. The default flow active timeout is 1800 seconds (30 minutes).

--udp-temp-timeout TEMPLATE_TIMEOUT

Set UDP template timeout in seconds if --ipfix is set to udp. As per RFC 5101 recommendations, yaf will attempt to export templates three times within TEMPLATE_TIMEOUT. The default template timeout period is 600 seconds (10 minutes).

--max-payload PAYLOAD_OCTETS

If present, capture at most PAYLOAD_OCTETS octets from the start of each direction of each flow. Non-TCP flows will only capture payload from the first packet unless --udp-payload is set. If not present, yaf will not attempt to capture payload. Payload capture must be enabled for payload export (--export-payload), application labeling (--applabel), and entropy evaluation (--entropy). Note that payload capture is still an experimental feature.

--max-flows FLOW_TABLE_MAX

If present, limit the number of open flows in the flow table to FLOW_TABLE_MAX by prematurely expiring the flows with the least recently received packets; this is analogous to an adaptive idle timeout. This option is provided to limit yaf resource usage when operating on data from large networks. By default, there is no flow table limit, and the flow table can grow to resource exhaustion.

--udp-payload

If present, capture at most PAYLOAD_OCTETS octets fom the start of each direction of each UDP flow, where PAYLOAD_OCTETS is set using the --max-payload flag.

--silk

If present, export flows in "SiLK mode". As of YAF 2.0, this will export TCP information (flags, ISN) in the main flow record instead of within the SubTemplateMultiList. This flag must be used when exporting to SiLK for it to collect TCP flow information. This also introduces the following incompatibilities with standard IPFIX export:

  • totalOctetCount and reverseTotalOctetCount are clamped to 32 bits. Any packet that would cause either of these counters to overflow 32 bits will cause the flow to close with flowEndReason 0x02 (active timeout), and will become the first packet of a new flow. This is analogous to forcing an active timeout when the octet counters overflow.

  • The high-order bit of the flowEndReason IE is set on any flow created on a counter overflow, as above.

  • The high-order bit of the flowEndReason IE is set on any flow created on an active timeout.

Since this changes the semantics of the exported flowEndReason IE, it should only be used when generating flows and exporting to rwflowpack, flowcap, or writing files for processing with rwipfix2silk.

--force-read-all

If present, yaf will process out-of-sequence packets. However, it will still reject out-of-sequence fragments.

Export Options

These options are used to modify the the data exported by yaf.

--export-payload

If present, export at most PAYLOAD_OCTETS (the argument to --max-payload) octets from the start of each direction of each flow. Non-TCP flows will only export payload from the first packet. By default, yaf will not export flow payload.

--uniflow

If present, export biflows using the Record Adjacency method in section 3 of RFC 5103. This is useful when exporting to IPFIX Collecting Processes that are not biflow-aware.

--mac

If present, export MAC-layer information; presently, exports source and destination MAC addresses.

--force-ip6-export

If present, force IPv4 flows to be exported with IPv6-mapped IPv4 addresses in ::FFFF/96. This will cause all flows to appear to be IPv6 flows.

--observation-domain DOMAIN_ID

Set the observationDomainID on each exported IPFIX message to the given integer value. If not present, the observationDomainId defaults to 0. This value is also used as the exportingProcessId in the yaf statistics Option Record as a Scope Field.

--udp-uniflow PORT

If present, export each UDP packet on the given port (or 1 for all ports) as a single flow, with flowEndReason set to YAF_END_UDPFORCE (0x1F). This will not close the flow. The flow will stay open until it closes naturally by the idle and active timeouts. Most useful with --export-payload in order to export every UDP payload on a specific port.

Application Labeler Options

If yaf is built with application labeler support enabled (using the --enable-applabel option to ./configure when yaf is built), then yaf can examine packet payloads and determine the application protocol in use within a flow, and export a 16-bit application label with each flow.

The exported application label uses the common port number for the protocol. For example, HTTP traffic, independent of what port the traffic is detected on, will be labeled with a value of 80, the default HTTP port. Labels and rules are taken from a configuration file read at yaf startup time.

Application labeling requires payload capture to be enabled with the --max-payload option. A minimum payload capture length of 384 octets is recommended for best results.

Application labeling is presently experimental. SiLK does support IPFIX import and translation of the application label via rwflowpack, flowcap, and rwipfix2silk.

--applabel

If present, export application label data. Requires --max-payload to enable payload capture.

--applabel-rules RULES_FILE

Read application labeler rules from RULES_FILE. If not present, rules are read by default from /usr/local/etc/yafApplabelRules.conf.

Entropy Measurement

If yaf is built with entropy measurement enabled (using the --enable-entropy option to ./configure when yaf is built,) then yaf can examine the packet payloads and determine a Shannon Entropy value for the payload. The entropy calculation does not include the network (IP) or transport (UDP/TCP) headers. The entropy is calculated in terms of bits per byte, (log base 2.) The calculation generates a real number value between 0.0 and 8.0. That number is then converted into an 8-bit integer value between 0 and 255. Roughly, numbers above 230 are generally compressed (or encrypted) and numbers centered around approximately 140 are English text. Lower numbers carry even less information content. Another useful piece of information is that SSL/TLS tends to zero pad its packets, which causes the entropy of those flows to drop quite low.

--entropy

If present, export the entropy values for both the forward and reverse payloads. Requires the --max-payload option to operate.

IPFIX Connection Options

These options are used to configure the connection to an IPFIX collector.

--ipfix-port PORT

If --ipfix is present, export flows to TCP, UDP, or SCTP port PORT. If not present, the default IPFIX port 4739 is used. If --tls is also present, the default secure IPFIX port 4740 is used.

--tls

If --ipfix is present, use TLS to secure the connection to the IPFIX collector. Requires the TRANSPORT_PROTOCOL to be tcp, as DTLS over UDP or SCTP is not yet supported. Requires the --tls-ca, --tls-cert, and --tls-key options to specify the X.509 certificate and TLS key information.

--tls-ca CA_PEM_FILE

Use the Certificate Authority or Authorities in CA_PEM_FILE to verify the remote IPFIX Collecting Process' X.509 certificate. The connection to the Collecting Process will fail if its certificate was not signed by this CA (or by a certificate signed by this CA, recursively); this prevents export to unauthorized Collecting Processes. Required if --tls is present.

--tls-cert CERT_PEM_FILE

Use the X.509 certificate in CERT_PEM_FILE to identify this IPFIX Exporting Process. This certificate should contain the public part of the private key in KEY_PEM_FILE. Required if --tls is present.

--tls-key KEY_PEM_FILE

Use the private key in KEY_PEM_FILE for this IPFIX Exporting Process. This key should contain the private part of the public key in CERT_PEM_FILE. Required if --tls is present. If the key is encrypted, the password must be present in the YAF_TLS_PASS environment variable.

--group SPREAD_GROUP_NAME

If --ipfix is present and set to spread, use --group to specify the spread group name(s) to publish output. It is possible to list more than one group name in a comma-seperated list. To use Spread as a manifold for different types of flows, use the format GROUP, GROUP_NAME:VALUE, GROUP_NAME:VALUE as the argument to --group and use the --groupby switch. This list should be contained in quotes if it contains spaces (yaf will ignore spaces in quotes). It is suggested to use one group as the catchall for all flows (no value listed) so flows are not lost. The --groupby switch must be used if --group uses GROUP:VALUE format. See the Spread Documentation, www.spread.org, for more details on Spread.

--groupby GROUPBY_TYPE

If --group is used with group values, use --groupby to specify what type of value should be used. Options are port, vlan, applabel, protocol, version. --groupby accepts only one argument. The port option is destination transport port of the flow. version is the IP version of the flow.

Privilege Options

These options are used to cause yaf to drop privileges when running as root for live capture purposes.

--become-user UNPRIVILEGED_USER

After opening the live capture device in --live mode, drop privilege to the named user. Using --become-user requires yaf to be run as root or setuid root. This option will cause all files written by yaf to be owned by the user UNPRIVILEGED_USER and the user's primary group; use --become-group as well to change the group yaf runs as for output purposes.

If running as root for live capture purposes and --become-user is not present, yaf will warn that privilege is not being dropped. We highly recommend the use of this option, especially in production environments, for security purposes.

--become-group UNPRIVILEGED_GROUP

--become-group can be used to change the group from the default of the user given in --become-user. This option has no effect if given without the --become-user option as well.

PCAP Options

These options are used to turn on and configure yaf's PCAP export options.

--pcap PCAP_OUTPUT_DIRECTORY

This option turns on PCAP export in yaf. It will capture and write packets for all network traffic yaf has received and processed to PCAP files in the given OUTPUT_DIRECTORY. OUTPUT_DIRECTORY must be a valid directory. Pcap files will have names in the form of "yaf_[datetime]_serialno.pcap". yaf will write to a file until the file size has reached --max-pcap or every --pcap-timer seconds (whichever happens first). By default, yaf rotates files every 5 MB. Files will be "locked" (".lock" will be appended to the filename) until yaf has closed the file. Be aware that your Operating System will have a limit on the maximum number of files in a directory and a maximum file size. If this limit is reached, yaf will write warning messages and terminate PCAP export. This may effect flow generation if yaf is also writing IPFIX files. Optionally, you can also export meta information about the flows in each rolling PCAP file with the --pcap-meta-file switch.

--pcap-per-flow

If present, yaf will write a pcap file for each flow in the output directory given to --pcap. This option is experimental and should only be used when reading pcap files of reasonable size. yaf only writes up to --max-payload bytes of each packet to the pcap file. Therefore, --max-payload must be set to an appropriate size to prevent packets from being truncated in the pcap file. yaf will use the last three digits of the flowStartMilliseconds as the directory and the flow key hash, flowStartMilliseconds, and serial number as the filename. See the included getFlowKeyHash program in scripts to easily calculate the name of the file for a given flow. When the pcap file has reached --max-pcap size, yaf will close the file, increment the serial number, and open a new pcap file with the same naming convention. Note that your operating system has a limit to the number of open file handles yaf can maintain at any given time. Therefore, the performance of yaf degrades when the number of open flows is greater than the maximum number of file handles.

--max-pcap MAX_FILE_MB

If present, set the maximum file size of pcap files to MAX_FILE_MB MB. The default is 5 MB.

--pcap-timer PCAP_ROTATE_DELAY

If present, yaf will rotate rolling pcap files every PCAP_ROTATE_DELAY seconds or when the file reaches --max-pcap size, whichever happens first. By default, yaf only rotates files based on file size.

--pcap-meta-file META_FILENAME

If present, yaf will export meta information on the flows contained in each rolling pcap file to the filename specified by META_FILENAME. yaf will write a line in the form flow_key_hash | flowStartMilliseconds | pcap_file_name for each flow in the pcap. If a flow exists across 3 pcap files, there will be 3 lines in the pcap-meta-file for that flow (each line having a different filename). The pcap-meta-file will rotate files every 10 rolling pcap files. A new file will be created in the form META_FILENAMEx where x starts at 1 and increments by 1 for every new pcap-meta-file created. This file can be uploaded to a database for flow correlation and flow-to-pcap analysis.

Logging Options

These options are used to specify how log messages are routed. yaf can log to standard error, regular files, or the UNIX syslog facility.

--log LOG_SPECIFIER

Specifies destination for log messages. LOG_SPECIFIER can be a syslog(3) facility name, the special value stderr for standard error, or the absolute path to a file for file logging. The default log specifier is stderr if available, user otherwise.

--loglevel LOG_LEVEL

Specify minimum level for logged messages. In increasing levels of verbosity, the supported log levels are quiet, error, critical, warning, message, info, and debug. The default logging level is warning.

--verbose

Equivalent to --loglevel debug.

--version

If present, print version and copyright information to standard error and exit.

Plugin Options

These options are used to load, configure, and run a yaf plugin.

--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]

Specify the plugin to load. The loaded plugin must follow the yaf plugin framework. LIBPLUGIN_NAME must be the full path to the plugin library name. The only plugin currently supported is the Deep Packet Inspection plugin. This flag will only be recognized if yaf is configured with --enable-plugins. There are also configure options to export only DNS Authoritative and NXDomain responses. Read the document on the DPI Plugin for more information.

--plugin-opts "OPTIONS[,OPTIONS...]"

Specify the arguments to the plugin given to --plugin-name. This flag will only be recognized if yaf is configured with --enable-plugins and --plugin-name is set to a valid plugin. For example, the DPI Plugin takes the well-known port of a protocol(s) to enable DPI (default for DPI is all protocols).

--plugin-conf CONF_FILE_PATH[,CONF_FILE_PATH...]

Specify the path to a configuration file for the plugin given to --plugin-name. This flag will only be recognized if yaf is configured with --enable-plugins and --plugin-name is set to a valid plugin. If this switch is not used, but the plugin requires a configuration file, the default location /usr/local/etc will be used.

Passive OS Fingerprinting (p0f)

These options are used to enable p0f in yaf. p0f is presently experimental. There is no support in yafscii or SiLK for printing p0f related data. Currently, yaf uses the p0f Version 2 SYN fingerprints (see p0f.fp).

--p0fprint

If present, export p0f data. This data consists of three related information elements; osName, osVersion, osFingerPrint. This flag requires yaf to be configured with --enable-p0fprinter.

--p0f-fingerprints

Location of the p0f fingerprint file(s), p0f.fp. Default is /usr/local/etc/p0f.fp. This version of YAF includes the updated CERT p0f fingerprints. See <https://tools.netsa.cert.org/confluence/display/tt/p0f+fingerprints> for updates.

--fpexport

If present, enable export of handshake headers for external OS fingerprinters. The related information elements are firstPacketBanner and secondPacketBanner. This flag requires yaf to be configured with --enable-fpexporter.

OUTPUT

Basic Flow Record

yaf's output consists of an IPFIX message stream. yaf uses a variety of templates for IPFIX data records; the information elements that may appear in these templates are enumerated below. For further information about the IPFIX information model and IPFIX message stream, see RFC 5102, RFC 5101, and RFC 5103. As of yaf 2.0, yaf nests some templates in an IPFIX subTemplateMultiList. In order to retain compatibility with the SiLK Tools, use --silk to prevent yaf from nesting TCP Information Elements. Below are descriptions of each of the templates YAF will export. See the Internet-Draft Export of Structured Data in IPFIX for more information on IPFIX lists.

yaf assigns information element numbers to reverse flow elements in biflow capture based on the standard IPFIX PEN 29305. This applies only for information elements defined in the standard IPFIX Information Model (RFC 5102) that do not have a reverse information element already defined. For information elements defined under the CERT PEN, a standard method is used to calculate their reverse element identifier. The method is that bit fourteen is set to one in the IE field, (e.g. 16384 + the forward IE number.)

flowStartMilliseconds IE 152, 8 octets

Flow start time in milliseconds since 1970-01-01 00:00:00 UTC. Always present.

flowEndMilliseconds IE 153, 8 octets

Flow end time in milliseconds since 1970-01-01 00:00:00 UTC. Always present.

octetTotalCount IE 85, 8 octets

Number of octets in packets in forward direction of flow. Always present. May be encoded in 4 octets using IPFIX reduced-length encoding.

reverseOctetTotalCount Reverse (PEN 29305) IE 85, 8 octets

Number of octets in packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.

packetTotalCount IE 86, 8 octets

Number of packets in forward direction of flow. Always present. May be encoded in 4 octets using IPFIX reduced-length encoding.

reversePacketTotalCount Reverse (PEN 29305) IE 86, 8 octets

Number of packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.

reverseFlowDeltaMilliseconds CERT (PEN 6871) IE 21, 4 octets

Difference in time in milliseconds between first packet in forward direction and first packet in reverse direction. Correlates with (but does not necessarily represent) round-trip time. Present if flow has a reverse direction.

sourceIPv4Address IE 8, 4 octets

IPv4 address of flow source or biflow initiator. Present for IPv4 flows without IPv6-mapped addresses only.

destinationIPv4Address IE 12, 4 octets

IPv4 address of flow source or biflow responder. Present for IPv4 flows without IPv6-mapped addresses only.

sourceIPv6Address IE 27, 16 octets

IPv6 address of flow source or biflow initiator. Present for IPv6 flows or IPv6-mapped IPv4 flows only.

destinationIPv6Address IE 28, 16 octets

IPv6 address of flow source or biflow responder. Present for IPv6 flows or IPv6-mapped IPv4 flows only.

sourceTransportPort IE 7, 2 octets

TCP or UDP port on the flow source or biflow initiator endpoint. Always present.

destinationTransportPort IE 11, 2 octets

TCP or UDP port on the flow destination or biflow responder endpoint. Always present. For ICMP flows, contains ICMP type * 256 + ICMP code. This is non-standard, and an open issue in YAF.

flowAttributes CERT (PEN 6871) IE 40, 2 octets

Miscellaneous flow attributes for the forward direction of the flow. Always present (YAF 2.1 or later). Current flag values:

- Bit 1: All packets in the forward direction have fixed size - Bit 2: Packet(s) in the forward direction was received out-of-sequence

reverseFlowAttributes CERT (PEN 6871) IE 16424, 2 octets

Miscellaneous flow attributes for the reverse direction of the flow. Always present (YAF 2.1 or later). Current flag values:

- Bit 1: All packets in the reverse direction have fixed size - Bit 2: Packet(s) in the reverse direction was received out-of-sequence

protocolIdentifier IE 4, 1 octet

IP protocol of the flow. Always present.

flowEndReason IE 136, 1 octet

Flow end reason code, as defined by the IPFIX Information Model. Always present. In --silk mode, the high-order bit is set if the flow was created by continuation.

silkAppLabel CERT (PEN 6871) IE 33, 2 octets

Application label, defined as the primary well-known port associated with a given application. Present if the application labeler is enabled, and was able to determine the application protocol used within the flow.

vlanId IE 58, 2 octets

802.1q VLAN tag of the first packet in the forward direction of the flow.

reverseVlanId Reverse (PEN 29305) IE 58, 2 octets

802.1q VLAN tag of the first packet in the reverse direction of the flow. Present if the flow has a reverse direction.

subTemplateMultiList IE 293, variable length

Represents a list of zero or more instances of a structured data type, where the data type of each list element can be different and corresponds with different template definitions. The Information Element Number will change upon updates to the IPFIX lists specification and libfixbuf releases.

TCP Flow Template

The following six Information Elements will be exported as a template within the subTemplateMultiList unless --silk is used.

tcpSequenceNumber IE 184, 4 octets

Initial sequence number of the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

reverseTcpSequenceNumber Reverse (PEN 29305) IE 184, 4 octets

Initial sequence number of the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

initialTCPFlags CERT (PEN 6871) IE 14, 1 octet

TCP flags of initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

unionTCPFlags CERT (PEN 6871) IE 15, 1 octet

Union of TCP flags of all packets other than the initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

reverseInitialTCPFlags CERT (PEN 6871) IE 16398, 1 octet

TCP flags of initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

reverseUnionTCPFlags CERT (PEN 6871) IE 16399, 1 octet

Union of TCP flags of all packets other than the initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

MAC Flow Template

The following two Information Elements will be exported as a template within the subTemplateMultiList.

sourceMacAddress, IE 56, 6 octets

Source MAC Address of the first packet in the forward direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.

destinationMacAddress, IE 80, 6 octets

Destination MAC Address of the first packet in the reverse direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.

Payload Flow Template

The following two Information Elements will be exported as a template within the subTemplateMultiList.

payload CERT (PEN 6871) IE 18, variable-length

Initial n bytes of forward direction of flow payload. Present if payload collection is enabled and payload is present in the forward direction of the flow. This element is contained in the yaf Payload template within the subTemplateMultiList.

reversePayload CERT (PEN 6871) IE 16402, variable-length

Initial n bytes of reverse direction of flow payload. Present if payload collection is enabled and payload is present in the reverse direction of the flow. This element is contained in the yaf Payload template within the subTemplateMultiList.

Entropy Flow Template

The following two Information Elements will be exported as a template within the subTemplateMultiList.

payloadEntropy CERT (PEN 6871) IE 35, 1 octet

Shannon Entropy calculation of the forward payload data. This element is contained in the yaf Entropy template within the subTemplateMultiList.

reversePayloadEntropy CERT (PEN 6871) IE 16419, 1 octet

Shannon Entropy calculation of the reverse payload data. This element is contained in the yaf Entropy template within the subTemplateMultiList.

p0f Flow Template

The following six Information Elements will be exported as a template within the subTemplateMultiList if present and only if p0f is enabled.

osName CERT (PEN 6871) IE 36, variable-length

p0f OS Name for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

reverseOsName CERT (PEN 6871) IE 16420, variable-length

p0f OS Name for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

osVersion CERT (PEN 6871) IE 37, variable-length

p0f OS Version for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

reverseOsVersion CERT (PEN 6871) IE 16421, variable-length

p0f OS Version for the reverse flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

osFingerPrint CERT (PEN 6871) IE 107, variable-length

p0f OS Fingerprint for the forward flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

reverseOsFingerPrint CERT (PEN 6871) IE 16491, variable-length

p0f OS Fingerprint for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

Fingerprint Exporting Template

The following four Information Elements will be exported as a template within the subTemplateMultiList if present and only if fpexport is enabled.

firstPacketBanner CERT (PEN 6871) IE 38, variable-length

IP and transport headers for first packet in forward direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

reverseFirstPacketBanner CERT (PEN 6871) IE 16422, variable-length

IP and transport headers for first packet in reverse direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

secondPacketBanner CERT (PEN 6871) IE 39, variable-length

IP and transport headers for second packet in forward direction (third packet in sequence) to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

reverseSecondPacketBanner CERT (PEN 6871) IE 16423, variable-length

IP and transport headers for second packet in reverse direction (currently not used). Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

Hooks Templates

yaf can export other templates within the subTemplateMultiList if plugins are enabled in yaf. See yafdpi(1) for descriptions of the yaf Deep Packet Inspection Information Elements.

Statistics Option Template

yaf will export information about its process periodically using IPFIX Options Template Record. This record gives information about the status of the flow and fragment table, as well as decoding information. This can be turned off using the --no-stats option. The following Information Elements will be exported:

exportedFlowRecordTotalCount IE 42, 8 octets

Total amount of exported flows from yaf start time.

packetTotalCount IE 86, 8 octets

Total amount of packets processed by yaf from yaf start time.

droppedPacketTotalCount IE 135, 8 octets

Total amount of dropped packets according to statistics given by libpcap, libdag, or libpcapexpress.

ignoredPacketTotalCount IE 164, 8 octets

Total amount of packets ignored by the YAF packet decoder, such as unsupported packet types and incomplete headers, from yaf start time.

expiredFragmentCount CERT (PEN 6871) IE 100, 4 octets

Total amount of fragments that have been expired since yaf start time.

assembledFragmentCount CERT (PEN 6871) IE 101, 4 octets

Total number of packets that been assembled from a series of fragments since yaf start time.

flowTableFlushEventCount CERT (PEN 6871) IE 104, 4 octets

Total number of times the yaf flow table has been flushed since yaf start time.

flowTablePeakCount CERT (PEN 6871) IE 105, 4 octets

The maximum number of flows in the yaf flow table at any one time since yaf start time.

exporterIPv4Address IE 130, 4 octets

The IPv4 Address of the yaf flow sensor.

exportingProcessId IE 144, 4 octets

Set the ID of the yaf flow sensor by giving a value to --observation-domain. The default is 0.

meanFlowRate CERT (PEN 6871) IE 102, 4 octets

The mean flow rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.

meanPacketRate CERT (PEN 6871) IE 103, 4 octets

The mean packet rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.

SIGNALS

yaf responds to SIGINT or SIGTERM by terminating input processing, flushing any pending flows to the current output, and exiting. If --verbose is given, yaf responds to SIGUSR1 by printing present flow and fragment table statistics to its log. All other signals are handled by the C runtimes in the default manner on the platform on which yaf is currently operating.

EXAMPLES

To generate flows from an pcap file into an IPFIX file:

yaf --in packets.pcap --out flows.yaf

To capture flows from a pcap interface and export them to files in the current directory rotated hourly:

yaf --live pcap --in en1 --out en1_capture --rotate 3600

To capture flows from an Endace DAG card and export them via IPFIX over TCP:

yaf --live dag --in dag0 --ipfix tcp --out my-collector.example.com

To convert a pcap formatted packet capture and convert that into IPFIX:

yaf <packets.pcap >flows.yaf

To publish to spread group TST_SPRD for a spread daemon running locally on port 4803:

yaf --live pcap --in eth1 --out 4803@localhost --ipfix spread --group TST_SPRD

To publish to spread groups based on application label for spread daemon running locally on port 4803:

yaf --live pcap --in eth1 --out 4803@localhost --ipfix spread --group "SPRD_CATCHALL, SPRD_DNS:53, SPRD_HTTP:80, SPRD_SMTP:25" --groupby applabel --applabel --max-payload=400

To run yaf with application labeling enabled and export via IPFIX over TCP:

yaf --live pcap --in eth1 --out 127.0.0.1 --ipfix tcp --ipfix-port=18001 --applabel --applabel-rules=/usr/local/etc/yafApplabelRules.conf --max-payload=300

To run yaf with BPF on UDP Port 53

yaf --live pcap --in en1 --out /path/to/dst/ --rotate 120 --filter="udp port 53"

To run yaf with Deep Packet Inspection enabled for HTTP, IMAP, and DNS:

yaf --in packets.pcap --out flows.yaf --applabel --max-payload=400 --plugin-name=/usr/lib/dpacketplugin.la --plugin-opts="80 143 53"

KNOWN ISSUES

YAF BPF Filtering does not operate with the Bivio Zero-Copy Library.

YAF BPF Filtering is ignored when using --live dag or napatech because libpcap is not used.

YAF PCAP Export options are ignored when using --live dag or napatech.

YAF 2.x requires libfixbuf 1.0.0 or later.

YAF 2.0 will not interoperate with the SiLK tools unless --silk is used, due to the TCP Information Elements being nested in the subTemplateMultiList. YAF 2.0 must be used with an IPFIX Collecting Process that can handle IPFIX lists elements, especially the subTemplateMultiList Information Element in order to retrieve certain flow information. Older versions of YAF can read YAF 2.0 flow files, but will ignore anything contained in the subTemplateMultiList.

The plugin infrastructure has been modified in YAF 2.0 to export templates in YAF's subTemplateMultiList element.

YAF 2.0 will export statistics in an Options Template and Options Data Records unless the --no-stats switch is given. The IPFIX Collecting Process should be able to differentiate between options records and flow records in order to prevent incorrect transcoding of statistics records into flow records.

YAF will not rotate output files if it is not seeing any flow data. However, it will continue to write process statistics messages at the configured interval time to the most recent output file.

YAF 0.7 does not interoperate with previous versions, because it no longer uses provisional information elements for the reverse direction of a biflow. YAF 0.7 must be used with an IPFIX Collecting Process that uses PEN 29305 for reverse information elements. For export to SiLK, this implies that the SiLK packer or rwipfix2silk utility must be built against libfixbuf 0.7.0 or later.

Presently, the destinationTransportPort information element contains ICMP type and code information for ICMP or ICMP6 flows; this is nonstandard and may not be interoperable with other IPFIX implementations.

Bug reports and feature requests may be sent via email to <netsa-help@cert.org>.

AUTHORS

Brian Trammell, Chris Inacio <inacio@cert.org>, Michael Duggan, Emily Sarneso, Dan Ruef, and the CERT Network Situational Awareness Group Engineering Team, <http://www.cert.org/netsa>.

SEE ALSO

yafscii(1), tcpdump(1), pcap(3), nafalize(1), Spread Documentation at www.spread.org, libp0f at <https://tools.netsa.cert.org/confluence/display/tt/libp0f>, and the following IETF Internet RFCs: Specification of the IPFIX Protocol for the Exchange of IP Traffic Flow Information RFC 5101, Information Model for IP Flow Information Export RFC 5102, Bidirectional Flow Export using IPFIX RFC 5103, Export of Structured Data in IPFIX RFC 6313