yaf deep packet inspection
yaf
can examine packet payloads, capture useful information for a specific protocol, and export it in a protocol-specific template within yaf's
SubTemplateMultiList if yaf
is built with plugin support enabled (using the --enable-plugins option to ./configure when yaf
is built),
The DPI plugin requires payload capture to be enabled with the --max-payload option. A minimum payload capture length of 384 octets is recommended for best results. --applabel is also required, as the application label determines how the inspection will execute.
DPI in yaf
is directly related to application labeling as it will only perform DPI if a match was found during the application labeling phase, and it will only execute an inspection specific to the protocol denoted by the application label.
In order to enable DPI in yaf
the following should be added to the command line:
--plugin-name=/usr/local/lib/dpacketplugin.la
You can also add the option flag to specify which protocols to perform DPI:
--plugin-opts="53 80 21"
The above will perform DPI for DNS, HTTP, and FTP.
DPI operates differently depending on whether the protocol is plugin-based or regex-based in the yafApplabelRules.conf file. If the protocol uses a regex rule for application labeling, it will have a list of regular expressions in the yafDPIIRules.conf file that are compared against the captured payload. Any matches are stored and later exported in an IPFIX information element. If the protocol is based on a plugin rule, it will store important information while it is decoding the payload using the dynamically loaded plugin listed in the yafApplabelRules.conf file. See the source code to the plugins included with yaf
for details on the specific protocol implementations.
The yafDPIRules.conf file should be in the same location as the yafApplabelRules.conf file. The file follows a similar format to yafApplabelRules.conf. The file is a list of label, element pair statements. A label statement begins with the keyword 'label', and has the following form:
label <N> element <N2> <element-rule>
where <N> is the application label (usually the well-known port) found in the yafApplabelRules.conf file (an unsigned 16-bit decimal integer in the range 0 to 65535), <N2> is the Information Element ID found in the /usr/local/yaf/CERT_IE.h and below, and <element-rule> is a PCRE regular expression and will be stored and associated with the ID number preceding it. There can be multiple lines for a single application label, however each should have a different <N2>. There should be parentheses around the substring you want to capture and store. If there is more than 1 set of parentheses in the regular expression, the most outer set is the substring captured. (See PCRE documentation for details on regular expressions and substring matching.)
To define your own information elements, use the following form:
label <N> user <E> name <element-name> <element-rule>
where <N> is the application label found in yafApplabelRules.conf file. <E> is the Information Element ID in the range of 0 to 65535 to be given to the element upon export. This number should be unique to this file and should NOT be defined in /usr/local/yaf/CERT_IE.h. This element will be added to the template upon processing of this file, and must be added to the yaf
collecting process in order to properly decode the IPFIX message. <element-name> is the name you want to give to this IPFIX Information Element. This name can consist of letters and numbers and underscores; it can not contain special characters or spaces. <label-rule> is the PCRE regular expression and will be stored and associated with the Information Element ID and name preceding it. There is a limit of 10 "user labels" per protocol that YAF will store and execute. To find out if yaf
accepted your elements, run yaf
with --verbose. All user elements will be exported using the CERT Private Enterprise Number (PEN) 6871. ONLY user labels for protocols FTP, HTTP, IMAP, SMTP, RTSP, SSH, and SIP will be added. All elements will be added to the bottom of the appropriate protocol template in the form of an fbBasicList_t.
A "#" smybol starts a comment for the entire line.
If a rule is not properly formatted, all subsequent rules may not be processed.
It is acceptable to comment out any yaf
DPI rules. yaf
rules commented out will not be executed against the payload but they will still exist in the template and record. User-defined information elements are added based on the configuration file at run time.
Upon yaf
startup and capture, you will be able to see if the rule files and their regular expressions were accepted using the --verbose flag.
[2011-04-15 19:39:25] DPI Running for ALL Protocols
[2011-04-15 19:39:25] Reading packets from packets.pcap
[2011-04-15 19:39:25] Initializing Rules from DPI File /usr/local/etc/yafDPIRules.conf
[2011-04-15 19:39:25] DPI rule scanner accepted 58 rules from the DPI Rule File
An unacceptable regular expression will be brought to your attention with the above statements. If you choose certain protocols for inspection using the --plugin-opts
flag, only the appropriate rule statements will be loaded into the DPI Rule Scanner.
The following options can be given to ./configure when yaf
is built to export DNS authoritative and NXDomain Responses only.
Enable export of DNS Authoritative Responses only. The default is to capture and export all DNS Responses. This flag can be used in conjunction with --enable-exportDNSNXDomain. It is only recognized if --plugin-name is set to the DPI plugin, application labeling is enabled, and --max-payload is set.
Enable export of DNS NXDomain Responses only. The default is to capture and export all DNS Responses. This flag can be used in conjunction with --enable-exportDNSAuth. It is only recognized if --plugin-name is set to the DPI plugin, application labeling is enabled, and --max-payload is set.
yaf
's output consists of an IPFIX message stream. yaf
uses a variety of templates for IPFIX data records; As of yaf
2.0, yaf
uses a subTemplateMultiList to export optional information elements, such as Deep Packet Inspection fields, relating to the flow. Below are templates that may appear in this subTemplateMultiList depending on the application label of the flow.
For more information on yaf
information elements see yaf(1). For more information on IPFIX Structured lists, see the Internet Draft, Export of Structured Data in IPFIX, <RFC 6313>. Most of the elements are exported as a basicList. An IPFIX basicList represents a list of zero or more instances of any Information Element (IE 32765).
File Transfer Protocol (FTP) Deep Packet Inspection is based on RFC 959. The following information elements are exported as a template in the subTemplateMultiList as basicLists of variable length elements in the following order:
FTP Commands or Replies.
FTP User Command Argument. This command will normally be the first command transmitted by the user.
FTP Password Command Argument. This command must be preceded by the user name command, and is usually required to complete authentication.
FTP Data Representation Type.
FTP Reply. This consists of a three digit number followed by some text.
HTTP Deep Packet Inspection is based on RFC 2616. The following information elements are exported as a template in the subTemplateMultiList as basicLists of variable length elements in the following order:
HTTP Server Response-header field. Contains information about the software used to handle the HTTP Request.
HTTP User-Agent Request-header field. Contains information about the user agent originating the request.
HTTP GET Command. Retrieves information identified by the following Request-URI.
HTTP Connection header fields. Contains options that are desired for a particular connection.
HTTP Referer request-header field. Address (URI) of the resource which the Request-URI was obtained.
HTTP Location response-header field. Used to redirect the recipient to a location to complete a request or identify a new resource.
HTTP Host Request-header. The Internet host and port number of the resource being requested.
HTTP Content-Length header. Indicates the size of the entity-body.
HTTP Age response-header. Argument is the sender's estimate of the time elapsed since the response.
HTTP Response Status Code. Usually a three-digit number followed by text.
HTTP Accept-Language Request-Header field. Restricts the set of natural languages that preferred.
HTTP Accept request-header field. Used to specify certain media types that are acceptable for the response.
HTTP Content Type entity-header field. Indicates the media type of the entity-body.
HTTP Version Number.
HTTP Cookie Header Field.
HTTP Set Cookie Header Field.
IMAP Deep Packet Inspection is based on RFC 3501. The following information elements are exported as a template in the subTemplateMultiList as basicLists of variable length elements in the following order:
IMAP Capability Command and Response. Captures the listing of capabilities that the server supports.
IMAP Login Command. Arguments are user name and password.
IMAP STARTTLS Command. Captures this command only as no arguments or responses are related.
IMAP Authenticate Command. Captures the authentication mechanism name of the server following this command.
Captures a variety of IMAP Commands and their arguments.
IMAP Exists Response. Reports the number of messages in the mailbox.
IMAP Recent Response. Reports the number of message with the Recent flag set.
Real Time Streaming Protocol (RTSP) Deep Packet Inspection is based on RFC 2326. The following information elements are exported as a template in the subTemplateMultiList as basicLists of variable length elements in the following order:
RTSP URL. Captures the address of the network resources requested.
RTSP Version Number.
RTSP Status-Line. Captures the RTSP Protocol version, numeric status code, and the textual phrase associated with the numeric code.
RTSP Content-Length Header Field. Contains the length of the content of the method.
RTSP Command. Captures the method to be performed and the Request-URI associated with the method.
RTSP Content Type.
RTSP Transport request header field. Captures the transport protocol used and the parameters that follow.
RTSP CSeq field. Contains the sequence number for an RTSP request-response pair.
RTSP Location header field.
RTSP Packets Received header field.
RTSP User Agent field. Contains information about the user agent originating the request.
RTSP Jitter Value.
Session Initiation Protocol (SIP) Deep Packet Inspection is based on RFC 3261. The following information elements are exported as a template in the subTemplateMultiList as basicLists of variable length elements in the following order:
SIP Invite Method. Contains the SIP address and SIP Version Number.
SIP Command. Contains a SIP Method, SIP address, and SIP Version Number.
SIP Via contains the SIP Version Number and the address the sender is expecting to receive responses.
SIP Max Forwards contains the limit of number of hops a request can make on the way to its destination.
SIP Address contains the argument of the To, From, or Contact Header Fields.
SIP Content Length header field. Contains the byte count of the message byte.
SIP User Agent Header Field. Contains information about the User Agent Client originating the request.
Simple Mail Transfer Protocol (SMTP) Deep Packet Inspection is based on RFC 2821. The following information elements are exported as a template in the subTemplateMultiList as basicLists of variable length elements in the following order:
SMTP Hello or Extend Hello command. Captures the command and the domain name of the SMTP client.
SMTP Mail Command. Contains the reverse-path of the sender mailbox.
The SMTP Recipient (RCPT) Command. Captures the command and the forward-path of the recipient of the mail data.
SMTP Content Type Header Field.
SMTP Subject. Contains the subject of the mail data.
SMTP Filename. Contains the name of the file attached to the mail message.
SMTP Content-Disposition Header field.
SMTP Replies. Consists of a three digit number followed by text.
Enhanced SMTP. Contains the ESMTP command with the following argument.
SMTP Size Header Field. Contains the size in bytes of the mail data.
SSH Version Number
Domain Name System (DNS) Deep Packet Inspection is based on RFC 1035. DNS Information is exported in the yaf
subTemplateMultiList as a subTemplateList of Resource Record Templates. Each resource record entry contains generic resource record information such as type, TTL, and name. There is also one element (subTemplateList) that contains resource record specific information based on the type of resource record (A Record vs NS Record, for example). The subTemplateList will contain one entry for each resource record in the packet. Due to alignment issues, the resource record specific element is the first element in the template and is therefore the first item listed below. The following information elements exist in the DNS resource record subTemplateList:
The following elements (in order) are contained in the DNS Resource Record Template.
An IPFIX subTemplateList. This list contains a "DNS Resource Record Type" Template. The type of this template depends on the type (dnsQRType) of resource record. See the DNS Resource Record Types listed below.
A DNS Query or Response Name. This field corresponds with the QNAME field in the DNS Question Section or the NAME field in the DNS Resource Record Section.
DNS Time To Live. This is an unsigned integer that specifies the time interval, in seconds, that the resource record may be cached for. This will contain a value of zero for DNS Queries.
DNS Query/Response Type. This corresponds with the QTYPE field in the DNS Question Section or the TYPE field in the DNS Resource Record Section. This field determines the type of subTemplateList found in this record.
DNS Query/Response header field. This corresponds with the DNS header one bit field, QR. If the message is a query (0), or a response (1).
DNS Authoritative header field. This corresponds with the DNS header one bit field, AA. This bit is only valid in responses (when dnsQueryResponse is 1), and specifies that the responding name server is an authority for the domain name in the question section.
DNS NXDomain or Response Code (RCODE). This corresponds with the DNS RCODE header field. This field will be set to 3 for a Name Error, 2 for a Server Failure, 1 for a Format Error, and 0 for No Error.
DNS Resource Record Section Field. This field will be set to 0 if the information is from the Question Section, 1 for the Answer Section, 2 for the Name Server Section, and 3 for the Additional Section.
DNS Transaction ID. This identifier is used by the requester to match up replies to outstanding queries.
This entry will exist if dnsQRType is 1 and the A Record contains an IP address.
IPv4 address of the host.
This entry will exist if dnsQRType is 2 and the NS Record contains an NSDNAME.
An authoritative name server domain-name.
This entry will exist if dnsQRType is 5 and the CNAME Record contains an CNAME.
A domain-name which specificies the canonical or primary name for the owner.
This entry will exist if dnsQRType is 6 and the SOA Record contains at least 1 of the following elements:
Corresponds to DNS SOA MNAME Field.
Corresponds to DNS SOA RNAME Field.
Corresponds to DNS SOA SERIAL Field.
Corresponds to DNS SOA REFRESH Field.
Corresponds to DNS SOA RETRY Field.
Corresponds to DNS SOA EXPIRE Field.
Corresponds to DNS SOA MINIMUM Field.
This entry will exist if dnsQRType is set to 12 and PTRDNAME exists.
Corresponds to DNS PTR PTRDNAME Field.
This entry will exist if dnsQRType is set to 16 and TXT-DATA exists.
Corresponds to DNS TXT TXT-DATA field.
This entry will exist if dnsQRType is set to 28 and the IPv6 Address exists. See RFC 3596.
An IPv6 Address found in the data portion of an AAAA Resource Record.
This entry will exist if dnsQRType is set to 33 and at least 1 of the following elements exist. See RFC 2782.
Corresponds to the Target Field in the DNS SRV Resource Record.
Corresponds to the Priority Field in the DNS SRV Resource Record.
Corresponds to the Weight Field in the DNS SRV Resource Record.
Corresponds to the Port Field in the DNS SRV Resource Record.
Secure Socket Layer (SSL)/Transport Layer Security (TLS) Deep Packet Inspection can identify and export handshake and certificate information if it is contained in the payload of the flow. Each certificate identified by yaf
is exported as a separate template in yaf's
subTemplateMultiList. The following handshake elements are exported as a template in the subTemplateMultiList in the following order:
sslCipher is exported by yaf
as a basicList that contains the list of CipherSuites suggested by the client in the ClientHello Message.
sslServerCipher is the CipherSuite chosen by the server in the ServerHello message.
sslClientVersion is the version it supports contained in the initial ClientHello message.
sslCompressionMethod is the compression method chosen by the server in the ServerHello message.
The following SSL Certificate information elements are exported as a separate entry in yaf's
subTemplateMultiList in the following order (one entry for each certificate captured):
The signature contained in a SSL certificate.
Certificate Issuer's Country Name.
Certificate Issuer's Organization Name.
Certificate Issuer's Organizational Unit Name.
Certificate Issuer's Zip Code.
Certificate Issuer's State.
Certificate Issuer's common name.
Certificate Issuer's locality name.
Certificate Issuer's street address.
Certificate Subject's Country Name.
Certificate Subject's Organization Name.
Certificate Subject's Organizational Unit Name.
Certificate Subject's Zip Code.
Certificate Subject's State.
Certificate Subject's common name.
Certificate Subject's locality name.
Certificate Subject's street address.
The Certificate Version.
Internet Relay Chat (IRC) Deep Packet Inspection is based on RFC 2812. The following information element is exported as a template in the subTemplateMultiList as a basicList of variable length elements in the following order:
IRC Chat or Join Message. This field contains any IRC Command and the following arguments.
Network News Transfer Protocol (NNTP) Deep Packet Inspection is based on RFC 977. The following information elements are exported as a template in the subTemplateMultiList in the following order:
NNTP Reply. This consists of a three digit status code and text message.
NNTP Command. Contains an NNTP Command and following argument(s).
Post Office Protocol 3 (POP3) Deep Packet Inspection is based on RFC 1939. The following information element is exported as a template in the subTemplateMultiList as a basicList of variable length elements:
POP3 Command and Replies. Contains any command or reply message found in POP3 payload data.
Service Location Protocol (SLP) Deep Packet Inspection is based on RFC 2608. The following information elements are exported as a template in the subTemplateMultiList in the following order:
Contains the text elements found in an SLP Service Request.
SLP Version Number.
SLP Message Type. This value should be between 1 and 11 and describes the type of SLP message.
Trivial File Transfer Protocol (TFTP) Deep Packet Inspection is based on RFC 1350. The following information elements are exported as a template in the subTemplateMultiList in the following order:
TFTP Name of File being transferred.
Contains the mode of transfer. (Currently supported: netascii, octet, mail).
MySQL Deep Packet Inspection is based on information found at http://forge.mysql.com/wiki/MySQL_Internals_ClientServer_Protocol. MySQL packet capture information is exported in the yaf
subTemplateMultiList as a subTemplateList of Command Code, Command Text pairs.
An IPFIX SubTemplateList. This type represents a list of zero or more instances of a structured data type, where the data type of each list element is the same and corresponds with a single Template Record. In this case, a list of MySQL Command Code, Command Text Pairs. There will be one element in the list for each MySQL Command found.
MySQL Command Text. For example, this can be a SELECT, INSERT, DELETE statement. This is the first element in the MySQL subTemplateList.
MySQL Command Code. This number should be between 0 and 28. This is the second element in the above MySQL subTemplateList.
MySQL Login User Name.
Emily Sarneso <ecoff@cert.org> and the CERT Network Situational Awareness Group Engineering Team, http://www.cert.org/netsa
yaf(1), yafscii(1), PCRE Documentation