Packets from Network

The backbone of the Netlogger, the very device which enables you to obtain information. Collects packets from one of your network interfaces and sends them into the Netlogger. In the properties window of the Packets from network tool, you select the interface you wish to Collect Packets from. It also allows you to filter on port number and protocol (TCP/UDP/ICMP) and discard ack-packets. Detailed description

Screenshots

Screenshots

Packets from File

Enables you to read files that you've recorded (using Write Packets tool) earlier. When reading the packets choose weather you want the recorded data to run through the Read Packet File tool once, or loop the file through the tool until you choose to stop the Read Packet File tool. You can also limit the number of packets read pr second to conserve system ressources, entering a zero sets the the value to unlimited. Furthermore the timestamp can be adjusted to realtime, this has no relevance unless you use the file for test purposes. In most cases you will want to keep the timestamp. Detailed description

Screenshots

Records from File

Enables you to read files that you've recorded (using Log to File tool) earlier or any other columbased tekst file and merge the data into the stream. Detailed description

Screenshots

Lookup ODBC

Enables you to read data from a database and merge the data into the stream. Detailed description

Screenshots

Lookup static

Basically is a user defined table containing information. An example:
Drop a Lookup Static tool on the canvas. Right click on the tool. A window appears with add, rename and remove columns buttons and remove row(s). Choose add column and type a name in this example "src_port". Add a new column and name it, in this example "text". Now you can fill in source ports and tekst or load it automatically from a file using "load data"
The Lookup Static tool is valuable when connected to the Join tool. Static set can enrich the information by adding information defined by the user. For further description see the description of the Join tool. Detailed description

Screenshots

All Netlogger log tools can be configured to perform automatic file rotation with respect to time and size. Backup of data and uplink to SAN and NAS is configured to individual needs.

Screenshots

Log to Database/ODBC

The Netlogger interfaces with almost all databases via ODBC, several native drivers are also supported.
Right click the Log to Database tool, and choose the database that you want to use from the drop down menu and configure it.
Detailed description
In the following examble a My SQL data base is installed directly on the netlogger.
By entering a table name and push the "check" buttom the netlogger will check if the table already exist.
The table properties can be viewed by pushing the view buttom.

Screenshots

If the table has not already been created the netlogger will type "Table does not exist" and the check buttom will change to "create"

Screenshots

Press the create buttom and enter the table properties in the create table dialog

Screenshots

Log to File

Log to File allows you to store files, that have been processed through the the Netlogger.
Above you see the dialog window that appear when you doubleclick on the Log to File tool.
First decide the location of the location of the log file. Then the duration between each log file shift. The interval between log file shifts can be between one minute and one week.
The BLOB Handling lets you decide weather you want to discard the BLOBs, put them all in one separate file or file them, each one in its own file.

A BLOB is a Binary Large Object. BLOB is the type a large field will be stored in. Ex. post from the Extract HTTP tool. Please consult Appendix for further information.
Field order enables you to choose the order of the different fields being logged in the file.
Format permits format change between the Netlogger default format and the W3C Common Log File Format. Selecting W3C in the Format field will predefine the rest of the selections in this box.

The drop down box in the Headers Field makes it possible to log the headers, either using the Netlogger standard or in a spreadsheet friendly format, or not at all.

The Field separator field allows you to choose your own field separator. The default Field separator is <TAB> , but you might want to change it, if your data contains the field separator. For this purpose you can use the Replace occurrences with.

Ex. if you choose to use <TAB> as filed separator and a <TAB> occurs in the data set, the selected separator in the Replace occurrences with field will replace the <TAB> making it possible to use the log file.

The Record separator marks the end of each record. You can make your own Record separator or use <LF>, which is the default Record separator in the Netlogger.

The replace Occurrences with field connected to Record separator works the same way, as the replace occurrences field in the Headers section. Detailed description

Screenshots

Write Packets

This is an extremely use full tool. Imagine an event where you need to record traffic instantaneously. Reasons for this could be that someone is trying to hack his/her way into your network, and you want to analyze the events. Once connected with the Packets from Network tool, you'll need to choose a location for the recorded traffic. After that, you can start recording for further analysis. The recorded packets are stored in pcap or raw IP file format, which can be utilized by reading the files with the Packet From File tool. Detailed description

Forward Packets

This tool allows you to stream the packets to other netloggers or servers for further processing.

Screenshots

The tool has 3 different packet streaming options:
Direct: Sends the raw packet stream to the interface specified in the interface field (eth0, eth1).
UDP (to specified port): Converts the raw packet stream to "packet caple" format (UDP/RTP).
UDP with NTP (to specified port): Converts the raw packet stream to "packet caple" format (UDP/RTP) with timestamps.
For UDP streams the destination IP address and Port number must be specified in the format "IP:PORT"
Detailed description

Log Sessions

Used in conjunction with the Session Info Tool this tool logs to a binary format optimised to minimise the storage requirement for lawful anti-terror logging.
The files can be rotated according to time and size and is easily searched and extracted by Unispeed custom build session viewer. Detailed description

Detect Protocol

Detect protocols and assigns information to the packet stream for the purpose of running the traffic through the right/proper Extractor. Other use could be to gather statistical info about what kind of traffic is running through your network.

Once connected with a tool supplying traffic, you will be able to view the following information, by right clicking on the tool and choosing Data Samples:

Screenshots

application: Provides information about which protocol the individual packet uses.
transport: Information about how the packet is transported. this can be TCP or UDP.
network: Provides layer 3 information about the individual packets. Usually this will be IP, however it can also detect protocols like ARP etc.
link: Layer 2 protocol information, usually ethernet.
packet: The actual packet. Detailed description

Extract Packet Header

The Extract Packet Headers tool extracts information from the packet headers.
The situation often arise, where it is opportune to extract information from the packet headers. If you have the need to filter by source IP address, packet type or mac address, You'll need to use this tool.

Screenshots

Double clicking on the Extract Packets Headers tool opens a window similar to the one shown on this screenshot.

This tool allows you to specify which type of information you want to extract.

Checking only the Ethernet Specific fields box, will provide this information:
dst_mac: The mac address of the ethernet card to which the packet was send.
src_mac: The mac address of the ethernet card from which the packet was send.
eth_proto: Could be for.ex. IP.
timestamp: The time the packet was received by the the Netlogger.
len: The length of the packet measured in bytes.

If you choose also to check the IP Specific fields checkbox. The Extract Packet Headers tool will give you this additional information:
dst_ip: Destination ip address.
src_ip: Source ip address.
ip_proto: Shows weather the packet type is: TCP, UDP or ICMP.
Adding IMCP Specific fields contributes this field information.
icmp_code: Depending on icmp_type
icmp_type: A typename describing problems in handling the packet. Some important ones which are widely used include: Echo Reply, Echo Request, Redirect, Destination Unreachable, Trace route and Time Exceeded.

If you want information about the ports to facilitate traffic, checki the UDP Specific fields checkbox:

dst_port: Destination port.
src_port: Source port.

By checking all of the check boxes, you will have all of the above mentioned information, and these extra coming from the TCP Specific fields:

win: Window size. The maximum number outstanding packets allowed, before retransmitting packets.
ack: The acknowledge number sent by the receiving port, verifying that the packet has been picked up by this port.
seq: The sequel number attached to any given packet. Used to verify that the packet has reached its destination.
tcp_flags: Various flags for this packet: URG, ACK, PSH, RST, SYN and FIN.
dst_port and src_port, also appear when checking TCP Specific fields. Detailed description

Note: Using the appropriate tool at the right time limits the Netlogger's resource usage.

Screenshots

Session info

The session info tool extracts sender and receiver IP -address and port number together with transport protocol and time-stamp for the first and terminating packets in a session. The retrieved data complies with the anti terror requirements described in the Danish anti-terror legislation.
Output from Session Info can be merged with unique user data via the ODBC Lookup tool, and data can be logged to files or data base. However the optimal log format is achieved with the Session log tool. Detailed description

Bandwidth measuring

Calculates individual client bandwidth Detailed description

Filter Packets

The Filter Packets tool allows you to divide your stream of traffic in two, namely the accepted packets, which meets the filter definition, defined in the properties window, and the rejected packets which doesn't.

Filtering through this tool can be done using many parameters. Filter Packets tool lets you choose between weather you want to filter for TCP, UPD or IMCP type packets. You can also choose to filter for either two or all three of the above mentioned packet types.

Furthermore you can choose to filter by IP addresses. Here Filter Packets allows you to filter by scope of, by several scopes of and by actual IP addresses. Using wild card '*' is also permitted. Either as a part of the IP address or simply by using '*' to describe all IP addresses.

Filtering by ports can likewise be quite use full. Normally SMTP traffic runs to Port 25, and HTTP traffic runs to Port 80. This gives you an opportunity to weed out types of traffic you have no interest in.

Screenshots

The Filter Packets tool has two output streams, often you need to weed out a range of IP- Addresses. This is done by entering the range you want to filter out and use the rejected stream as your positive output. Several filter tools can be connected to allow for comprehensive packet filtering. Detailed description

Screenshots

Traffic Measuring

This tool is an out of the box traffic-billing tool which calculates traffic volume for any combination of protocol, source IP, destination IP, and if appropriate source port, and destination port. Detailed description

Screenshots

Billing can be performed based on Payload, Transport Layer, Network layer or Ethernet layer.

Screenshots

The output from Traffic Measure shoul look like this.

Screenshots

The Extract tools basically translates the collected packets into a readable format. There are of course different Extractors connected to each protocol. Below we will explain the different function of each individual Extractor.

Extract DHCP

This tool extract information from DHCP request in order to track which user is assigned to a specific IP-address. Detailed description

Extract DNS

Very simple Extractor which Extracts the information in a name server request and response.

Screenshots

When viewing Data Samples you will see the following type of information:

processing_time: The time it took the server to process the query measured in seconds.
status: Shows you weather the query was handled according to what was expected.
answer: Provides answer to the query.
query: The actual query.
query_type: Informs about the type of query. ex. type: MX is a mail domain query, while type: A is a query for an A record.
server_ip: Notifies you about the ip address of the server, that handled the query.
client_ ip: Provides information about the ip_address of the client.
timestamp: The time the request was processed in a unix time stamp format. Detailed description

Extract Syslog

This tool extracts Syslog packets into a stream of records. Detailed description

Extract SMTP

Extracts header and content information from mail traffic, when attempting to send a mail, or when two mail servers are communicating.
The Extract SMTP tool presents you with different extraction options. First You will have to choose the port from which you want to extract. This will in most cases be port 25. Furthermore the properties window in the Extract SMTP tool enables you to extract a body snippet, which is the first part of the mail being sent, or a body BLOB, which is the entire mail. Detailed description

Screenshots

When looking at Data Samples from the Extract SMTP tool, note the detail that you are presented with:
body: Will not be shown in the Data Samples window. However if you decide to log the extracted data into a file or data base, you will be able to view the body.
snippet: Will be shown in this window, is a sample of what this mail contains.
subject: The subject line in the senders e-mail.
to_addr: The address which the sender has typed into the to field of his/her mail.
recipient: The actual address to which the e-mail is being sent. In most cases to_addr and recipient are identical.
from _addr: The address by which the sender has identified him/herself.
sender: The from address, from the e-mail body. In most cases from_addr and sender are identical.
size: Size of mail measured in bytes.
server_ip: The ip address that handles the sending of the e-mail.
client_ip: The senders ip address.
timestamp: unix timestamp, documenting when the transaction took place.

Screenshots

Extract POP3

Extracts header and content information from mail traffic, One of the methods of receiving mails, from a mail client.

The Extract POP3 tool presents you with different extraction options. First You will have to choose the port from which you want to extract. This will in most cases be port 110. The properties window in the Extract POP3 tool enables you to extract the header or the entire mail. Furthermore limiting the amount of data, by omitting emails containing more than x megabyte, going to the Extract POP3, is an option. Detailed description

Screenshots

When looking at Data Samples from the Extract tool, you are presented with the following information:
Password: Mail senders password.
username: Mail senders username.
content: The actual content of the mail, which can not be shown using the GUI, but can be stored in a log file.
message_id: Unique identification code allocated to each individual mail by the mail server.
subject: The text in the subject line of the mail.
recipient: The mail address of the recipient.
sender: The name and e-mail address of the sender.
message size: The size of the message measured in bytes.
server_ip: IP address of the server that handled the mail.
client_ip: The senders IP address.
timestamp: Unix timestamp, documenting when the transaction took place.

Screenshots

Screenshots

Extract IMAP

Extracts header and content information from mail traffic, One of the methods of receiving mails, from a mail client.

The Extract IMAP tool presents you with different extraction options. First You will have to choose the port from which you want to extract. This will in most cases be port 143. The properties window in the Extract IMAP tool enables you to extract the header or the entire mail. Furthermore limiting the amount of data, by omitting emails containing more than x megabyte, going to the Extract IMAP, is an option.

When looking at Data Samples from the Extract tool, you are presented with the following information:

Screenshots

password: Mail senders password.
username: The mail senders username.
content: The actual content of the mail, which can not be shown using the GUI, but can be stored in a log file.
message_id: Unique identification code allocated to each individual mail by the mail server.
subject: The text in the subject line of the mail.
recipient: The mail address of the recipient.
sender: The name and e-mail address of the sender.
message size: The size of the message measured in bytes.
server_ip: IP address of the server that handled the mail.
client_ip: The senders IP address.
timestamp: Unix timestamp, documenting when the transaction took place. Detailed description

Extract NNTP

Extracts USENET News Transfer Protocol header information. First You will have to choose the port from which you want to extract. This will in most cases be port 119. The properties window in the Extract NNTP tool enables you to extract the header or the entire mail. Furthermore limiting the amount of data, by omitting emails containing more than x megabyte, going to the Extract NNTP, is an option.

Screenshots

By right clicking on the Extract NNTP tool, and clicking on Data Samples, you will get the following information:
message_id: Unique identification code allocated to each individual mail by the mail server.
subject: The text in the subject line of the mail.
recipient: The mail address of the recipient
sender: The name and e-mail address of the sender.
message size: The size of the message measured in bytes.
server_ip: IP address of the server that handled the mail.
client_ip: The senders IP address.
timestamp: Unix timestamp, documenting when the transaction took place. Detailed description

Screenshots

Extract FTP

Extracts FTP information, header information and file content. Once connected with a Collect Packets tool (or another tool) you can draw the following information from the Extract FTP tool:

Screenshots

File: The transferred file. While it cannot be shown in the GUI, you are able to log the file for further analysis.

The two server responses: Response2 and Response1, are part of the communication between client and FTP server.
Command: Clients request to the FTP server.
Filename: Name of the transferred file.
Password: Clients password.
Username: Clients username.
Server_Port: The FTP service port.
Server_Address: The servers IP address.
Client_Port: The port used by the client.
Client_Address: Clients IP address.
Id: Number of bytes from the start of the BLOB file, this BLOB will be found. Detailed description

Extract MSN IM

Extracts MSN Instant Messenger information, header information and file content.
Once connected with a Collect Packets tool (or another tool), you need to select weather you want the Extract MSN IM to draw out the entire information, or only the header information.

Screenshots

By right clicking on the Extract MSN_IM tool, and choosing Data Samples, you will get the following information:

Screenshots

content: The actual content of the message.
message_type: Provides information about the message type. Ex. text/plain is a plain text message.
recipient_nick: Recipients nickname
sender_nick: Senders nickname
recipient: The mail address of the recipient
sender: The name and e-mail address of the sender.
im_proto: In this case this will always be MSN.
local_sender: Has the value 0 or 1. 1 signifies that the message has been captured on the way to the microsoft server, while 0 implies that the message has been captured on the way from the microsoft server to the recipient.
server_port: The MSN_IM service port.
client_port: The port used by the client.
server_ip: IP address of the server that handled the mail.
client_ip: The senders IP address.
timestamp: Unix timestamp, documenting when the transaction took place. Detailed description

Extract HTTP

The Extract HTTP tool will once connected to a packet stream, provide you with detailed information about each HTTP request that is contained in the traffic.

Right click on the Extract HTTP tool, and choose, first what port you want the Extract HTTP tool to collect its traffic from (usually this will be port 80, or port 0 to monitor all ports).

Then choose weather you want the Extract HTTP tool to Include POST BLOB, include content BLOB or handle request only.

Screenshots

Right click on the Extract HTTP tool, choose Data samples and look at the information in the window that pops up. You'll see a mass of constantly changing information. The reason for this is that the Netlogger every four seconds picks a sample and shows it to you. Starting from the top we'll explain what information each field provides you with:

length: The size of the request measured in bytes.
set_cookie: Web server attempt to set a cookie
content_type: The data type of the content. ex: text/html, text/css, image/gif, image/jpeg, application/x-javascript.
code: Describes the response code of the request. ex. 404 = not found, 200 = done.....
The Netlogger has extra codes build in which provides with additional information. These are:
700: Client has pushed stop or refresh before server could respond.
701: Client pushed stop or refresh while server was responding.
702: No server response within timeout.
703: Server response contained fewer bytes than expected.
time taken: The time it took the server to process the HTTP-request, measured in seconds.
post_len: The length of a post, measured in bytes.
referrer: The referrer URL.
ref_args, ref_uri and ref_host, are all part of the referrer.
forwarded_for: Client ip address added by gateway.
via: Can be added by a gateway.
agent: Shows you which browser and operating system the client is using.
cookie: The cookie the client sends to the server in the request.
args: The query string consisting of zero or more arguments separated by '&'.
uri: The Uniform Resource Identifier (URI) identifying the page on the server.
host: Also known as virtual host or domain. The host is the name of the server.
method: In this case the method will mostly be: GET, POST or HEADER.
server_port: The web server port number.
server_ip: The servers ip address
client_ip: The clients ip address
time: The time the request was captured by the Netlogger. (unix timestamp). Detailed description

Screenshots

Extract VOIP SIP

This tool extracts VOIP SIP packets into a stream of records. Detailed description

Screenshots

Filter Records

When filtering records it is necessary to write a Filter rule. Doing this implies using a variety of operations which will be described in the following section.
> : greater than
< : smaller than
=> : greater or equal to
<= : smaller or equal to
!= : different from
== : exactly equal to
like :
an example could be: content_type like "image/*"
This example of a Filter Rule sends records containing images to accepted records.

nlike :
an example: content_type nlike "*/text"
This example of a Filter rule sends packets containing any kind of text to rejected records.

contains :
Just like "like" but it is not necessary/possible to use wild cards.

and & or:
Chaining two rules together making a new rule.
An example: content_type like "image/*" and (host = "ads.foobar.org" or host = "ads2.foobar.org") Detailed description

Screenshots

Aggregate

Aggregates information over time, according to the users selection. Double click on the tool after having dropped it on the canvas. A window with the following Fields will appear.

Keys: The tool can aggregate over one or more keys, the keys being the data about which more information is desired.
Avg: The average of the key. This of course only make sense if the average of the key bares any relevance. Example: the length of a packet (len), will produce the average size of the packets within the chosen time frame.
Max: The maximum number within the chosen dataset.
Min: The minimum number within the chosen dataset.
Sum: The sum of the value of all occurrences of the key, within the chosen timeframe.
Include count: If this check box is check it provides the dataset with, the number of occurrences of the key, within the chosen time frame. Detailed description

Screenshots

Classify

This powerful tool matches records against a binary tree of rules and classifies each record according to the rules.
When a rule is applied to a record two things can happen, either it can forward the record to another rule or it can classify the record. Each rule evaluates to true or false and and at each truth value either another rule or a classifier can me inserted. If a rule is inserted the record is forwarded to this rule if the truth value is met and if a classifier is inserted and the truth value is met the record is classified and output.
E-commerce businesses know that only a few percents of the visitors actually end up buying something. A well-known analysis to identify problems in the buying process is the check point analysis. The analysis identifies visitor click patterns through identified spots (check points).
The Rule Based Classifier can easily identify your check points and thus make the whole check point analysis easier for you.
Detailed description

Screenshots

Merge

Merges two record streams into one. By default the Merge tool ignores fields, that are only: represented in one of the datasets entering the Merge tool. But by choosing properties you can force the Merge tool to include data fields that is only represented in one of the data-sets. To do this, check the check boxes.

Detailed description

Script

This advanced tool requires Python know-how and allows for the user to write python scripts manipulating and adding fields. Please consult http://www.python.org for further information about using Python.

In the screen shots below, you'll find: a commented example of a python script.

Screenshots

Change fields

This tool enables you to change the data types or remove any fields in the records.

Screenshots

For each of the fields you can choose between the following data types, from the drop down box:
Packet, BLOB, STRING32, STRING128, STRING512, INT32, UINT32, INT64, UINT64, FLOAT, IPADDR, TIMESTAMP or you can choose to <skip> the field. If you choose a data format conflicting with data type, a warning text on the canvas will draw attention to the error. Detailed description

Screenshots

Join

This tool makes it possible to join a record stream with a record set.
In the tool description of the Static Set tool we made a small table, which will be usefull now.
Drop a Read Packet File tool, a Extract HTTP tool, a Join tool and a Static Set tool on the canvas. Connect the Read Packet File tool with the Extract HTTP tool, and the Extract HTTP tool with the Join tool, using the dark blue arrow. Now connect the Static Set tool with the Join tool, using the yellow arrow.

Screenshots

Now double click on the Join tool, and you'll see this window. Choose code in the "Stream join field" and code in the "Set join field" and check the "Outer join" checkbox, press "OK".

Your configuration should look something like this.

Screenshots

Right clicking on the Join tool, choosing Data Samples, will show you a window similar to this

Screenshots

As you can tell the Join tool has added a new field, the text field. Which in this case informs the user that the query was handled according to what was expected: OK. Detailed description

Transition

Calculates costumers behaviour on a web site Detailed description

Screenshots

Unique ODBC

By selecting a field from a list of available fields on the input, it can map each unique value in the selected field to a corresponding unique id that is invented at the time a value is first seen, and kept for a later discoveries of the same value. This value/id pair is stored in a database table, if you do not chose read-only mode of operation. Similar, if the database table already contains value/id pairs, these are looked up and used as ids for values already in the table. Detailed description

Screenshots

Double Coverage

This tool allows you to detect relations between two data sets. Detailed description

Screenshots

Pageview

This tool determines whether a hit is to be considered a page impression.
The user can use an exclude-set with url's that must not be considered a page. The tool will then set the 'is_page'-field to '0' for records matching this url's in the exclude-set.
Updates done to the exclude-set might not take effect immediately since the tool maintains a cache that is updated periodically. However, the effect should be visible within app. 60 seconds. Detailed description

Schedule

This tool allows you to set the start-time and end-time for a chain of tools
This is particular convenient when a logging job needs to be performed out side working hours Detailed description

Screenshots

HTTP Argument

This tool extracts URI arguments on the form `?x1=y1&x2=y2' into additional stream fields. The tool can also be used to extract cookies in standard format Detailed description

Derive Agent

This tool derives browser and platform information from the agent string and appends fields with this information.
Detailed description

Visualisation

This tool allows you to create a real-time visualisation from the output of an aggregation tool.
The tool automatically adopts the key identifier when connected to the aggregate tool.
When you Right-click the tool and select options you can choose a name for your visualisation and select the Value field. Detailed description

Screenshots

The actual visualisation window is opened from the top menu "Netlogger drop-down window", and the different outputs are selected by Right-clicking the visualisation window. You can select between a top and a bottom list. the number of lines is sized by changing the size of the output window

Screenshots

Session Info Dialog

This tool allows you to retrieve data from previously stored binary session Info logs.
The tool can extract sets of data filtered by time and IP-address.
Detailed description

Screenshots

The session info dialog will query you where to store the retrieved log data.

Screenshots

The out put from the Session info dialog can be viewed by any speadsheet and contains the nessecary information to comply with the Danish anti-terror legislation.

Screenshots

---