Read netflow_hawaii.pdf text version

www.pervasivetechnologylabs.iu.edu

Netflow - what is it, and why do we hate it?

David A. J. Ripley Advanced Network Management Laboratory Pervasive Technology Labs Indiana University

1

www.pervasivetechnologylabs.iu.edu

David A. J. Ripley

MSc., ARCS

[email protected]

Lead Network Security Developer, Advanced Network Management Laboratory Indiana University Network security infrastructure development and research for the ANML. Background in physics, image processing, satellite remote sensing, system administration.

2

Overview

www.pervasivetechnologylabs.iu.edu

· What is a "flow"? · What is Netflow specifically? · Netflow collection infrastructure. · Netflow processing, online and offline. · Facts and figures, problems and issues.

3

Netflow Recap

www.pervasivetechnologylabs.iu.edu

· Q. What is a flow? · A. In a general sense, a flow is a series of packets with

some attribute(s) in common.

4

Netflow Recap

www.pervasivetechnologylabs.iu.edu

· Common attributes define a flow

· · · ·

Source and/or destination of the traffic. Protocol - TCP, UDP, ICMP? Timing - start, end, and duration of the traffic. Routing information - interfaces, AS, etc.

5

Netflow Recap

www.pervasivetechnologylabs.iu.edu

· Flows can be unidirectional or bidirectional - the latter

adds possible information.

· Aggregated flows. · Application flows - classify packets by inspecting their

contents

· We're not going to worry too much about these cases.

6

Netflow Recap

www.pervasivetechnologylabs.iu.edu

· As far as we're concerned, a flow is a series of packets

with the same:

· · ·

Protocol (UDP, TCP, ICMP) Source and destination ports Source and destination addresses

7

Netflow Recap

www.pervasivetechnologylabs.iu.edu

· The recording of a flow is subject to idiosyncrasies of

sampling frequency and sampling window

· ·

Flows longer than one minute will appear as two flow records Multiple flows (with the same characteristics) within a single one minute window will appear as a single flow record

·

Bucket timeout - systems typically consider one minute windows.

·

Sampling rate

·

Router will only consider one out of every N packets; N=??? - data loss vs. expensive operations.

8

An example

www.pervasivetechnologylabs.iu.edu

· Host A gets a web page from Host B · This will show up as two flows (usually)

· ·

Host A, port 12345 Host B, port 80 Host B, port 80 Host A, port 12345

9

Why Netflow?

www.pervasivetechnologylabs.iu.edu

· What kinds of information can we gather?

· · · · · ·

What percentage of traffic on the network is web traffic? ssh? IRC? What is the average transfer rate for network communications? Who uses the network the most? Have usage patterns changed over time? For the Chicago region, how much of the traffic of the region is staying in the region? Many others

10

Why Netflow?

www.pervasivetechnologylabs.iu.edu

· Historically, traffic accounting, acceptable use

enforcement;

· ·

Researchers and engineers needed to answer all kinds of questions about network traffic. Traffic accounting in the form of flow records provided that information.

11

Why Netflow?

www.pervasivetechnologylabs.iu.edu

· Traffic Engineering/Accounting

· · · ·

How traffic is shared with competitors; how customers are billed.

· Security/Policy monitoring

DoS/DDoS detection

· Research

Measuring the growth of networks Identifying how the network is being used.

12

What data is there?

www.pervasivetechnologylabs.iu.edu

· It depends. · We keep talking about "flows" - we really mean Cisco's

Version 5 flow records

A Cisco-defined "standard" Used on Abilene - so that's what we use.

· ·

13

Netflow Version 5

www.pervasivetechnologylabs.iu.edu

· Cisco-defined de-facto standard

· · ·

Efforts are underway in the IETF to make this standard official

· Flows are exported as UDP packets

Each packet contains a number of flow records plus a header with information common to these records Delivery is not guaranteed!

·

There are sequence numbers so we know how many packets we've lost.

14

Netflow V5 Header

www.pervasivetechnologylabs.iu.edu

Byte 1 Version

Byte 2

Byte 3 Count SysUpTime

Byte 4

UNIX Seconds (seconds since Epoch) UNIX Nanoseconds (residual nanoseconds) Flow Sequence Number Engine Type Engine ID Reserved

15

Netflow V5 Record

www.pervasivetechnologylabs.iu.edu

Byte 1

Byte 2

Byte 3

Byte 4

Source IP Address Destination IP Address Next Hop IP Address Input ifIndex Packets Bytes Start time of flow End time of flow Source port Padding Source AS Source Mask Length Destination Mask Length TCP Flags IP Protocol Destination AS Padding Destination port TOS Output ifIndex

16

V5 Header Details

www.pervasivetechnologylabs.iu.edu

Version Count sysUpTime Unix Seconds Unix nanoseconds Flow sequence number Engine Type Engine ID

In our case, 5 Number of records in packet Milliseconds since boot Number of seconds since epoch Residual nanoseconds Sequence number for flow packet User defined User defined

17

V5 Record Details

www.pervasivetechnologylabs.iu.edu

Source IP Destination IP Next Hop IP Input ifIndex Output ifIndex Packets Bytes Start time End time

Source IP Address of flow packets Destination IP Address of flow packets The next hop to which the flow was forwarded SNMP Index value for incoming interface SNMP Index value for outgoing interface Number of packets in flow Number of bytes in flow Time (since boot) of flow start Time (since boot) of flow end

18

V5 Record Details Cont.

www.pervasivetechnologylabs.iu.edu

Source Port Destination Port TCP Flags IP Protocol TOS Source AS Destination AS Source Mask Destination Mask

IP Source port IP Destination port Logical or of all TCP flags seen IP Protocol value Type of service bits set Source Autonomous System Destination Autonomous System Source address prefix mask bits Destination address mask bits

19

Convenience, or lack of it

www.pervasivetechnologylabs.iu.edu

· Flow records are exported in a format that is

convenient for the router, not for us.

· ·

e.g. The flow start and end times are in a form that is not immediately useful, milliseconds since system boot. We have to combine data from individual flow records with header data.

·

Seconds since epoch is the Right Thing

Flow Start Time = Unix Seconds + Unix Nanoseconds - sysUpTime + flow_start (After we've converted all these to the right units)

·

ICMP Type is stored in the destination port field

20

Netflow Collection

www.pervasivetechnologylabs.iu.edu

· A simple flow collection architecture

Router A Router B

flow manager Router C Flow Archiver

21

Netflow Collection

www.pervasivetechnologylabs.iu.edu

· Closer to the truth...

HSTN flow collector IPLS NMS

NMS ATLA KSCY NMS NMS

22

Netflow Collection

www.pervasivetechnologylabs.iu.edu

· ANML Flow collector is a Dual Xeon with quite a bit

of memory and disk space.

· Collects flow data from Abilene core routers.

· ·

Archives raw records (up to 3 months) Redirects to other lab machines

23

Netflow Collection

www.pervasivetechnologylabs.iu.edu

· Much of the grunt work is done using flow-tools

·

(http://www.splintered.net/sw/flow-tools/)

· Also some gobs of glue (cron, perl) · These take care of basic collection, archiving,

expiration.

· flow-tools jolly useful for collection, filtering etc. · Here's how we collect our data:

flow-capture -w /huge/flow -z9 -V5 -m 255.255.248.0 -E800G 0/0/4000 -S5 24

Pump up the volume

www.pervasivetechnologylabs.iu.edu

· How much data gets collected?

· · · · · ·

We're looking at all of Abilene. Large, busy, network. 3700-6000 flow records per second ~424 million records per day Almost 13 billion records per month Even zlib compressed, this comes out at typically 200GB of data per month. A lot!

25

Examining Netflow

www.pervasivetechnologylabs.iu.edu

· Part of our job is using netflow data to see what's

happened/is happening on the network

· We spend a significant amount of time processing the

archived data looking for particular behaviors.

26

Examining Netflow Data

www.pervasivetechnologylabs.iu.edu

$ flow-cat ./data | flow-nfilter -f filters -F tcp_only | flow-stat -f9 -S1 # --- ---- ---- Report Information --- --- --# # Fields: Total # Symbols: Disabled # Sorting: Descending Field 1 # Name: Source IP # # Args: flow-stat -f9 -S1 # # # IPaddr flows octets # 130.49.72.0 30415 6881695 128.223.216.0 28716 540202269 130.49.88.0 24267 10770631 etc.

packets 36110 427566 32186

27

Examining Netflow Data

www.pervasivetechnologylabs.iu.edu

· Our example:

$ flow-cat ./data | flow-nfilter -f filters -F tcp_only | flow-stat -f9 -S1

· We have to have config files like this:

filter-primitive tcp_only type ip-protocol permit 6 filter-primitive udp_only type ip-protocol permit 17 filter-definition tcp_only match ip-protocol tcp_only filter-definition udp_only match ip-protocol udp_only

· We'd like an easier way

28

Fun with Perl

www.pervasivetechnologylabs.iu.edu

· Sometimes limitations of existing tools require rolling

your own.

#!/usr/bin/perl -w use Socket; $i=<STDIN>; while(defined($i=<STDIN>)){ @fields = split /,/, $i ; $fields[10]=inet_ntoa(pack(N,unpack(N,inet_aton($fields[10])) & 0xfffff800)); $fields[11]=inet_ntoa(pack(N,unpack(N,inet_aton($fields[11])) & 0xfffff800)); print(join(",",@fields)); }

29

Might there be a better way?

www.pervasivetechnologylabs.iu.edu

· Looking for a more flexible way of querying data. · But we didn't want to write our own tools.

·

Laziness, impatience (and hubris?)

· Might there be some Structured way to Query the

data, perhaps even an entire Language?

30

SQL? WTF?

www.pervasivetechnologylabs.iu.edu

· Why not put everything into a relational database?

· · ·

There are plenty of reasons not to; It's a lot of data. There are existing ways to query the data.

31

SQL? WTF?

www.pervasivetechnologylabs.iu.edu

· Why do it?

· · ·

It makes it easy to construct very complex queries It's (potentially) fast and efficient Well-established interfaces with familiar front-ends

·

via ODBC etc.

32

Netflow database

www.pervasivetechnologylabs.iu.edu

· Set up a PostgreSQL Server · Created a table or two · Started stuffing data into it.

33

Netflow Database

www.pervasivetechnologylabs.iu.edu

· Each row has all data

from flow record, plus header data from packet.

· We didn't do

anything to the data before dumping it into the table. tables (AS map)

· Added some other

netflow=# \d v5_rawnetflow_current Table "public.v5_rawnetflow_current" Column | Type | Modifiers -------------+---------+----------unix_secs | integer | unix_nsecs | integer | sysuptime | bigint | exaddr | inet | dpkts | integer | doctets | integer | first | bigint | last | bigint | engine_type | integer | engine_id | integer | srcaddr | inet | dstaddr | inet | nexthop | inet | input | integer | output | integer | srcport | integer | dstport | integer | prot | integer | tos | integer | tcp_flags | integer | src_mask | integer | dst_mask | integer | src_as | integer | dst_as | integer |

34

Lesson #1

www.pervasivetechnologylabs.iu.edu

· We learned very quickly that we were stepping over

some kind of line

Disk flushing Numbers of buffers, size of buffers

· · ·

Shared memory (kernel)

35

MS RPC Vulnerability

www.pervasivetechnologylabs.iu.edu

· First test! · Look for machines showing signature behaviour;

· · ·

We have no access to the contents of packets; Host A sends to RPC listener on B; remote shell opens on B; A sends tftp GET command to remote shell; B gets executable from tftp server on A. Exactly the sort of thing we wanted.

36

MS RPC Vulnerability

www.pervasivetechnologylabs.iu.edu

· Broke this down into several tasks;

· · ·

Server-side, digest the main archive looking for traffic on the ports/protocols we were interested in. Query reduced data set looking for matching hosts (hit by scanning source, subsequent tftp activity)

·

We know that there are issues with this approach

(Bonus: AS to AS Name lookup.)

37

MS-RPC Vulnerability

www.pervasivetechnologylabs.iu.edu

· The initial digestion of the raw flow records proved by

far the most time consuming operation

· Subsequent operations were quite quick. · Used ODBC to get this into a spreadsheet.

38

MS-RPC Vulnerability

www.pervasivetechnologylabs.iu.edu

39

Flows per Hour

1000000

1200000

1400000

1600000

1800000

2000000

200000

400000

600000

800000

0 8-11 0500 8-11 2200 8-12 1500 8-13 0800 8-14 0100 8-14 1800 8-15 1100 8-16 0400 8-16 2100 08-17 1400 08-18 0700 08-19 0000 08-19 1700 08-20 1000 08-21 0300 08-21 2000 08-22 1300 08-23 0600 08-23 2300 08-24 1600 08-25 0900 08-26 0200 08-26 1900 08-27 1200 08-28 0500 08-28 2200 08-29 1600 08-30 0900 08-31 0200 08-31 1900 09-01 1200 09-02 0600

www.pervasivetechnologylabs.iu.edu

MS-RPC (Attempts)

Infection Attempts over Abilene

Date/Time (UTC)

40

Flows Per Hour

100 120 140 20 40 60 80 0

2003-09-08 09:00:00 2003-09-08 10:00:00 2003-09-08 11:00:00 2003-09-08 12:00:00 2003-09-08 13:00:00 2003-09-08 14:00:00 2003-09-08 15:00:00 2003-09-08 16:00:00 2003-09-08 17:00:00 2003-09-08 18:00:00 2003-09-08 19:00:00 2003-09-08 20:00:00 2003-09-08 21:00:00 2003-09-08 22:00:00 2003-09-08 23:00:00 2003-09-09 00:00:00 2003-09-09 01:00:00 2003-09-09 02:00:00 2003-09-09 03:00:00 2003-09-09 04:00:00 2003-09-09 05:00:00 2003-09-09 06:00:00 2003-09-09 07:00:00 2003-09-09 08:00:00 2003-09-09 09:00:00 2003-09-09 10:00:00 2003-09-09 11:00:00 2003-09-09 12:00:00 2003-09-09 13:00:00 2003-09-09 14:00:00 2003-09-09 15:00:00 2003-09-09 16:00:00 2003-09-09 17:00:00 2003-09-09 18:00:00 2003-09-09 19:00:00 2003-09-09 20:00:00 2003-09-09 21:00:00 2003-09-09 22:00:00 2003-09-09 23:00:00 2003-09-10 00:00:00 2003-09-10 01:00:00 2003-09-10 02:00:00 2003-09-10 03:00:00 2003-09-10 04:00:00 2003-09-10 05:00:00 2003-09-10 06:00:00 2003-09-10 07:00:00 2003-09-10 08:00:00 2003-09-10 09:00:00 2003-09-10 10:00:00

www.pervasivetechnologylabs.iu.edu

Successful New Infections

MS-RPC Infections (Maybe)

Date/Time (UTC)

41

Added Value

www.pervasivetechnologylabs.iu.edu

· Some operations are very computationally cheap and

easy to do as we go along, so it would be silly not to do them.

Traffic to and from different ports, broken down by time. This lets us get basic information quickly and easily, e.g.

netflow=# select * from digested_tcp_dst where portnum=80 order by time_stamp; time_stamp | portnum | count ------------------------+---------+-------2004-01-19 18:15:04-05 | 80 | 240206 2004-01-19 18:30:00-05 | 80 | 230117 2004-01-19 18:45:00-05 | 80 | 210618 2004-01-19 19:00:00-05 | 80 | 201564 2004-01-19 19:15:00-05 | 80 | 191944 etc...

· ·

42

Added Value

www.pervasivetechnologylabs.iu.edu

# of flows/15 minutes, TCP, Destination port 80

600000

500000

400000

300000

200000

100000

0 1/19/04 12:00 1/20/04 0:00 1/20/04 12:00 1/21/04 0:00 1/21/04 12:00 1/22/04 0:00 1/22/04 12:00 1/23/04 0:00 1/23/04 12:00 1/24/04 0:00

Not hugely exciting, but it was free.

43

Performance?

www.pervasivetechnologylabs.iu.edu

· We were feeling quite pleased with ourselves at this

point.

· It quickly became apparent that we were going to have

performance issues.

Storage volume Raw processing power - operations were taking much longer than we might have liked.

· ·

44

Performance?

www.pervasivetechnologylabs.iu.edu

· How could we improve the situation?

· · · · ·

We're having to scan an immense amount of data Indexes makes searching data fast! The First Law of Database Thermodynamics Indexes make inserting data slow! We're having to insert an immense amount of data

45

Performance issues?

www.pervasivetechnologylabs.iu.edu

· What else could we try?

· · ·

Reduce the amount of raw data that we have to process

· · · ·

Keep a rotating archive

This creates more problems than it solved.

You have to delete data; this is expensive. You have to clean up; this is also expensive.

Bigger, faster hardware, distribute tasks.

This would be nice.

46

What else?

www.pervasivetechnologylabs.iu.edu

· We can also try and reduce the volume of data and/or

improve the efficiency of the database with better normalization

Data common across records Single flows generate records from multiple routers. More expense

· · ·

47

What else?

www.pervasivetechnologylabs.iu.edu

· We can do as much pre-processing ahead of time

· · ·

You can never cover all the bases You never know what you're going to be looking for Waste time generating products that you'll never use.

48

What else?

www.pervasivetechnologylabs.iu.edu

· $$$

· ·

Distributed setup for pre-processing, raw archive, post-processing. The sky is the limit.

49

Conclusion

www.pervasivetechnologylabs.iu.edu

· Netflow has the potential to be incredibly useful for

network security and performance maintenance

Although it does have limitations

·

· Maintaining an archive that can be quickly and

efficiently queried would be super

Especially if we can do compound queries But we need all the data - you never know what you're going to be looking for. But that was harder than we hoped.

· · ·

50

Information

50 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate

1137026