netflow v9

Netflow V9

July 31st, 2013,由 frank撰写

最近研究了一下Netflow，发现这东西还是很不错的。建议如果流量异常有问题或负载分担问题，直接让客户配上monitor，不用配置exporter，直接看cache里的内容就可以了。Netflow里面有些迷惑的地方，这回整个测试了下，发现这东西还是比较有意思的。

1. Template:

在v9的版本中，引进了template这个概念，这个东西感觉像个索引，告诉网管netflow的架构，默认export时间为1800s，也就是30mi，在抓netflow报文时，如果没有抓下template，那么你会发现通过wireshark无法解开cflow的报文，你也就没发分析了。另外template分两种，一种是正常的数据flow，另一种是option的flow，这就带来了强大的扩展。在下面的信息里，我把template的timeout改成了10s，cache timeout没改，所以你发现短时间内没有抓到任何flow流量：

RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0
Wed Jul 24 08:02:27.314 UTC
Flow Exporter: test 
Flow Exporter memory usage: 3280868
Used by flow monitors: test-monitor

Status: Normal
Transport   UDP
Destination 12.1.1.1        (9995)      VRF default         
Source      123.123.123.123 (29117)
Flows exported:                                   0 (0 bytes) >>> 刚开始为空 Flows dropped:                                    0 (0 bytes)

Templates exported:                               7 (644 bytes) >>> 数据template已经exported 7个，对应Flows exported Templates dropped:                                0 (0 bytes)

Option data exported:                             0 (0 bytes) >>> 刚开始为空 Option data dropped:                              0 (0 bytes)

Option templates exported:                       14 (392 bytes) >>> Option template已经exported 14个，对应Option data exported Option templates dropped:                         0 (0 bytes)

Packets exported:                                21 (1036 bytes)
Packets dropped:                                  0 (0 bytes)

Total export over last interval of:
  1 hour:                                        18 pkts
                                                888 bytes
                                                  0 flows
  1 minute:                                      18 pkts
                                                888 bytes
                                                  0 flows
  1 second:                                       0 pkts
                                                  0 bytes
                                                  0 flows

RP/0/RSP0/CPU0:ios#sh flow exporter-map test 
Wed Jul 24 11:28:59.354 UTC

Flow Exporter Map : test
-------------------------------------------------
Id                 : 1
DestinationIpAddr   : 12.1.1.1
SourceIfName        : Loopback0
SourceIpAddr        : 123.123.123.123
DSCP                : 0
TransportProtocol   : UDP
TransportDestPort   : 9995

Export Version: 9
  Common Template Timeout : 10 seconds
  Options Template Timeout : 10 seconds
  Data Template Timeout : 10 seconds
  Interface-Table Export Timeout : 0 seconds
  Sampler-Table Export Timeout : 0 seconds

RP/0/RSP0/CPU0:ios#show flow monitor-map test-monitor 
Wed Jul 24 11:31:12.103 UTC

Flow Monitor Map : test-monitor
-------------------------------------------------
Id:                1
RecordMapName:     ipv4-raw
ExportMapName:     test
CacheAgingMode:    Normal
CacheMaxEntries:   1000000
CacheActiveTout:   1800 seconds
CacheInactiveTout: 15 seconds
CacheUpdateTout:   N/A

详细请看附件 “flow-test-1.pcap”

2. Export timeout：

在Netflow中，有多个export的时间点。
当流是active的时候，到达ActiveTimeout，就会把现有cache中这些active的数据组成UDP并发送出去，默认30mi；
当流是inactive的时候（就是在设备上没有这个流的任何数据包了），到达InactiveTimeout，就会把这些inactive的数据包组成UDP发送出去，默认15s，这个很快，所以如果遇到间接性的攻击流时，很难去cache中发现问题flow。

>>>>>>>>> 测试Inactive Timeout <<<<<<<<<

RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0
Wed Jul 24 11:36:53.723 UTC
Cache summary for Flow Monitor test-monitor:
Cache size:                        1000000
Current entries:                         0
High Watermark:                     950000
Flows added:                             8
Flows not added:                         0
Ager Polls:                          12919
  - Active timeout                       7
  - Inactive timeout                     1
  - TCP FIN flag                         0
  - Watermark aged                       0
  - Emergency aged                       0
  - Counter wrap aged                    0
  - Total                                8
Periodic export:
  - Counter wrap                         0
  - TCP FIN flag                         0
Flows exported                           8

Matching entries:                        0 >>>现在没有流量 RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0                   
Wed Jul 24 11:37:08.309 UTC
Flow Exporter: test 
Flow Exporter memory usage: 3280868
Used by flow monitors: test-monitor

Status: Normal
Transport   UDP
Destination 12.1.1.1        (9995)      VRF default         
Source      123.123.123.123 (29117)
Flows exported:                                   8 (456 bytes) >>>做测试时没有清掉，所以是8 Flows dropped:                                    0 (0 bytes)

Templates exported:                            1295 (119140 bytes)
Templates dropped:                                0 (0 bytes)

Option data exported:                             0 (0 bytes)
Option data dropped:                              0 (0 bytes)

Option templates exported:                     2588 (72464 bytes)
Option templates dropped:                         0 (0 bytes)

Packets exported:                              3891 (192572 bytes)
Packets dropped:                                  0 (0 bytes)

**************************开始打流量*****************************

RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0
Wed Jul 24 11:37:11.493 UTC
Flow Exporter: test 
Flow Exporter memory usage: 3280868
Used by flow monitors: test-monitor

Status: Normal
Transport   UDP
Destination 12.1.1.1        (9995)      VRF default         
Source      123.123.123.123 (29117)
Flows exported:                                   8 (456 bytes) >>> 没有变化 Flows dropped:                                    0 (0 bytes)

Templates exported:                            1296 (119232 bytes)
Templates dropped:                                0 (0 bytes)

Option data exported:                             0 (0 bytes)
Option data dropped:                              0 (0 bytes)

Option templates exported:                     2588 (72464 bytes)
Option templates dropped:                         0 (0 bytes)

Packets exported:                              3891 (192572 bytes)
Packets dropped:                                  0 (0 bytes)

RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0
Wed Jul 24 11:37:15.053 UTC
Cache summary for Flow Monitor test-monitor:
Cache size:                        1000000
Current entries:                         1
High Watermark:                     950000
Flows added:                             9
Flows not added:                         0
Ager Polls:                          12940
  - Active timeout                       7
  - Inactive timeout                     1
  - TCP FIN flag                         0
  - Watermark aged                       0
  - Emergency aged                       0
  - Counter wrap aged                    0
  - Total                                8
Periodic export:
  - Counter wrap                         0
  - TCP FIN flag                         0
Flows exported                           8

IPV4SrcAddr      IPV4DstAddr      L4SrcPort  L4DestPort IPV4Prot IPV4TOS  InputInterface  ForwardStatus        ByteCount    PacketCount  Dir 
8.8.8.8          10.144.254.1     63         63         udp      0        Gi0/2/0/10      Fwd                  21870        135          Ing 

Matching entries:                        1 <<< 在cache里已经能看到了

**************************停止打流量*****************************

RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0
Wed Jul 24 11:37:22.749 UTC
Cache summary for Flow Monitor test-monitor:
Cache size:                        1000000
Current entries:                         1
High Watermark:                     950000
Flows added:                             9
Flows not added:                         0
Ager Polls:                          12948
  - Active timeout                       7
  - Inactive timeout                     1
  - TCP FIN flag                         0
  - Watermark aged                       0
  - Emergency aged                       0
  - Counter wrap aged                    0
  - Total                                8
Periodic export:
  - Counter wrap                         0
  - TCP FIN flag                         0
Flows exported                           8

IPV4SrcAddr      IPV4DstAddr      L4SrcPort  L4DestPort IPV4Prot IPV4TOS  InputInterface  ForwardStatus        ByteCount    PacketCount  Dir 
8.8.8.8          10.144.254.1     63         63         udp      0        Gi0/2/0/10      Fwd                  31914        197          Ing 

Matching entries:                        1 <<< 还存在 RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0                   
Wed Jul 24 11:37:26.684 UTC
Flow Exporter: test 
Flow Exporter memory usage: 3280868
Used by flow monitors: test-monitor

Status: Normal
Transport   UDP
Destination 12.1.1.1        (9995)      VRF default         
Source      123.123.123.123 (29117)
Flows exported:                                   8 (456 bytes) >>>仍然没变 Flows dropped:                                    0 (0 bytes)

Templates exported:                            1297 (119324 bytes)
Templates dropped:                                0 (0 bytes)

Option data exported:                             0 (0 bytes)
Option data dropped:                              0 (0 bytes)

Option templates exported:                     2592 (72576 bytes)
Option templates dropped:                         0 (0 bytes)

Packets exported:                              3897 (192868 bytes)
Packets dropped:                                  0 (0 bytes)

RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0
Wed Jul 24 11:37:32.666 UTC
Cache summary for Flow Monitor test-monitor:
Cache size:                        1000000
Current entries:                         1
High Watermark:                     950000
Flows added:                             9
Flows not added:                         0
Ager Polls:                          12957
  - Active timeout                       7
  - Inactive timeout                     1
  - TCP FIN flag                         0
  - Watermark aged                       0
  - Emergency aged                       0
  - Counter wrap aged                    0
  - Total                                8
Periodic export:
  - Counter wrap                         0
  - TCP FIN flag                         0
Flows exported                           8

IPV4SrcAddr      IPV4DstAddr      L4SrcPort  L4DestPort IPV4Prot IPV4TOS  InputInterface  ForwardStatus        ByteCount    PacketCount  Dir 
8.8.8.8          10.144.254.1     63         63         udp      0        Gi0/2/0/10      Fwd                  34020        210          Ing 

Matching entries:                        1
RP/0/RSP0/CPU0:ios#show flow monitor test-monitor cache brief location 0/2/cpu0
Wed Jul 24 11:37:35.756 UTC
Cache summary for Flow Monitor test-monitor:
Cache size:                        1000000
Current entries:                         0
High Watermark:                     950000
Flows added:                             9
Flows not added:                         0
Ager Polls:                          12961
  - Active timeout                       7
  - Inactive timeout                     2
  - TCP FIN flag                         0
  - Watermark aged                       0
  - Emergency aged                       0
  - Counter wrap aged                    0
  - Total                                9
Periodic export:
  - Counter wrap                         0
  - TCP FIN flag                         0
Flows exported                           9

Matching entries:                        0 >>> 被clear了，去看下export信息 RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0                   
Wed Jul 24 11:37:38.797 UTC
Flow Exporter: test 
Flow Exporter memory usage: 3280868
Used by flow monitors: test-monitor

Status: Normal
Transport   UDP
Destination 12.1.1.1        (9995)      VRF default         
Source      123.123.123.123 (29117)
Flows exported:                                   9 (513 bytes) >>> 增加到9了 Flows dropped:                                    0 (0 bytes)

Templates exported:                            1299 (119508 bytes)
Templates dropped:                                0 (0 bytes)

Option data exported:                             0 (0 bytes)
Option data dropped:                              0 (0 bytes)

Option templates exported:                     2594 (72632 bytes)
Option templates dropped:                         0 (0 bytes)

Packets exported:                              3902 (193229 bytes)
Packets dropped:                                  0 (0 bytes)

>>>>>>>>> 测试Active Timeout <<<<<<<<<

RP/0/RSP0/CPU0:ios#config ter
Wed Jul 24 11:45:34.523 UTC
RP/0/RSP0/CPU0:ios(config)#flow monitor-map test-monitor 
RP/0/RSP0/CPU0:ios(config-fmm)#cache ?
  entries    Specify the number of entries in the flow cache
  permanent  Disable removal of entries from flow cache
  timeout    Specify the flow cache timeouts
RP/0/RSP0/CPU0:ios(config-fmm)#cache timeout ?
  active    Specify the active flow timeout
  inactive  Specify the inactive flow timeout
  update    Specify the update timeout
RP/0/RSP0/CPU0:ios(config-fmm)#cache timeout active 30 
RP/0/RSP0/CPU0:ios(config-fmm)#commit
Wed Jul 24 11:47:20.443 UTC
RP/0/RSP0/CPU0:ios(config-fmm)#end
RP/0/RSP0/CPU0:ios#
RP/0/RSP0/CPU0:ios#sh run int g0/2/0/10
Wed Jul 24 11:47:35.958 UTC
interface GigabitEthernet0/2/0/10
 ipv4 address 33.33.33.1 255.255.255.0
 negotiation auto
 load-interval 30
 flow ipv4 monitor test-monitor sampler 1-1000 ingress
!

RP/0/RSP0/CPU0:ios#config ter
Wed Jul 24 11:47:37.483 UTC
RP/0/RSP0/CPU0:ios(config)#int g0/2/0/10
RP/0/RSP0/CPU0:ios(config-if)#no flow ipv4 monitor test-monitor sampler 1-1000 ingress
RP/0/RSP0/CPU0:ios(config-if)#commit
Wed Jul 24 11:47:50.972 UTC
RP/0/RSP0/CPU0:ios(config-if)#do sh flow monitor-map test-monitor 
Wed Jul 24 11:48:29.900 UTC

Flow Monitor Map : test-monitor
-------------------------------------------------
Id:                1
RecordMapName:     ipv4-raw
ExportMapName:     test
CacheAgingMode:    Normal
CacheMaxEntries:   1000000
CacheActiveTout:   30 seconds >>>已经改了 CacheInactiveTout: 15 seconds
CacheUpdateTout:   N/A
RP/0/RSP0/CPU0:ios(config-if)#flow ipv4 monitor test-monitor sampler 1-1000 ingress
RP/0/RSP0/CPU0:ios(config-if)#commit
Wed Jul 24 11:48:47.966 UTC >>>开始计时 RP/0/RSP0/CPU0:ios(config-if)#end
RP/0/RSP0/CPU0:ios#
RP/0/RSP0/CPU0:ios#

RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0
Wed Jul 24 11:49:22.123 UTC >>> 大概过了35s Flow Exporter: test 
Flow Exporter memory usage: 3280868
Used by flow monitors: test-monitor

Status: Normal
Transport   UDP
Destination 12.1.1.1        (9995)      VRF default         
Source      123.123.123.123 (30838)
Flows exported:                                   1 (57 bytes) >>>已经被export出去了 Flows dropped:                                    0 (0 bytes)

Templates exported:                               5 (460 bytes)
Templates dropped:                                0 (0 bytes)

Option data exported:                             0 (0 bytes)
Option data dropped:                              0 (0 bytes)

Option templates exported:                        8 (224 bytes)
Option templates dropped:                         0 (0 bytes)

Packets exported:                                14 (805 bytes)
Packets dropped:                                  0 (0 bytes)

Total export over last interval of:
  1 hour:                                         0 pkts
                                                  0 bytes
                                                  0 flows
  1 minute:                                      14 pkts
                                                748 bytes
                                                  1 flows
  1 second:                                       2 pkts
                                                156 bytes
                                                  1 flows

RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0
Wed Jul 24 11:49:53.291 UTC >>> 大概又过了30s Flow Exporter: test 
Flow Exporter memory usage: 3280868
Used by flow monitors: test-monitor

Status: Normal
Transport   UDP
Destination 12.1.1.1        (9995)      VRF default         
Source      123.123.123.123 (30838)
Flows exported:                                   2 (114 bytes) >>>已经被export出去了 Flows dropped:                                    0 (0 bytes)

Templates exported:                               8 (736 bytes)
Templates dropped:                                0 (0 bytes)

Option data exported:                             0 (0 bytes)
Option data dropped:                              0 (0 bytes)

Option templates exported:                       14 (392 bytes)
Option templates dropped:                         0 (0 bytes)

Packets exported:                                24 (1370 bytes)
Packets dropped:                                  0 (0 bytes)

>> 再次抓包，并在端口上初始化flow配置，确认template和正常的cflow信息 <<
抓包信息请看: “flow-test-2.pcap”

RP/0/RSP0/CPU0:ios#show flow exporter test location 0/2/cpu0
Wed Jul 24 12:10:10.049 UTC
Flow Exporter: test 
Flow Exporter memory usage: 3280868
Used by flow monitors: test-monitor

Status: Normal
Transport   UDP
Destination 12.1.1.1        (9995)      VRF default         
Source      123.123.123.123 (381)
Flows exported:                                   3 (171 bytes) >>> 跟抓包信息对应 Flows dropped:                                    0 (0 bytes)

Templates exported:                              15 (1380 bytes) >>> 跟抓包信息对应 Templates dropped:                                0 (0 bytes)

Option data exported:                             0 (0 bytes) >>> 可以看到只有option的templates，没有option的data Option data dropped:                              0 (0 bytes)

Option templates exported:                       26 (728 bytes)  
Option templates dropped:                         0 (0 bytes)

Packets exported:                                43 (2379 bytes)
Packets dropped:                                  0 (0 bytes)

Total export over last interval of:
  1 hour:                                        41 pkts
                                               2152 bytes
                                                  3 flows
  1 minute:                                      20 pkts
                                               1044 bytes
                                                  1 flows
  1 second:                                       0 pkts
                                                  0 bytes
                                                  0 flows

P/0/RSP0/CPU0:ios#show flow monitor-map test-monitor       
Wed Jul 24 12:14:55.437 UTC

Flow Monitor Map : test-monitor
-------------------------------------------------
Id:                1
RecordMapName:     ipv4-raw
ExportMapName:     test
CacheAgingMode:    Normal
CacheMaxEntries:   1000000
CacheActiveTout:   30 seconds
CacheInactiveTout: 15 seconds
CacheUpdateTout:   N/A
RP/0/RSP0/CPU0:ios#show flow exporter-map test 
Wed Jul 24 12:15:07.023 UTC

Flow Exporter Map : test
-------------------------------------------------
Id                 : 1
DestinationIpAddr   : 12.1.1.1
SourceIfName        : Loopback0
SourceIpAddr        : 123.123.123.123
DSCP                : 0
TransportProtocol   : UDP
TransportDestPort   : 9995

Export Version: 9
  Common Template Timeout : 10 seconds
  Options Template Timeout : 10 seconds
  Data Template Timeout : 10 seconds
  Interface-Table Export Timeout : 0 seconds
  Sampler-Table Export Timeout : 0 seconds

3. Cache中的flow

当在端口配上“flow ipv4 monitor xxx sampler xxx ingress”时，就会激活netflow，进来的数据包会首先进到flow cache中，这个cache可以更改，最大1000000个条目。这里面的每个条目在被export出去之前都是累加的，例如在下面的信息中，我用了sample 1/1000。

RP/0/RSP0/CPU0:ios#sh flow monitor test-monitor cache location 0/2/cpu0
Thu Jul 25 16:43:29.250 UTC
Cache summary for Flow Monitor test-monitor:
Cache size:                        1000000
Current entries:                         1
High Watermark:                     950000
Flows added:                             7
Flows not added:                         0
Ager Polls:                           1776
  - Active timeout                       5
  - Inactive timeout                     1
  - TCP FIN flag                         0
  - Watermark aged                       0
  - Emergency aged                       0
  - Counter wrap aged                    0
  - Total                                6
Periodic export:
  - Counter wrap                         0
  - TCP FIN flag                         0
Flows exported                           6

IPV4SrcAddr      IPV4DstAddr      L4SrcPort  L4DestPort BGPDstOrigAS BGPSrcOrigAS BGPNextHopV4     
IPV4DstPrfxLen  IPV4SrcPrfxLen  IPV4Prot IPV4TOS  InputInterface  OutputInterface 
L4TCPFlags   ForwardStatus        FirstSwitched   LastSwitched ByteCount PacketCount Dir SamplerID
8.8.8.8          10.144.254.1     63         63         0            0            0.0.0.0          
24              0               udp      0        Gi0/2/0/10      Gi0/2/0/0       
0            Fwd                  00 00:27:24:562 00 00:31:09:544 1034954 22499 Ing 1

Matching entries:                        1

在这里可以看到，22499是累计值，就像我上面说的，他会一直累积到Timeout并Export，另外这也是sample后的数据包个数，所以此时真实的数据包大概应该有22499000，但你可能会发现这个数值跟端口的 packets input计数不一样，即使同时clear端口和flow，也不一样，而且差很多。端口数据和netflow的数据应该是通过不同方式获得的，所以可能会有些延时差异。

4. 如何通过netflow信息绘图

例如下面的，ActiveTimout是300s，实际是317s，所以此流的平均数据包为33358000/317s = 105230 PPS，这个数也大致等于端口的100k PPS。
对于字节数，可以通过 (1534468 * 8)/300s = 40919147bits/sec，大致等于端口统计的48Mbits/sec。
对于这个平均值要打上什么时间，可以是flow的start时间，也可以是flow的end时间，只要统一就ok。
这样一个点就绘出来了，在lab我只打了1条流，如果多个流存在，要把所有流加起来求平均值，这样就可以计算出端口总共的带宽。

另外默认netflow不计算2层包头，所以如果打64byte的小包，而且流量很大，那么统计出来的结果会差很多，因为每个包用于计算的大小只有64-14=50byte。

RP/0/RSP0/CPU0:ios#sh int g0/2/0/10 | i input
Thu Jul 25 16:31:20.059 UTC
  output flow control is off, input flow control is off
  Last input 00:00:00, output 00:17:08
  30 second input rate 48000000 bits/sec, 100000 packets/sec 105600753 packets input, 6336051711 bytes, 0 total input drops
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort