Written by: Nick Stockhaus
From Eye of the Storm - November 2008

FW MONITOR-FTW?

      So tcpdump might seem like a pain in the rear to some because you might have 12 interfaces and need answers NOW, so having to figure out which interfaces you should be looking at isn’t always a high priority. This is a good time to introduce the fw monitor command, because it doesn’t care about interfaces. The fw monitor command is a great tool to use for diagnosing routing and most NAT related issues. The common syntax for fw monitor is: fw monitor -e 'accept src=x.x.x.x or dst=x.x.x.x;' This command can be a little more straight-forward compared to tcpdump, but the results are more confusing until you understand what they mean. The only variables you need to worry about for basic fw monitor troubleshooting are the “src=” and “dst=” arguments. The above example will show both forward traffic and return traffic to/from a single host. This differs from tcpdump where a simple 'host x.x.x.x' would do the same, but provide greater flexibility when looking for more specific connections. You may also get more descriptive in your filter to lock down just connections between x.x.x.x and y.y.y.y by issuing the filter: 'accept (src=x.x.x.x or dst=x.x.x.x) and (src=y.y.y.y or dst=y.y.y.y);' The major benefit with fw monitor is that it shows all stages of the connection within a single capture command. Here is an example of fw monitor vs tcpdump for a single PING request from x.x.x.x to 4.2.2.2:

[Expert@firewall]#fw monitor -e 'accept src=4.2.2.2 or dst=4.2.2.2;' eth1:i[84]: x.x.x.x -> 4.2.2.2 (ICMP) len=84 id=0 ICMP: type=8 code=0 echo request id=51029 seq=1 eth1:I[84]: x.x.x.x -> 4.2.2.2 (ICMP) len=84 id=45984 ICMP: type=8 code=0 echo request id=51029 seq=1 eth0:o[84]: x.x.x.x -> 4.2.2.2 (ICMP) len=84 id=45984 ICMP: type=8 code=0 echo request id=51029 seq=1 eth0:O[84]: N.N.N.N -> 4.2.2.2 (ICMP) len=84 id=45984

       Notice how there are four entries for a single request - two for each interface. Look just after the interface name (eth1, eth0) it shows either an i,I,o,O. The i indicates that the packet was inbound on eth1 and unprocessed by the Check Point software, or ‘on the wire’. The I is that same packet after being processed by the software. Since nothing differs it means no changes were made. The o shows the packet outbound on eth0 before being processed. The notice how the O shows a different source IP address, N.N.N.N. The O shows traffic after being processed by the Check Point software, in this case it applied a Hide NAT to the traffic since it was Internet bound. Both i and O are traffic as it appears ‘on the wire’, or as tcpdump would see it. Compare this to tcpdump below looking for the same traffic:

[Expert@firewall]#tcpdump -ni eth1 host x.x.x.x 11:08:12.977134 x.x.x.x > 4.2.2.2: icmp: echo request (DF)

      In this instance, I am sniffing on the internal interface for traffic from x.x.x.x. It appears as expected.

[Expert@firewall]#tcpdump -ni eth0 host x.x.x.x

      In this instance, I’ve run the same capture but on the external interface. This time I see nothing because the traffic has had the Hide NAT applied before hitting the wire, so the source x.x.x.x is not seen.

[Expert@firewall]#tcpdump -ni eth1 host 4.2.2.2 11:07:26.636870 N.N.N.N > 4.2.2.2: icmp: echo request (DF)

       By changing the host filter to the destination IP, 4.2.2.2, we see the PING request, which is now sourced as N.N.N.N, or the public IP of the firewall.

      It took three tcpdump commands to figure out less than what one fw monitor command told us. That might not be much more than some additional typing to some, but if you don’t already know to look specifically for this, then hours could elapse between the first and third tcpdump command. Trust me, I have seen it happen.

      What else did fw monitor show us that tcpdump did not? It shows us the routing. Let say that your ping was failing, but the SmartView Tracker logs showed it as being accepted. Assuming that this is the same firewall as the above captures, what if you ran fw monitor and it showed the following:

[Expert@firewall]#fw monitor -e 'accept src=4.2.2.2 or dst=4.2.2.2;' eth1:i[84]: x.x.x.x -> 4.2.2.2 (ICMP) len=84 id=0 ICMP: type=8 code=0 echo request id=51029 seq=1 eth1:I[84]: x.x.x.x -> 4.2.2.2 (ICMP) len=84 id=45984 ICMP: type=8 code=0 echo request id=51029 seq=1 eth2:o[84]: x.x.x.x -> 4.2.2.2 (ICMP) len=84 id=45984 ICMP: type=8 code=0 echo request id=51029 seq=1 eth2:O[84]: N.N.N.N -> 4.2.2.2 (ICMP) len=84 id=45984

       Only one thing has changed between these two sets of fw monitor results: Where the first set showed eth0 for o and O, now it shows eth2. Why would this be, considering that eth0 is the external interface? My first guess would be to check the routing table, perhaps somebody mis-routed something. Either way, fw monitor has tipped you off to a potential problem and you now have direction to follow to continue troubleshooting. Next month: TCPDUMP...Round 2!