Troubleshooting transmission slowness and packet retransmits could be a puzzling task, especially when it's over an IPsec tunnel.
Last week I had the opportunity to troubleshoot a problem with slow website loading times on a webserver across the link. It was difficult to troubleshoot as the site would appear intermittently and was slow to load. A ping or a telnet to the server on the side returned packets swiftly without any issues. I verified that the tunnel was up and was transmitting without any problems. Where could the problem be?
With a simple wireshark capture I found out that retransmissions were occuring very frequently. This was when I found out the packets were fragmented quite a bit and realized that the VPN concentrator had been set with a very small MTU. This was nasty, as it had almost been intentionally tampered with to create an effect of slowness. Such transmission slowness is extremely difficult to troubleshoot as there was no issue with the connectivity itself.
Here's a link with a detailed explanation on how MTU affects performance.
Friday, November 6, 2009
Tuesday, November 3, 2009
100Mb Half Duplex problem between Cisco switch and Linux system
Today I ran into a weird scenario where a Linux system's NIC would set itself to 100Mbit half duplex regardless whether the device and NIC was manually configured to operated at full duplex. After trolling through Cisco's website
http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00800a7af0.shtml
I found out that the problem usually relies on the NIC. Despite both the switch and NIC have been manually set to 100 Full duplex, there is often times when the NIC still runs auto-negotiation in the background. To resolve this problem, I had to manually set the NIC to turn off auto negotiation. Once that was done. The problem went away.
http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00800a7af0.shtml
I found out that the problem usually relies on the NIC. Despite both the switch and NIC have been manually set to 100 Full duplex, there is often times when the NIC still runs auto-negotiation in the background. To resolve this problem, I had to manually set the NIC to turn off auto negotiation. Once that was done. The problem went away.
Thursday, October 15, 2009
Demystifying ASA/PIX Nat 0 vs Static
Firstly, Nat 0 and static can be used to achieve the same result of bypassing NAT, at least logically :)
However both are fundamentally different.
Here's an example:
nat (inside) 0 192.168.1.1 255.255.255.255
and
static (inside,dmz) 192.168.1.1 192.168.1.1 netmask 255.255.255.255
Both statements preserves the IP address 192.168.1.1 for traffic going from inside to dmz.
Statement 1 however is outbound only. Only traffic initiated from the inside gets natted (or bypasses natting). A ping from the DMZ network will not be able to reach the inside host 192.168.1.1 even with ACLs.
Statement 2 however creates a permanent (static) NAT entry in the table. This allows networks in the DMZ zone to access the IP in the inside zone. A ping initiated from the DMZ to 192.168.1.1 is possible if ACL permits.
Hence static is commonly used when traffic needs to flow from a lower security zone to higher security zone. I.e Outside -> DMZ -> Inside.
Where as NAT is used from Higher to lower if you do not want the lower zone traffic to reach back. I.e Inside -> DMZ -> Outside
Note: Don't confuse this with FW stateful inspection. As the FW allows return packets from the destination when initiated by the source.
However both are fundamentally different.
Here's an example:
nat (inside) 0 192.168.1.1 255.255.255.255
and
static (inside,dmz) 192.168.1.1 192.168.1.1 netmask 255.255.255.255
Both statements preserves the IP address 192.168.1.1 for traffic going from inside to dmz.
Statement 1 however is outbound only. Only traffic initiated from the inside gets natted (or bypasses natting). A ping from the DMZ network will not be able to reach the inside host 192.168.1.1 even with ACLs.
Statement 2 however creates a permanent (static) NAT entry in the table. This allows networks in the DMZ zone to access the IP in the inside zone. A ping initiated from the DMZ to 192.168.1.1 is possible if ACL permits.
Hence static is commonly used when traffic needs to flow from a lower security zone to higher security zone. I.e Outside -> DMZ -> Inside.
Where as NAT is used from Higher to lower if you do not want the lower zone traffic to reach back. I.e Inside -> DMZ -> Outside
Note: Don't confuse this with FW stateful inspection. As the FW allows return packets from the destination when initiated by the source.
Friday, May 29, 2009
Cisco ASA allowing management access from VPN
The inside interface of the PIX (also applies to ASA) cannot be accessed from the outside, and vice-versa, unless the management-access is configured in global configuration mode. Once management-access is enabled, Telnet, SSH, or HTTP access must still be configured for the desired hosts.
Wednesday, May 20, 2009
Cisco ASA Failover Tips and misc.
When setting up Failover on Cisco ASA try to follow the following rules & tips:
1) Do not use a crossover Ethernet cable or a fiber-optic patch cable to directly connect the two failover LAN interfaces if the firewalls are located close to each other:
Instead, each interface should connect to a switch port so that the link status is always up to one firewall interface if the other firewall interface fails. Otherwise, both units sense a link-down condition and assume that their own interfaces have a failure.
2) You should also prepare the switch ports where the LAN-based failover interfaces connect so that failover communication can begin almost immediately. You should enable Spanning Tree Protocol PortFast and disable trunking and EtherChannel negotiation. You can use the following IOS Software commands to configure the switch ports:
Switch# configure terminal
Switch(config)# interface type mod/num
! Enable PortFast for immediate traffic forwarding
Switch(config-if)# spanning-tree portfast
! Disable trunking by making it an access switch port
Switch(config-if)# switchport mode access
! Disable EtherChannel negotiation
Switch(config-if)# no channel-group
3) You can use one dedicated LAN interface (10/100 or Gigabit Ethernet) to carry both LAN-based failover and stateful failover information. The interface bandwidth must be large enough to carry the aggregate failover load.
However, it is always best to keep the LAN-based failover and stateful failover data streams on separate interfaces. The stateful failover data stream is usually much larger than the LAN-based failover because of the usually large number of connections that come and go. Therefore, you should set aside the fastest firewall interface that is available for stateful failover.
4) In addition, LAN-based failover messages must be able to travel between the two units without being lost or delayed. Otherwise, the loss of LAN-based failover messages indicates that one or both units have failed.
You can link the two stateful failover interfaces directly with a fiber-optic or crossover patch cord without connecting them to intermediate switches. However, neither firewall unit can determine which unit has had an interface failure, because the link status is lost on both units simultaneously.
The best-practice recommendations stress the need for an active device such as a switch to connect the stateful failover interfaces. If one unit loses an interface, a switch would keep the link status up for the other firewall unit.
5)In the case of FWSMs, they each have a 6-Gbps internal trunk link to the switch backplane. With their high performance, stateful failover information can easily burst up to the link bandwidth. Therefore, if two FWSMs are located in separate chassis, you should provide a stateful failover VLAN link of at least 6 Gbps. You can do this by aggregating Gigabit Ethernet links into a Gigabit EtherChannel.
6) All stateful failover updates are sent and received over the interface named if_name (stateful, for example). Stateful failover can share the same interface as LAN-based failover if needed. However, you should always try to keep stateful and LAN-based failover isolated on two separate interfaces set aside for these purposes.
1) Do not use a crossover Ethernet cable or a fiber-optic patch cable to directly connect the two failover LAN interfaces if the firewalls are located close to each other:
Instead, each interface should connect to a switch port so that the link status is always up to one firewall interface if the other firewall interface fails. Otherwise, both units sense a link-down condition and assume that their own interfaces have a failure.
2) You should also prepare the switch ports where the LAN-based failover interfaces connect so that failover communication can begin almost immediately. You should enable Spanning Tree Protocol PortFast and disable trunking and EtherChannel negotiation. You can use the following IOS Software commands to configure the switch ports:
Switch# configure terminal
Switch(config)# interface type mod/num
! Enable PortFast for immediate traffic forwarding
Switch(config-if)# spanning-tree portfast
! Disable trunking by making it an access switch port
Switch(config-if)# switchport mode access
! Disable EtherChannel negotiation
Switch(config-if)# no channel-group
3) You can use one dedicated LAN interface (10/100 or Gigabit Ethernet) to carry both LAN-based failover and stateful failover information. The interface bandwidth must be large enough to carry the aggregate failover load.
However, it is always best to keep the LAN-based failover and stateful failover data streams on separate interfaces. The stateful failover data stream is usually much larger than the LAN-based failover because of the usually large number of connections that come and go. Therefore, you should set aside the fastest firewall interface that is available for stateful failover.
4) In addition, LAN-based failover messages must be able to travel between the two units without being lost or delayed. Otherwise, the loss of LAN-based failover messages indicates that one or both units have failed.
You can link the two stateful failover interfaces directly with a fiber-optic or crossover patch cord without connecting them to intermediate switches. However, neither firewall unit can determine which unit has had an interface failure, because the link status is lost on both units simultaneously.
The best-practice recommendations stress the need for an active device such as a switch to connect the stateful failover interfaces. If one unit loses an interface, a switch would keep the link status up for the other firewall unit.
5)In the case of FWSMs, they each have a 6-Gbps internal trunk link to the switch backplane. With their high performance, stateful failover information can easily burst up to the link bandwidth. Therefore, if two FWSMs are located in separate chassis, you should provide a stateful failover VLAN link of at least 6 Gbps. You can do this by aggregating Gigabit Ethernet links into a Gigabit EtherChannel.
6) All stateful failover updates are sent and received over the interface named if_name (stateful, for example). Stateful failover can share the same interface as LAN-based failover if needed. However, you should always try to keep stateful and LAN-based failover isolated on two separate interfaces set aside for these purposes.
Thursday, April 2, 2009
VMWare ESXi secret console
VMWare's ESXi is a free version and lightweight version of the VMWares ESX platform. It supports most of the key features that ESX has but comes in appliance like package which doesn't let you configure anything more than just the basic hostname/IP settings. THIS IS NOT TRUE! Infact, ESXi is built from a similar linux variant as the ESX. This is a hidden secret feature.
To reach this secret console go through the following steps.
1) Get to the ESXi console (TUI - Text User interface)
2) Press Alt - F1
3) Enter "unsupported" and hit enter (You will not see any typed)
4) Login in with your root password
5) Congratulations! you have unlocked the secret console!
With this console you can enable ssh with the following steps
1) Login with the above steps
2) Edit the /etc/inetd.conf file
3) Uncomment the SSH session and save the file
4) ps |grep inetd (to get the PID)
5) kill -HUP (to reload the inetd process)
6) Login with your favourite ssh client!
To reach this secret console go through the following steps.
1) Get to the ESXi console (TUI - Text User interface)
2) Press Alt - F1
3) Enter "unsupported" and hit enter (You will not see any typed)
4) Login in with your root password
5) Congratulations! you have unlocked the secret console!
With this console you can enable ssh with the following steps
1) Login with the above steps
2) Edit the /etc/inetd.conf file
3) Uncomment the SSH session and save the file
4) ps |grep inetd (to get the PID)
5) kill -HUP
6) Login with your favourite ssh client!
Monday, February 16, 2009
PIX Firewall operating in one arm configuration
I have been befuddled once with setting the Cisco PIX/ASA in a one arm routing configuration. To picture below explains this:

Basically when you set the ASA as a default gateway, you are unable to have to ASA forward (or route) packets in and out of the same interface. This was the case in version 6.0 of the PIX OS. However with version 7.2(1) and later it can now be done via the command "same-security-traffic permit intra-interface".
A link with the how to can be found on the cisco website

Basically when you set the ASA as a default gateway, you are unable to have to ASA forward (or route) packets in and out of the same interface. This was the case in version 6.0 of the PIX OS. However with version 7.2(1) and later it can now be done via the command "same-security-traffic permit intra-interface".
A link with the how to can be found on the cisco website
Subscribe to:
Posts (Atom)