Springdom Kingdom has moved!

Hey I've moved to a much better looking and much better managed blogging platform. There you will find all my previous posts and new ones! What are you waiting for? Checkout www.alfredtong.com now!

Troubleshooting VPN slowness - A look at MTU

Troubleshooting transmission slowness and packet retransmits could be a puzzling task, especially when it's over an IPsec tunnel.

Last week I had the opportunity to troubleshoot a problem with slow website loading times on a webserver across the link. It was difficult to troubleshoot as the site would appear intermittently and was slow to load. A ping or a telnet to the server on the side returned packets swiftly without any issues. I verified that the tunnel was up and was transmitting without any problems. Where could the problem be?

With a simple wireshark capture I found out that retransmissions were occuring very frequently. This was when I found out the packets were fragmented quite a bit and realized that the VPN concentrator had been set with a very small MTU. This was nasty, as it had almost been intentionally tampered with to create an effect of slowness. Such transmission slowness is extremely difficult to troubleshoot as there was no issue with the connectivity itself.

Here's a link with a detailed explanation on how MTU affects performance.

100Mb Half Duplex problem between Cisco switch and Linux system

Today I ran into a weird scenario where a Linux system's NIC would set itself to 100Mbit half duplex regardless whether the device and NIC was manually configured to operated at full duplex. After trolling through Cisco's website

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00800a7af0.shtml

I found out that the problem usually relies on the NIC. Despite both the switch and NIC have been manually set to 100 Full duplex, there is often times when the NIC still runs auto-negotiation in the background. To resolve this problem, I had to manually set the NIC to turn off auto negotiation. Once that was done. The problem went away.

Demystifying ASA/PIX Nat 0 vs Static

Firstly, Nat 0 and static can be used to achieve the same result of bypassing NAT, at least logically :)

However both are fundamentally different.

Here's an example:

nat (inside) 0 192.168.1.1 255.255.255.255

and

static (inside,dmz) 192.168.1.1 192.168.1.1 netmask 255.255.255.255

Both statements preserves the IP address 192.168.1.1 for traffic going from inside to dmz.

Statement 1 however is outbound only. Only traffic initiated from the inside gets natted (or bypasses natting). A ping from the DMZ network will not be able to reach the inside host 192.168.1.1 even with ACLs.

Statement 2 however creates a permanent (static) NAT entry in the table. This allows networks in the DMZ zone to access the IP in the inside zone. A ping initiated from the DMZ to 192.168.1.1 is possible if ACL permits.

Hence static is commonly used when traffic needs to flow from a lower security zone to higher security zone. I.e Outside -> DMZ -> Inside.

Where as NAT is used from Higher to lower if you do not want the lower zone traffic to reach back. I.e Inside -> DMZ -> Outside

Note: Don't confuse this with FW stateful inspection. As the FW allows return packets from the destination when initiated by the source.

Cisco ASA allowing management access from VPN

The inside interface of the PIX (also applies to ASA) cannot be accessed from the outside, and vice-versa, unless the management-access is configured in global configuration mode. Once management-access is enabled, Telnet, SSH, or HTTP access must still be configured for the desired hosts.

Cisco ASA Failover Tips and misc.

When setting up Failover on Cisco ASA try to follow the following rules & tips:

1) Do not use a crossover Ethernet cable or a fiber-optic patch cable to directly connect the two failover LAN interfaces if the firewalls are located close to each other:

Instead, each interface should connect to a switch port so that the link status is always up to one firewall interface if the other firewall interface fails. Otherwise, both units sense a link-down condition and assume that their own interfaces have a failure.

2) You should also prepare the switch ports where the LAN-based failover interfaces connect so that failover communication can begin almost immediately. You should enable Spanning Tree Protocol PortFast and disable trunking and EtherChannel negotiation. You can use the following IOS Software commands to configure the switch ports:

Switch# configure terminal
Switch(config)# interface type mod/num
! Enable PortFast for immediate traffic forwarding
Switch(config-if)# spanning-tree portfast
! Disable trunking by making it an access switch port
Switch(config-if)# switchport mode access
! Disable EtherChannel negotiation
Switch(config-if)# no channel-group

3) You can use one dedicated LAN interface (10/100 or Gigabit Ethernet) to carry both LAN-based failover and stateful failover information. The interface bandwidth must be large enough to carry the aggregate failover load.

However, it is always best to keep the LAN-based failover and stateful failover data streams on separate interfaces. The stateful failover data stream is usually much larger than the LAN-based failover because of the usually large number of connections that come and go. Therefore, you should set aside the fastest firewall interface that is available for stateful failover.

4) In addition, LAN-based failover messages must be able to travel between the two units without being lost or delayed. Otherwise, the loss of LAN-based failover messages indicates that one or both units have failed.

You can link the two stateful failover interfaces directly with a fiber-optic or crossover patch cord without connecting them to intermediate switches. However, neither firewall unit can determine which unit has had an interface failure, because the link status is lost on both units simultaneously.

The best-practice recommendations stress the need for an active device such as a switch to connect the stateful failover interfaces. If one unit loses an interface, a switch would keep the link status up for the other firewall unit.

5)In the case of FWSMs, they each have a 6-Gbps internal trunk link to the switch backplane. With their high performance, stateful failover information can easily burst up to the link bandwidth. Therefore, if two FWSMs are located in separate chassis, you should provide a stateful failover VLAN link of at least 6 Gbps. You can do this by aggregating Gigabit Ethernet links into a Gigabit EtherChannel.

6) All stateful failover updates are sent and received over the interface named if_name (stateful, for example). Stateful failover can share the same interface as LAN-based failover if needed. However, you should always try to keep stateful and LAN-based failover isolated on two separate interfaces set aside for these purposes.

VMWare ESXi secret console

VMWare's ESXi is a free version and lightweight version of the VMWares ESX platform. It supports most of the key features that ESX has but comes in appliance like package which doesn't let you configure anything more than just the basic hostname/IP settings. THIS IS NOT TRUE! Infact, ESXi is built from a similar linux variant as the ESX. This is a hidden secret feature.

To reach this secret console go through the following steps.
1) Get to the ESXi console (TUI - Text User interface)
2) Press Alt - F1
3) Enter "unsupported" and hit enter (You will not see any typed)
4) Login in with your root password
5) Congratulations! you have unlocked the secret console!

With this console you can enable ssh with the following steps
1) Login with the above steps
2) Edit the /etc/inetd.conf file
3) Uncomment the SSH session and save the file
4) ps |grep inetd (to get the PID)
5) kill -HUP (to reload the inetd process)
6) Login with your favourite ssh client!

PIX Firewall operating in one arm configuration

I have been befuddled once with setting the Cisco PIX/ASA in a one arm routing configuration. To picture below explains this:



Basically when you set the ASA as a default gateway, you are unable to have to ASA forward (or route) packets in and out of the same interface. This was the case in version 6.0 of the PIX OS. However with version 7.2(1) and later it can now be done via the command "same-security-traffic permit intra-interface".

A link with the how to can be found on the cisco website

Datacenter Capacity Planning

A lot needs to be considered when planning for a datacenter. Power, cooling, UPS, generators, rack space, cost. All these come into play.

Here are a few links to some vendor calculators I came across that has helped me.

DELL

HP

Displaying 32/64 rpms installed on a Redhat system

It's been a while since I've prepared an Oracle Linux server. Today I got handed a hot potato - "An Oracle Linux server with missing Libraries". This is often the case with oracles installer - It won't install without the proper libraries. What had happened was the previous sysadmin installed the 32bit versions of the libraries and Oracle was looking for the 64bit version. However, with Redhat's default rpm -qa command it often displays only the package but doesn't tell you the architecture of the package. Here's a neat command that would display the architecture of the installed package.

rpm -qa --queryformat "%{NAME}-%{VERSION}.%{RELEASE} (%{ARCH})\n"

Give this a try the next time you install a 64bit Redhat system.

iSCSI @ Home

iSCSI or internet SCSI is a standard for running SCSI over an ip based network. For people familiar with this technology it basically enables you to mount remote disks over your existing IP infrastructure.

Today I setup a simple iSCSI mount on Linux for my local windows desktop at home over my home network. Lets start with the basic terminology and requirements.

You need a Linux box with plenty of storage space, preferably with RAID and a decent network connection - 1G acting as your SAN. This will be your iSCSI target.

Then you need a Linux or Windows desktop to act as your mount point - The iSCSI initiator.

Here are some links on the web which explains this more in detail
Setting up iSCSI in 5 mins
Enterprise iSCSI for Linux
Microsoft iSCSI iniator

Building a simple iSCSI was as simple as following those links and instructions and downloading the appropriate files.

In my quick and dirty implementation I didn't have a high performance disk setup nor a proper LVM setup to export my devices. However, the Linux Enterprise iSCSI target supports exporting files as a virtual file/block device. So I simply created an empty 10G file as my iscsi mount by using dd.

dd if=/dev/zero of=/tmp/iscsi-file bs=1M count=10240

I then exported the /tmp/iscsi-file as LUN0 in the ietd.conf file. It's as simple as that!

Setting up the Windows iSCSI initiator was as simple as eating lettuce.

Next, I'll be posting about setting up the Linux iSCSI initiator as that is a bit more complicated and may require more troubleshooting steps.

##Update
Someone recommended me to a more powerful open source tool for SAN,NAS,iSCSI.
It's called openfiler. Seems like an awesome tool!

Vsftp chroot - Unable to follow symlinks!!

Today I setup vsftp for local user ftp access. I turned on 'chroot_local_user=YES' which essentially puts the ftp user into a chroot jail. But what happens when you want to access something outside of user's home directory?

What? Symlinks don't work?!

Here's the solution I found on the net: Link

Basically you have to mount the external directory into your current directory using the --bind option. That nice link in the article tells you how to do it and how to put it in your fstab file to make it permanent after a reboot!

Happy reading!

Upgrading an Entrepreneur ASP infrastructure - PART III

VMware - The leading industry Virtualization Provider. It comes in two flavors VMWare ESX and VMWare Server (formerly GSX). The former comes at a cost and the latter FREE!

In a low cost setup Vmware server (Free version) is the best recommended solution. Although I must mention that the ESX server provides additional much wanted enterprise features such as support for Network attached SAN, iscsi, clustering and resource pooling.

VMWare is supported on many OSes, Windows, Redhat, CentOS, Ubuntu and more.

In this setup I recommend using CentOS as the base operating system as it is built from the stable Redhat Enterprise Linux source and is free!

Below are steps that should be followed to prepare for VMWare installation.

1. Prepare mirrored hardware RAID if available.
2. Partition the system as follows during OS installation:

- Boot (ext3,primary) = 100 mb
- LVM (LVM,primary) = All remaining space
- LVM name = RAID1
- SWAP = 2G (or 2 x RAM)
- TMP (/TMP) ext3 = 1G
- VAR_LOG (/var/log) ext3 = 2G
- ROOT (/) ext3 = 5G
- VAR_LIB_VMWARE (/var/lib/vmware) ext3 = 100G+
- Leave Free space or allocate all to VAR_LIB_VMWARE

In this setup there are two logical partitions Boot and LVM. Inside LVM contains additional logical partitions with the key note that VAR_LIB_VMWARE is where all the guest OS images reside. This mount point should be in it's own partition in order to prevent allocating too much space affecting the host OS.

A LVM is a Logical Volume Manager. In current releases of Linux, LVM is installed by default for partitioning. LVM allows an administrator to dynamically resize partitions much like partition magic for windows. Conventional Linux partitions suffer from the inability to resize easily. Using Fdisk often requires the system to be booted in rescue mode and risk loss of data. By using LVM, logical volumes are now grouped into a single resource pool. In this pool is a collection of logical extents, these extents are fixed in size and are usually several megabytes. This pool of extents are then allocated to each logical volume to form a partition. When more space is needed, additional free logical extents can be added to those volumes live.

Upgrading an Entrepreneur ASP infrastructure - PART II

After analyzing the situation I have identified several key problems.

1. System Availability - Systems fail frequently due to hardware failures, DoS, and application failures.
2. Collocation is far and administration requires frequent visits
3. Mini tower servers consume space and the 1/2 rack space maybe reaching capacity limits.
4. Network lacks sufficient protection against malicious attacks.
5. Subnet is small and may reach IP assignment limits.

A) The culprit to system availability is the use of low cost hardware. Low cost motherboards and network cards can often fail as well as that different systems built during different time periods usually leads to a mix of components that may not be supported by Enterprise Linux.

RAM, CPU, Motherboard, power supply and Hard disk failures fail at different intervals with Hard disks failure being the most frequent. Most of this is attributable to combination of poor cooling and poor quality parts. As well as this, the power supply is a key component in a system that cannot be neglected as a low quality power supply can lead to more frequent component failures.

Recommendation 1 - Use enterprise grade servers such as Dell and HP rack mountable servers. Such systems are built of much high quality components and provide N+1 redundancy for components that fail often. Dual power supplies and Mirror Raid Hard drives are a necessity. It is important to use Hardware raid for added performance and to ease administration during a failure. Commercial servers provide enterprise grade device driver support. Search and recompiling drivers are a past. Furthermore, Dell's DRAC and HP's iLO are remote access tools that allows a user to remotely administer the system at a BIOS level. Using enterprise grade servers provide increase efficiency, speed and scalability for additional RAM slots and division of CPU cores.

Recommendation 2 - Embrace Virtualization. Virtualization allows multiple OS to run from a single system taking advantage from the systems unused resources such as CPU, HD and RAM by sharing them accross multiple Virtual instances (VM Guests). By combining VMWare with an enterprise server, system stability can be leveraged therefore increasing availability.

B) VMware addresses the need for on-site administration. It allows an administrator to remotely connect to VMWare server to control the guests, performing remote operations such as reboot, allocate additional Network interfaces, RAM, and Hard disk space. All of this is shared from a resource pool belonging to the underlying server. Other neat features include remote mounting external devices and creating a template VM instance allowing the administrator to stamp out pre-configured OS installations with minimal time. Another great advantage of VMware is it allows multiple different OSes such as windows and Linux to coexist in a single host. However, there is one disadvantage which due to the fact that all the eggs are in one basket. An entire system failure could cause all virtual instances to fail. To ensure this risk is minimized 2 or more hosts should be in place in case of failure.

C) By employing VMWare and DELL/HP rack mountable servers, rack space should be reduced significantly leading for more room for expansion.

D) As a secondary phase of the project a robust Firewall needs to be in place to protect against outside DoS and hack attempts. This is a vital piece of equipment which cannot be neglected as it will reduce or remove malicious attacks completely. It also helps hide the underlying network and can help map external IPs to internal IPs and allow only the ports necessary for access. By using a hardware firewall, the OS firewall can be switched off. As well as this, such appliances offer VPN capabilities for protected administrative access to the systems. Such a device is highly sophisticated and it is recommended to use no other brand than CISCO for it's reliability and feature set. The Cisco ASA 5505 unlimited user license is a low cost entry point for such a scale of setup. Due to the price of even the lowest model, the second hand market may need to be considered.

E) By employing a firewall, NAT overloading and static natting can be performed to allow more than one system to use a single WAN IP therefore reducing the need for a large address space.

Note: Noting that the current administrator may not have sufficient knowledge to administer the device, my recommendation is to hold off on the purchase until the systems have reached a certain stability and scale. An experienced administrator needs to be hired to help configure and maintain the device.

Upgrading an Entrepreneur ASP infrastructure - PART I

Any startup entrepreneurial hosting business usually run into many technical challenges. They face difficult business decisions and often have to trade off between stability, scalability and underlying profit.

There is no win/win situation, however I do know one thing - time is money. In a setup where systems experience frequent downtime, hardware failures or even just frequent visits to the data center for administration can be costly in time to the business.

Here's a case study of a hosting business. Netdreamland is a service provider, providing hosting services for various clients from simple web hosting to sophisticated application services requiring administrative access to the systems. Currently Netdreamland rents half a rack from a remote collocation facility with seven low budget mini tower servers. The systems are assigned individual public IPs as they are fed off of a 3com unmanaged switch directly connected to the ISP. The business owner currently faces a dilemma; he is a one man team who manages the business side and the administration side of the business, He has no time to attend to the systems. Moreover, Netdreamlands systems often fail due to hardware failures, application failures and Denial of Service attacks which renders the system unreachable. This is consuming a lot of his time and energy as he often finds himself driving to the data center for simple reboots to hardware replacements, often in the middle of the night. The collocation is quite a drive away from the office and his last visit to the site was to install a new system for a new customer. He is now afraid to expand his customer base too aggressively as it will increase his visits to the point where he cannot tend other business matters. Netdreamland is profitable but is at a point where further expansion will jeopardize service availability.

Cisco SPAN, SNMP and Wireshark

Today I was assigned a task to find out and explain a certain network anomaly we are experiencing in our network. The mission started out to be a bandwidth monitoring task against a specific router. This router however was owned by a third party so we didn't have access to it. Simple enough, I started looking at the next hop from that router - the switch port it was connected which belonged to the company I was working for. Using a combination of SNMP and a nice graphing/monitoring tool called "intermapper" I was able to obtain a pretty graph with traffic going in and out of the interface.

Eventually, looking at the graph we pin pointed the time of the day which we saturated the pipe going out the router which was only a fractional T1 at 64K. However, on the graph we spotted some unexplainable traffic spikes occurring every 5 mins. We couldn't explain why such traffic would occur going out the interface to this router. This warranted for some deeper packet inspection.

Here we used something called the SPAN feature on a Cisco switch. SPAN is just another fancy name for port mirroring. Since we didn't want to impact the production network, we simply mirrored the port on the Cisco switch. The command was easy on our IOS C2960G:

In configuration terminal mode:
monitor session 1 source gi /24
monitor session 1 destination gi 0/1

The setting was straight forward, specify the source port to monitor and the destination port to dump the packets onto.

After that, plug the destination port to a workstation with wireshark aka ethereal and capture the packets! With Wireshark we can sniff whatever traffic that is traversing the interface with some useful statistics and summary reporting.

It turned out to be a strange broadcast to that vlan, resulting from a faulty application.

Layer 2 Best Practices

I think I should be stepping up my security practices. Here's an article that I came across summarized all the things one should know about Layer 2 security practices on Cisco switches.

http://www.networkworld.com/community/node/30682

Configuring VLAN in Linux

This link from the Redhat knowledge base talks all about it.

Here's the list of steps and pointers I had to go through.

1. Make sure that the 8021q module is loaded
modprobe 8021q

2. This step is important as it defines the vlan that it listens on
Make a new interface file named "ifcfg-ethX.Y" where
X - the interface it will listen on.
Y - is the VLAN ID.

Add this to the config file
VLAN=yes

3. Create the neccessary IP configurations on the new interface file

4. Make sure that physical interface file looks like this. In my case it's eth0
DEVICE=eth0
ONBOOT=yes

5. service network restart

6. Done!

How to do a HTTP POST with Curl

This page proves there are people out there who forget about the same things I forget.

Here's the POST I wanted to run today.

curl -d "platform=CHANNEL&processor_target=CERT&charge_type=PING" http://app01.qa.paygateway.com:31080/Quicksilver/

Cisco ASA/PIX Bandwidth limiting

Today, I got a chance to setup some bandwidth limit on our Firewall. The goal was to choke the speed of traffic going to our backup server to 250Mb/s. As our backup traffic goes through a firewall here's how I did it on the my ASA5520s

First Define the traffic I want to choke using an access-list:

access-list backup_traffic extended permit ip any host BACKUP01

Then create a policy map to and set the speed

policy-map backup_traffic
class backup_traffic
police input 250000000

Then apply the policy map on the interface

service-policy backup_traffic interface outside

All done! That was easy. There's a lot more you can do in terms of QoS on the PIX/ASA. All this information can be found on the cisco site.