Traceroute sends a series of UDP packets with the destination address of the device you want a path to. * By default, traceroute sends sets of three packets to discover each hop. Traceroute sets the TTL field in the first three packets to a value of 1 so that they are discarded by the first router on the path. When the ICMP TIME_EXCEEDED messages are returned by that router, traceroute records the source IP address of these ICMP messages. This is the IP address of the first hop on the route to the destination.

Next, three packets are sent with their TTL field set to 2. These will be discarded by the second router on the path. The ICMP messages returned by this router reveal the IP address of the second router on the path. The program proceeds in this manner until a set of packets finally has a TTL value large enough so that the packets reach their destination.

Typically, when the probe packets finally have an adequate TTL and reach their destination, they will be discarded and an ICMP PORT_UNREACHABLE message will be returned. This happens when traceroute sends all its probe packets with what should be invalid port numbers, i.e., port numbers that aren’t usually used. To do this, traceroute starts with a very large port number, typically 33434, and increments this value with each subsequent packet. Thus, each of the three packets in a will have three different unlikely port numbers. The receipt of ICMP PORT_UNREACHABLE messages is the signal that the end of the path has reached.

Should a packet be lost, an asterisk is printed in the place of the missing time. In some cases, all three times may be replaced with asterisks. This can happen for several reasons. First, the router at this hop may not return ICMP TIME_EXCEEDED messages. Second, some older routers may incorrectly forward packets even though the TTL is 0. Third possibility is that ICMP messages may be given low priority and may not be returned in a timely fashion. Finally, beyond some point of the path, ICMP packets may be blocked.

Options

• -n: disable name resolution.
• -v: enable verbose option which will log source and packet sizes of the probes will be reported for each packet.
• -m: define maximum number of hops where default is 30 hops before halting.
• -p: traceroute usually receives a PORT_UNREACHABLE message when it reaches its final destination because it uses a series of unusually large port numbers as the destination ports. Should the number actually match a port that has a running service, the PORT_UNREACHABLE message will not be returned. This is rarely a problem since three packets are sent with different port numbers, but, if it is, the option lets you specify a different starting port so these ports can be avoided.
• -q: traceroute sends three probe packets for each TTL value with a timeout of three seconds for replies. This can be changed using –q option.
• -w: define the default timeout value for the probe packets.

(Source: Network Troubleshooting Tools” by Joseph D. Sloan)

The overall speed of a particular network connection is limited by (a) the slowest link in the connection chain and (b) the length of the chain. Identifying the slowest link is difficult and may not even be consistent: it can vary at different times of the day or for different communication paths. A network communication path lead from an application through a TCP/IP stack (which adds various layers of headers, possibly encrypting and compressing data as well), then through the hardware interface, through a modem, over a phone line, through another modem, over to a service provider’s router, through many heavily congested data lines of various carrying capacities and multiple routers with different maximum throughputs and configurations, to a machine at the other end with its own hard interface, TCP/IP stack and application. A typical web download route is just like this. In addition, there are dropped packets, acknowledgments, retries, bus contention, and so on.

Because so many possible causes of bad network performance are external to an application, one option you can consider including in an application is a network speed testing facility that reports to the user. This should test the speed of data transfer from the machine to various destinations: to itself, to another machine on the local network, to the Internet Service Provider, to the target server across the network, and to any other destinations appropriate. This type of diagnostics report can tell users that they are obtaining bad performance from something other than your application. If you feel that the performance of your application is limited by the actual network communication speed, and not by other (application) factors, this facility will report the maximum possible speeds to your user.

Latency

Latency is different from the load-carrying capacity (bandwidth) of a network. Bandwidth refers to how much data can be sent down the communication channel for a given period of time and is limited by the link with the lowest bandwidth in the communication chain. The latency is the amount of time a particular data packet takes to get from one end of the communication channel to the other. Bandwidth tells you the limits within which your application can operate before the performance become affected by the volume of data being transmitted. Latency often affects the user’s view of the performance even when bandwidth isn’t a problem.

In most cases, especially Internet traffic, latency is an important concern. You can determine the basic round-trip time for a data packets from any two machines using the ping utility. (Refer to “Understanding Network – How Ping works?”). However, the time measure is for a basic underlying protocol (ICMP packet) to travel between the machines. If the communication channel is congested and the overlying protocol requires re-transmissions (often the case for Internet traffic), one transmission at the application level can actually be equivalent to many round trips.

It is important to be aware of these limitations however it is also often possible to tune the application to minimize the number of transfers by (a) packing data together, (b) caching and (c) redesigning the distributed application protocol to aim for a less conversational mode of operation. At the network level, you need to monitor the transmission statistics (using the ping and netstat utilities and packet sniffers) and consider tuning any network parameters that you have access to in order to reduce re-transmissions.

TCP/IP Stacks

The TCP/IP stack is the section of code that is responsible for translating each application-level network request (send, receive, connect, etc.) through the transport layers down to the wire and back up to the application at the other end of the connection. Because the stacks are usually delivered with the operation system and performance-tested before delivery (since a slow network connection on an otherwise fast machine and fast network is pretty obvious), it is unlikely that the TCP/IP stack itself is a performance problem.

In addition to the stack itself, stacks include several tunable parameters. One parameter worth mentioning is the maximum packet size. When your application sends data, the underlying protocol breaks the data into packets that are transmitted. There is an optimal size for packets transmitted over a particular communication channel, and the packet size actually used by the stack is compromise. Smaller packets are less likely to be dropped, but they introduced more overhead, as data probably has to be broken up into more packets with more header overhead.

If your communication takes place over a particular set of endpoints, you may want to alter the packet sizes. For a LAN segment with no router involved, the packets can be big (e.g. 8KB). For a LAN with routers, you probably want to set the maximum packet size to the size the routers allow to pass unbroken. (Routers can break up the packets into smaller ones; 1500 bytes is the typical maximum packet size and the standard for the Ethernet. The maximum packet size is configurable by the router’s network administrator.) If your application is likely to be sending data over the Internet and you cannot guarantee the route and quality of routers it will pass through, 500 bytes per packet is likely to be optimal.

Network Bottlenecks

Other causes of slow network I/O can be attributed directly to the load or configuration of the network. For example, a LAN may become congested when many machines are simultaneously trying to communicate over the network. The potential throughput of the network could handle the load, but the algorithms to provide communication channels slow the network, resulting in a lower maximum throughput. A congested Ethernet network has an average throughput approximately one third the potential maximum throughputs. Congested networks have other problems, such as dropped network packets. If you are using TCP, the communication rate on a congested network is much slower as the protocol automatically resends the dropped packets. If you are using UDP, your application must resend multiple copies for each transfer. Dropping packets in this way is common for the Internet. For LANs, you need to coordinate closely with network administrators to alert them to the problem. For single machines connected by a service provider, suggesting improvements. The phone line to the service provider may be noisier than expected: if so, you also need to speak to the phone line provider. It is also worth checking with the service provider, who should have optimal configurations they can demonstrate.

Dropped packets and re-transmissions are a good indication of network congestion problems, and you should be on constant lookup for them. Dropped packets often occur when routers are overloaded and find it necessary to drop some of the packets being transmitted as the router’s buffer overflow. This means that the overlying protocol will request the packets to be resent. The netstat utility lists re-transmission and other statistics that can identify these sorts of problems. Re-transmissions may indicate that the maximum packet size is too large.

DNS Lookup

Looking up network address is an often-overlooked cause of bad network performance. When your application tries to connect to a network address such as foo.bar.somthing.org, your application first translates foo.bar.somthing.org into a four-byte network IP address such as 10.33.6.45. This is the actual address that the network understands and uses for routing network packets. DNS translation works as follows:

1. The machine running the application sends the text string of the hostname (e.g. foo.bar.something.org) to the DNS server.
2. The DNS server checks its cache to find an IP address corresponding to that hostname. If the server does not find an entry in the cache, it asks its own DNS server (usually further up the Internet domain-name hierarchy) until ultimately the name is resolved. (This may be by components of the name being resolved, e.g. first .org, then something.org, etc, each time asking another machine as the search request is successively resolved.) This resolved IP address is added to the DBS server’s cache.
3. The IP address s returned to the original machine running the application.
4. The application uses the IP address to connect to the desired destination.

The address lookup does not need to be repeated once a connection is established, but any other connections (within the same session of the application or in other session s at the same time and later) need to repeat the lookup procedure to start another connection.

You can improve this situation by running a DNS server locally on the machine, or on a local server if the application uses a LAN. A DNS server can be run as a “caching-only” server that resets its cache each time the machine is rebooted. There would be little point in doing this if the machine used only one or two connections per hostname between successive reboots. For more frequent connections, a local DNS server can provide a noticeable speedup to connections. Nslookup is useful for investigating how a particular system does translations.

(Source: “Java Performance Tuning”)

Once we move to multi-hop paths, a third consideration enters the picture – the delay introduced from processing packets at intermediate devices such as routers and switches. This is usually called the queuing delay since, for the most part, it arises from the time packets spend in queues within the device. The total delay in delivering a packet is the sum of these three delays. Transmission and propagation delays are usually quite predictable and stable. Queuing delays, however, can introduce considerably variability.

The term bandwidth is typically used to describe the capacity of a link.

Throughput is a measure of the amount of data that can be sent over a link in a given amount of time. Throughput estimates, typically obtained through measurements based on the bulk transfer of data, are usually expressed in bits per second or packets or second. Throughput is frequently used as an estimate of the bandwidth of a network, but bandwidth and throughput are really two different things. Throughput measurement may be affected by considerable overhead that is not included in bandwidth measurements. Consequently, throughput is a more realistic estimator of the actual performance you will see.

Throughput is generally an end-to-end measurement. When dealing with multi-hop paths, however, the bandwidths may vary from link to link. The bottleneck bandwidth is the bandwidth if the slowest link on a path, i.e., the link with the lowest bandwidth.

(Source: Network Troubleshooting Tools” by Joseph D. Sloan)

Taking the above in a real-live environment (depending on the role you play in), bandwidth is fixed in a load test and usually controlled by the network guys through allocation. Therefore, there isn’t much to be done if you are a load tester. On the other hand, throughput is the amount of load that is transmitted over the network and we suggest to use it as the measurement for network usage. After the load test, you can advise the network guys that there is a predictable amount of data (expressed in throughput) being transferred before the system is to go-live. This will allow them to have better planning on the bandwidth allocation and expectation of slowness if a high level of data is being transferred when go-live.

You should begin with your loopback interface. Use either localhost or 127.0.0.1. Next, ping your interface by IP number. (Run ifconfig –a or ipconfig /all if in doubt.) If either of these fails, you know that you have a problem with the host.

IP Address (local machine)

Next, try a host on a local network that you know is operational. Use its IP address rather than its hostname. If this fails, there are several possibilities. If other hosts are able to communicate on the local network, then you likely have problems with your connection to the network. This could be your interface, the cable to your machine, or your connection to a hub or switch. Of course, you can’t rule out configuration errors such as media type on the adapter or a bad IP address or mask.

Hostname (local machine)

Next, try to reach the same host by name rather than number. This is to determine if the DNS and name resolution is the cause of the problem. If this fails, you almost certain to have problems with name resolution.

Router Connectivity

Try reaching the near and far interfaces of the router. This will turn up any basic routing problems you may have on your host or connectivity problems getting to your router.

Traceroute Remote Machines

If all goes well here, you are ready to ping remote computers. (You will need to know the IP address of the intermediate devices to do this test. If in doubt, use traceroute to determine the machines.) Realize, of course, that if you start having failures at this point, the problem will likely lie beyond your router. For example, your ICMP ECHO_REQUEST packets may reach the remote machine, but it may not have a route to your machine to use for the ICMP ECHO_REPLY packets.

When faced with failure at this point, your response will depend on who is responsible for the machines beyond your router. If this is still part of your network, you will want to shift your tests to machines on the other side of the router and try to work in both directions.

Before you seek help… If these machines are outside your responsibility or control, you will need to enlist the help of the appropriate person. Before you contact this person, you should collect as much as information as you can. There are three things you may want to do. First, go back to using IP numbers if you have been using names. As said before, if things start working, you have a name resolution problem.

Second, if you were trying to ping a device several hops beyond your router, go back to closer machines and try to zero in on exactly where you first encountered the problem.

Finally, be sure to probe form more than one machine. While you may have a great deal of confidence in your local machine at this point, your discussion with the remote administrator may go much more smoothly if you can definitely say that you are seeing this problem from multiple machines instead of just one.

(Source: Network Troubleshooting Tools” by Joseph D. Sloan)

Clearly, for the program to work, the networking protocol must support this query/response mechanism. The ping program is based on Internet Control Message Protocol (ICMP), part of the TCP/IP protocol. ICMP was designed to pass information about network performance between network devices and exchange error messages which supports a wide variety of message types, including query/response mechanism.

The normal operation of ping relies on two specific ICMP messages, ECHO_REQUEST and ECHO_REPLY, but it may response to ICMP messages other than ECHO_REPLY when appropriate. In theory, all TCP/IP-based network equipment should respond to an ECHO_REQUEST by returning the packet to the source, but this is not always the case.

Interpreting Results

In different flavors of ping, results vary. However, for each packet we are given the size and source of each packet, an ICMP sequence number, a Time-To-Live (TTL) counter, and the round-trip times. Of course, the sequence number and round trip time are the most revealing when evaluating basic connectivity.

When each ECHO_REQUEST packet is sent, the time the packet is sent is recorded in the packet. This is copied into the corresponding ECHO_REPLY packet by the remote host. When an ECHO_REPLY packet is received, the elapsed time is calculated by comparing the current time to the time recorded in the packet, i.e., the time the packet was sent. This difference, the elapsed time, is reported along with ECHO_REPLY packet is received that matches a particular sequence number, that packet is resumed lost. The size and the variability of elapsed times will depend on the number and speed of intermediate links as well as the congestion on those links.

It may seem that TTL field could be used to estimate the number of hops on a path. Unfortunately, this is problematic. When a packet is sent, the TTL field is initialized and is subsequently decremented by each router along the path. If it reaches zero, the packet is discarded. This imposes a finite lifetime on all packets ensuring that, in the event of a routing loop, the packet won’t remain on the network indefinitely. Unfortunately, the TTL field may or may not be reset at the remote machine and, if reset, there is little consistency in what it is set to. Thus, you need to know very system-specific information to use the TTL field to estimate the number of hops on a path.

Options

• -c: allow you to specify the number of packets you want to send.
• -f: used to flood packets onto network. This option is to send as fast as the receiving host can handle them which is useful for stress testing a link or to get some indication of the comparative performance of interfaces. This is restricted to root.
• -l: used to flood packets onto network. It takes a count and sends out that many packets as fast as possible which eventually falls back to normal mode. This could be used to see how the router handles a flood of packets. This is restricted to root.
• -i: allows the user to specify the amount of time in seconds to wait between sending consecutive packets.
• -n: restricts output to numeric form which is useful if you have DNS problems.
• -v: used for verbose output.
• -q, -Q: used for quiet output.
• -s: specifies how much data to send. If set too small, less than 8, there won’t be space in the packet for a time-stamp. Setting the packet size can help in diagnosing a problem caused by path Maximum Transmission Unit (MTU) settings (the largest frame size that can be sent on the path) or fragmentation problems. (Fragmentation is dividing data among multiple frames when a single packet is too large to cross a link. It is handled by the IP portion of the protocol stack.) The general approach is to increase packet sizes up to the maximum allowed to see if at some point you have problems. When this option isn’t used, ping defaults to 64 bytes, which may be too small a packet to reveal some problems. Also, remember that ping does not count the IP or ICMP header in the specified length so that your packets will be 28 bytes larger than you specify.

You could conceivably see MTU problems with protocols, such as PPP, that use escaped characters as well. With escaped characters, a single character may be replaced by two characters. The expansion of escaped characters increases the size of the data frame and can cause problems with MTU restrictions or fragmentation.

• -p: allows you to specify a pattern for the data included within the packet after the timestamp.

The above are not the entire list of options. As such, be sure to consult the documentation if things don’t work as expected.

Problems with Ping

The program does not exist in isolation, but depends on the proper functioning of other elements of the network. Ping usually depends upon ARP and DNS. As previously mentioned, if you are using a hostname rather than an IP address as destination, the name of the host will have to be resolved before ping can send any packets. You can bypass DNS by using IP address.

It is also necessary to discover the host’s link level address for each host along the path to the destination. Although this is rarely a problem, should ARP resolution fail, then ping will fail. You could avoid this problem, in part; by using start ARP entries to ensure that the ARP table is correct. A more common problem is that the time reported by ping for the first packet sent will often be distorted since it reflects both transit times and ARP resolution times. On some networks, the first packet will often be lost. You can avoid this problem by sending more than one packet and ignoring the results for the first packet.

(Source: Network Troubleshooting Tools” by Joseph D. Sloan)