On-line switch diagnostics

Details On-line switch diagnostics On-line switch diagnostics

1999-09-09

2002-11-26

Yao, Kwang Bin (Department: 2664)

Multiplex communications

Pathfinding or routing

Switching a message which includes an address header

Reexamination Certificate

active

06487208

ABSTRACT:

CROSS-REFERENCE TO RELATED APPLICATIONS
This application is related to U.S. patent applications entitled “Switch Adapter Testing” and “Error Injection Apparatus and Method,” filed on even date, which are assigned to the assignee of the present patent application and are incorporated herein by reference.
FIELD OF THE INVENTION
The present invention relates generally to packet-switched computer networks, and specifically to methods and apparatus for testing and diagnosing malfunctions in such networks.
BACKGROUND OF THE INVENTION
Packet-switched, source-routing computer networks are used in a growing range of applications. Such networks link multiple computer processors, or nodes, via multiple switches. Typically, a packet of data sent from one of the nodes to another passes through a number of different switches. Each switch along the way reads routing information, which is commonly contained in a header of the data packet, and passes the packet on to the next switch along the way, or to the destination node. Typically, there are multiple different paths available through the network over which any given pair of nodes can communicate. An example of this type of network is the well-known Asynchronous Transfer Mode (ATM) network, which is used in communications between separate computers. Such networks are also used in multi-processor computers, such as the RS/6000 Scalable POWERParallel System (SP) series of computers produced by International Business Machines Corporation (Armonk, N.Y.). In the SP computer, as well as in certain other networks, successive packets in a communication stream between the nodes may be sent over different routes.
Because of the complex topology and hardware of packet-switched networks, when a fault occurs in such a network it can be difficult to identify the exact location and nature of the fault. The difficulty is exacerbated by the fact, noted above, that by their nature such networks use multiple different paths between nodes and are fault-tolerant. A network fault will typically appear not as a total breakdown (which would be relatively easy to find), but rather will present more subtle symptoms. For example, there may be a reduction in throughput between some or all of the nodes, or an increase in the number of “bad packets”—data packets whose content is corrupted and must be discarded—at one or more of the nodes.
There are few efficient tools known in the art for diagnosis of such faults. The diagnostic process is time-consuming and heavily reliant on the intuition and experience of a human system administrator (or service engineer) in deciphering and drawing conclusions from the limited information that is available. This information is typically collected in various system files, such as topology files, error logs and trace files, as are known in the art. These files may be recorded at different nodes of the network and must somehow be collated and analyzed by the administrator. Because few network administrators have the know-how to perform this sort of diagnosis, costly service calls are frequently required.
A further problem in diagnosing network faults is non-deterministic failures, which may occur only under certain conditions, and may not arise at all while the diagnostic tests are being performed. Such failures are referred to with terms such as “sporadic,” “intermittent,” “overheating,” “lightning,” “aging,” or “statics,” which generally mean only that the cause of the problem is unknown. For example, a high-speed switch or adapter may behave normally in light traffic, and break down only under certain particular stress conditions. At times the only way to find such a problem is to systematically bombard each suspect component of the network with packets from different sources, at controlled rates, gradually eliminating components from consideration until the failure is found. Such a process is difficult to automate, and may require that the network be taken off-line for an extended period. The cost of such down-time for prolonged testing and repair can be enormous. There is therefore a need for systematic methods of diagnostic testing, which can be performed while the network is on-line.
There is a similar lack of tools and techniques for systematically testing the response of switch-related network software to hardware fault conditions. Such techniques are needed particularly in software development and testing stages, to ensure that the software responds properly when faults occur. Current methods of testing use specially-designed simulation hardware, such as cables with broken pins, together with debugging clauses that can be activated in the software itself and dedicated debugging fields in associated data structures. The fault situations created by such methods, however, are limited to a small range of scenarios, which are for the most part different from the real hardware faults that occur in actual networks. Similarly, the software used in debugging mode for fault simulation is different from the actual software product that will be used in the field. Moreover, these testing tools are incapable of simulating the type of transient, non-deterministic failures described above. They do not allow errors to be injected and altered on the fly during a simulation.
SUMMARY OF THE INVENTION
It is an object of some aspects of the present invention to provide improved methods for fault simulation and diagnostics in packet-switched data networks.
It is a further object of some aspects of the present invention to provide diagnostic methods and apparatus for use in testing a switch in a network while the network is on-line.
Preferred embodiments of the present invention operate in the context of a packet data network, which comprises a plurality of nodes, or processors, mutually coupled by a plurality of switches, such that typically any one of the nodes can communicate with any other one of the nodes, preferably over multiple links. Each of the nodes is coupled to a respective port of one of the switches by a switch adapter, which performs data link functions, as are known in the art, with respect to each data packet sent or received through the network by the node. One of the nodes is a primary node, which manages the configuration of elements of the network, such as the other nodes and switches in the network.
In preferred embodiments of the present invention, the primary node controls testing and diagnosis of elements of the network in real time, while the network is on-line, or at least with minimal interruption of on-line operation, by appropriately setting parameters of the nodes and switches. The testing preferably includes diagnostic testing to locate suspected faults in the switches and switch adapters. Additionally or alternatively, for the purposes of testing, errors are intentionally injected into the network so as to simulate the response of the network elements to faults that may occur.
In some preferred embodiments of the present invention, a switch that is suspected of being faulty is isolated from the remainder of the network for testing, while the remainder of the network continues to operate in a substantially normal fashion. The suspect switch has multiple ports, each of which may be connected either to a node or to a port of another switch. Data packets are passed through the switch via different pairs of the ports, at controllable, preferably high rates, until conditions are created in which the switch fails, or until it is ascertained that the switch is not faulty. By isolating the suspect switch in this manner, it is possible to test the switch quickly and thoroughly, while minimizing interruption to the rest of the network.
Preferably, for every port of the suspect switch, the primary node assigns one of the nodes of the network to be an assistant node. Preferably, the assistant nodes are chosen to be the nodes closest in the network to the respective ports of the suspect switch. The assistant nodes do not need to be connected directly to the respective ports, but may rather be connected through other switches in t

Affiliated with

Also associated with

Rating

On-line switch diagnostics does not yet have a rating. At this time, there are no reviews or comments for this patent.

If you have personal experience with On-line switch diagnostics, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and On-line switch diagnostics will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFUS-PAI-O-2992230