Error detection/correction and fault detection/recovery – Data processing system error or fault handling – Reliability and availability
Reexamination Certificate
1999-09-09
2003-05-06
Beausoliel, Robert (Department: 2184)
Error detection/correction and fault detection/recovery
Data processing system error or fault handling
Reliability and availability
C714S043000, C714S703000
Reexamination Certificate
active
06560720
ABSTRACT:
FIELD OF THE INVENTION
The present invention relates generally to packet-switched computer networks, and specifically to methods and apparatus for testing and diagnosing malfunctions in such networks.
BACKGROUND OF THE INVENTION
Packet-switched, source-routing computer networks are used in a growing range of applications. Such networks link multiple computer processors, or nodes, via multiple switches. Typically, a packet of data sent from one of the nodes to another passes through a number of different switches. Each switch along the way reads routing information, which is commonly contained in a header of the data packet, and passes the packet on to the next switch along the way, or to the destination node. Typically, there are multiple different paths available through the network over which any given pair of nodes can communicate. An example of this type of network is the well-known Asynchronous Transfer Mode (ATM) network, which is used in communications between separate computers. Such networks are also used in multi- processor computers, such as the RS/6000 Scalable POWERParallel System (SP) series of computers produced by International Business Machines Corporation (Armonk, N.Y.). In the SP computer, as well as in certain other networks, successive packets in a communication stream between the nodes may be sent over different routes.
Because of the complex topology and hardware of packet-switched networks, when a fault occurs in such a network it can be difficult to identify the exact location and nature of the fault. The difficulty is exacerbated by the fact, noted above, that by their nature such networks use multiple different paths between nodes and are fault-tolerant. A network fault will typically appear not as a total breakdown (which would be relatively easy to find), but rather will present more subtle symptoms. For example, there may be a reduction in throughput between some or all of the nodes, or an increase in the number of “bad packets”—data packets whose content is corrupted and must be discarded—at one or more of the nodes.
There are few efficient tools known in the art for diagnosis of such faults. The diagnostic process is time-consuming and heavily reliant on the intuition and experience of a human system administrator (or service engineer) in deciphering and drawing conclusions from the limited information that is available. This information is typically collected in various system files, such as topology files, error logs and trace files, as are known in the art. These files may be recorded at different nodes of the network and must somehow be collated and analyzed by the administrator. Because few network administrators have the know-how to perform this sort of diagnosis, costly service calls are frequently required.
A further problem in diagnosing network faults is non-deterministic failures, which may occur only under certain conditions, and may not arise at all while the diagnostic tests are being performed. Such failures are referred to with terms such as “sporadic,” “intermittent,” “overheating,” “lightning,” “aging,” or “statics,” which generally mean only that the cause of the problem is unknown. For example, a high-speed switch or adapter may behave normally in light traffic, and break down only under certain particular stress conditions. At times the only way to find such a problem is to systematically bombard each suspect component of the network with packets from different sources, at controlled rates, gradually eliminating components from consideration until the failure is found. Such a process is difficult to automate, and may require that the network be taken off-line for an extended period. The cost of such down-time for prolonged testing and repair can be enormous. There is therefore a need for systematic methods of diagnostic testing, which can be performed while the network is on-line.
There is a similar lack of tools and techniques for systematically testing the response of switch-related network software to hardware fault conditions. Such techniques are needed particularly in software development and testing stages, to ensure that the software responds properly when faults occur. Current methods of testing use specially-designed simulation hardware, such as cables with broken pins, together with debugging clauses that can be activated in the software itself and dedicated debugging fields in associated data structures. The fault situations created by such methods, however, are limited to a small range of scenarios, which are for the most part different from the real hardware faults that occur in actual networks. Similarly, the software used in debugging mode for fault simulation is different from the actual software product that will be used in the field. Moreover, these testing tools are incapable of simulating the type of transient, non-deterministic failures described above. They do not allow errors to be injected and altered on the fly during a simulation.
SUMMARY OF THE INVENTION
It is an object of some aspects of the present invention to provide improved methods for fault simulation and diagnostics in packet-switched data networks.
It is a further object of some aspects of the present invention to provide apparatus and methods for systematically injecting errors into a data network, for purposes of debugging and diagnostics.
Preferred embodiments of the present invention operate in the context of a packet data network, which comprises a plurality of nodes, or processors, mutually coupled by a plurality of switches, such that typically any one of the nodes can communicate with any other one of the nodes, preferably over multiple links. Each of the nodes is coupled to a respective port of one of the switches by a switch adapter, which performs data link functions, as are known in the art, with respect to each data packet sent or received through the network by the node. One of the nodes is a primary node, which manages the configuration of elements of the network, such as the other nodes and switches in the network.
In preferred embodiments of the present invention, the primary node controls testing and diagnosis of elements of the network in real time, while the network is on-line, or at least with minimal interruption of on-line operation, by appropriately setting parameters of the nodes and switches. The testing preferably includes diagnostic testing to locate suspected faults in the switches and switch adapters. Additionally or alternatively, for the purposes of testing, errors are intentionally injected into the network so as to simulate the response of the network elements to faults that may occur.
In some preferred embodiments of the present invention, one of the nodes of the network is used as an error injector, and is isolated for this purpose from the remaining nodes in the network. While the remaining nodes and switches carry out their normal functions on-line, the error injector injects errors into the network in order to test by simulation the response of elements of the network to actual errors that may occur. Generally, if the network is operating properly, the errors will be rejected and, preferably, reported to and logged by the primary node, while normal functions continue without substantial interruption.
In some of these preferred embodiments, the error injector is used to simulate the effect of a faulty switch adapter, by injecting bad packets into the network.
In other preferred embodiments, the error injector is used to simulate switch faults, such as failures in the ports or central queue or in send and/or receive logic of the switch. Preferably, the error injector sends a command packet to one of the switches, such as a packet instructing the switch to reset its queue or initialize operation, while the switch is in the midst of its normal functions. The error injector prepares the command packet in such a manner that it appears to have originated from the primary node, which is the only node ordinarily entitled to issue such commands. The primary mode generally issues a reset or initialization comma
Chirashnya Igor
Machulsky George
Ross Rony
Shalev Leah
Beausoliel Robert
Darby & Darby
International Business Machines - Corporation
McGrath Christopher R.
LandOfFree
Error injection apparatus and method does not yet have a rating. At this time, there are no reviews or comments for this patent.
If you have personal experience with Error injection apparatus and method, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Error injection apparatus and method will most certainly appreciate the feedback.
Profile ID: LFUS-PAI-O-3003281