2004 IPLS Abilene full router instrumentation

This is the third and final stage of our joint project with Internet2 to fully instrument a modern busy backbone router, in this case a Juniper T640, and study its performance under regular traffic as well as stress conditions.

Technical Specifications

The fiber optic taps are all by NetOptics, for the specific models please see the table below. All OCxMONs are based on Dell PowerEdge 2650 2U rack mountable servers. The operating system is Debian Linux featuring Endace DAG driver softare 2.4.14. The instrumentation uses a range of network monitoring cards from Endace: DAG6.1 OC192c PoS, DAG4.2 OC48c PoS, DAG4.3GE Gigabit Ethernet SX, DAG3.8 OC3c/OC12c. Time is provided by an EndRun Technologies Praecis Ct CDMA time receiver, redistributed to the DAG cards via an Endace TDS-24 Time Distribution Server box.

The T640 has a total of 12 fiber optic links operating at speeds from 155 MBits/sec OC3c to 10 Gigabits/sec OC192c. We are using a set of eight Dell servers to capture the traffic from the 12 bidirectional links using a total of 16 special purpose network measurement cards (NMC). They form three OC192MONs, an OC48MON, a dual-OC12MON, a dual-OC3MON and a pair of dual-GIGEMONs.

Each of the links instrumented has specific properties that have to be taken into consideration. The fiber can be singlemode (SM) versus multimode (MM), in the pictures below identified as being yellow versus orange. The carrier speed and framing type determines the type of NMC to use. The cards in turn have specific requirements towards the optical signal strength, which determines the type of splitter and ratio to use. At the same time we need to ensure the resulting attenuation to the link budget does not destabilize the operations of the long distance circuit. In one case (Indiana GigaPOP link), we have to use a converter tap to change from Gigabit Ethernet LX technology (SM at 1310) to SX technology (MM at 850) in order to match the capabilities of the NMC. A total of 36 pairs of fibers are passing through the splitter setup (see pictures below). The Dell servers are wired each with -48VDC power, generic IP connectivity via Ethernet, the 12 pairs of fibers from the taps, plus 16 1PPS signals from the TDS-24 as replica of the Praecis-Ct CDMA time receiver (resting on the converter splitter in the picture below).

Needless to say, an instrumentation like this is a major logistical challenge. To put things into perspective, this setup is perhaps an order of magnitude more complicated in terms of wiring and configurations than the Abilene backbone node itself, yet there are weeks and weeks of preparation time available for launching a node like this. With the router instrumentation, many things are hot staged in the crunch of time, there is no cookbook, the rules have to be made up as you go, and you have to work around multiple space/time constraints.

Under those conditions, success critically depends on your colleagues and collaborators who understand the constraints and are flexible and supportive, and will go the extra mile for you. More than once!

Our special thanks go to:

• Jim Hale, NLANR/MNA systems administrator, SDSC
• Caroline Carver, Indiana GlobalNOC Network Engineer
• Chris Small, Indiana GlobalNOC Network Engineer
• John Hicks, TransPAC and Indiana University
• Rick Summerhill, Internet2 Director Network Research, Architecture and Technology
• Matt Zekauskas, Chair, Internet2 Measurement Working Group

This project received excellent support by the team of: