Interop Lab Testing for VoIP Devices (2008 edition) - ECG

In an Interoperation (Interop) Lab, devices are made to work together. When they seem to, the vendors of the devices claim that they "interop with" each other. This is necessary, but not sufficient, to know things will work together.

Background

Suppose you make telephone soft-switch or application server, such as BroadSoft BroadWorks, Sylantro, MetaSwitch, or the Alcatel-Lucent Network Gateway (aka Lucent Compact Switch). The customers of your several-hundred-thousand-dollar device want to use VoIP telephones with it, but they've tried a few cheap ones and had trouble. And they had another one they liked, but when they upgraded it, call-hold stopped working. After opening several tickets with you and with the phone vendor, they give up and realize that neither of you did anything wrong.

The phone was always doing something reasonable -- though not quite what they hoped -- and the softswitch was doing something reasonable -- though not quite what they hoped. This is the curse of the VoIP Implementor: everybody can follow the rules, and working together, get nothing done.

After complaints from customers about this difficulty, you (the softswitch vendor) launch an interop testing program. Depending on the sort of technical staff you have, you might do it in-house, or outsource it.

Your goal is to confirm that specific products "work with" your product. You want your customers to be able to choose products confidently. It's a laudable goal.

A common interop lab setup might look like this, for a softswitch vendor's lab, when testing a new VoIP phone:

You've got the new phone (the Device Under Test, or DUT), and your piece of gear. Plus you connect them with an ordinary Ethernet switch. And you have another gold phone that you know to work with your platform, because phone calls require two phones.

Advantages and Limitations of Interop Testing

Interop testing of this variety can be very useful. SIP VoIP telephony is an evolving body of standards. There are more than one way to do something (such as put a call on hold, or make one phone number appear on two different phone (shared call appearance (SCA))). It is very useful to identify fundamental incompatibilities. Often, there are certain settings required on both sides: to use this device, The "rfc2543_hold" option must be turned on. Somebody has to figure this out: it had might as well be two the vendors involved.

This testing is only as good as the test plan. If the test plan doesn't include something that you need to do, then the testing isn't quite complete for you.

In addition, no two devices have all the same features. The vendor may test for T.38, but if the DUT doesn't do T.38, they don't want to mark it as failed for T.38. Instead, the testing engineer is likely just to mark that feature as Not Supported. Depending on policy, or haste, the testing engineer may just mark anything that doesn't work as Not Supported. There is no "one true standard" for interoperability. Ultimately, if the DUT and his softswitch work together at all, the two are "certified" as interoperable. It is to neither vendor's advantage to anger the other by claiming they're not interoperable.

Finally, Interop lab testing only tests the interoperation between a pair of device. Devices X and Y work together. But real systems are made of devices A through Z.

Real Integration Testing

A network for a VoIP Service Provider using SIP peering might look like this:

A real network for an SS7-connected CLEC might look more like this:

There are lots of devices. And many of them may affect the success you'll have with the DUT.

Take, for example, a simple SIP phone. It's easily possible that the signaling path for a PSTN call, in a CLEC case, is

• SIP Phone
• Customer Premise ALG
• Session Border Controller
• Softswitch
• PSTN Gateway
• SS7 STP networks
• PSTN Class 5 Telephone Switch
• PBX
• PBX handset

It's definitely possible that the SIP phone could work with the softswitch, but irritate one of the other components. For example,

• The SIP phone and the softswitch may use Requires: a specific SIP feature package, but the SBC doesn't allow it or support it.
• Or perhaps yet-another form of caller ID is dreamed up, and the PSTN gateway can't support it, so that all calls from this phone appear to have no caller ID.
• Or maybe the phone signals some sort of ISUP calling party category in SIP that the PSTN gateway passes through, and confuses a downstream PSTN telephone switch.
• What if the phone needs SIP over TCP, but the ALG doesn't support it properly?
• Or the DUT signals RFC2833 support properly, but the PSTN gateway expects telephone-events to have a specific codec number (i.e., tickling a bug that was already present)?
• Perhaps it works fine if the SBC is configured with a "classic" configuration, but breaks miserably if you switch to the "new" configuration.
• Maybe it can download configurations from the lab FTP server, but chokes if the FTP server is a little slow

My point is that we have to test components an in and-to-end environment to really get confidence that they work. This may change, but only if the complexity here is accidental (a side-effect of today's state of things) and not essential complexity (fundamental to the job). With all the new features that VoIP telephony systems try to support, and the upgradability the protocol designers intend, I'm not sure I can distinguish which it is now.

Show me a "simple SIP phone", and I'll show you a phone that nobody likes because it's not flexible enough to be configured to do crazy things.