IMS is much like other IP Services (e.g., Cloud Computing, Smart Grid, MPLS and IP Data Services, etc.) in that it is a complex mélange of a variety of protocols that all have to function properly individually and in concert with each other. The latter attribute is what adds a great deal of complexity to the equation. The legacy testing industry has focused on testing each protocol in isolation, usually to establish metrics related to questions of load: How many packets per second? How many bytes per second? How many connections per second?
There is no doubt that such questions are important and necessary to quantify the readiness of a component to do its job under challenging circumstances. However, legacy test tools usually focus on sending traffic that is standards-compliant. There are at least two pitfalls in the previous sentence: 1) Most real implementations have bugs that result in traffic being generated that might be close enough to the standard to mostly work, but that isn't what the load-oriented tools test. Their results are strictly only applicable when the traffic is perfectly compliant with the standards. 2) Test tools that transmit standards-compliant traffic must evolve at the pace of the standards. Having worked in the IETF and IEEE, I can tell you that the process is not speedy. It is deliberate, because the most important aspect of the deliverable is that the standard be correct.
However, legacy test tool vendors that want to deliver solutions based on standards will have to either wait for them to be "close enough" and then revise their tools (meaning that results from different versions of their tool may not be strictly comparable), or wait for them to be done, and possibly lose in the marketplace.
There is a hidden pitfall, too, in that the test tools that are focused on testing protocols in isolation are, by definition, not testing the service holistically. They will always miss the forest for the trees. In fact, testing the relevant applications like IPTV or transport like LTE in conjunction with IMS becomes increasingly important.
IMS involves SIP (with 3GPP extensions), as well as Diameter (many interfaces, again with a 3GPP flavor), and other protocols, from the essential but mundane (Routing, etc.) to the essential (MGCP and H.248, for interfacing to the legacy telephone network; DNS, for resolving names but also for ENUM and other telephony issues).
Testing IP Services holistically involves a core element of modeling the service in a meaningful way. That means that a testing solution must not be restricted to only testing one protocol at a time. The test tool is effectively the conductor of an orchestra, making the right protocol exchanges happen in the right order, under the right circumstances. Once a model exists that is comprehensive enough, one wants to leverage that into multiple types of testing.
For instance, one test methodology popular in software testing is called "data-driven" testing. The idea is that each test case involves setting a group of parameters with particular values and bundling one or more assertions with each test case so that the test tool can evaluate a pass/fail criterion with each set of parameter values that define that particular test case.
Lastly, the other important attribute of testing based on a model of the IP Service is the ability to re-use the same basic model to do multiple types of testing, such as the data-driven testing outlined above. One could imagine being able to use the model for conformance and interoperability testing, for regression testing, and for many other types of stateful testing--at the service level, not at the component level!
IMS is a service that involves a carefully orchestrated set of protocols to offer a seamless user experience. Users don't interact with SIP. They make phone calls. Users don't interact with LDAP or Diameter... they use their Address Book or Contact database. These services have to work flawlessly for the user, so the test equipment should interact with the service as a user would. It should be able to test subsets of the service at any meaningful level of granularity.
Tom Maufer is a Board Member of the NGN IMS Forum and Director, Solutions Architecture, at Mu Dynamics. You can reach Tom at [email protected].