Ever heard of XMPP Interop Testing? It’s this cool project that helps make sure different XMPP servers can all work together smoothly. Our XMPP Interop Testing project provides a suite of automated tests that can be integrated into CI/CD pipelines to verify the compliance and interoperability of XMPP server implementations.
Late last year, we reported that we had secured funding graciously provided by NLnet that allowed us to massively build out this project. This blog has been a bit quiet since then, but work has progressed. Significantly.
We have just released version 1.6.0 of all our test runners. With this release, we (again) more than doubled the total number of XMPP interop tests! By my count, our project now lets loose 933 tests on your XMPP server implementation!
The biggest chunk of work has gone into tests that verify parts of the basic XMPP protocol, notably for testing functionality that involves roster management (as specified in section 2 of RFC 6121) and for server rules for processing XML stanzas (section 8 of RFC 6121).
Additionally, a couple of new specifications are now being tested by our framework! Tests have been added for:
- XEP-0133: Service Administration
- XEP-0410: MUC Self-Ping (Schrödinger’s Chat)
- XEP-0421: Occupant identifiers for semi-anonymous MUCs
- XEP-0433: Extended Channel Search
This table gives a complete comparison of test coverage between versions 1.5.0 and 1.6.0.
| Specification | v1.5.0 | v1.6.0 | Difference |
|---|---|---|---|
| unknown | 13 | 13 | 0 |
| RFC 6120 | 1 | 1 | 0 |
| RFC 6121 | 11 | 402 | 391 |
| XEP-0030 | 19 | 19 | 0 |
| XEP-0045 | 252 | 252 | 0 |
| XEP-0048 | 1 | 1 | 0 |
| XEP-0050 | 4 | 4 | 0 |
| XEP-0054 | 10 | 10 | 0 |
| XEP-0060 | 24 | 24 | 0 |
| XEP-0080 | 2 | 2 | 0 |
| XEP-0085 | 1 | 1 | 0 |
| XEP-0092 | 1 | 1 | 0 |
| XEP-0096 | 2 | 2 | 0 |
| XEP-0107 | 2 | 2 | 0 |
| XEP-0115 | 12 | 12 | 0 |
| XEP-0118 | 2 | 2 | 0 |
| XEP-0133 | 0 | 44 | 44 |
| XEP-0198 | 10 | 10 | 0 |
| XEP-0199 | 2 | 2 | 0 |
| XEP-0215 | 6 | 6 | 0 |
| XEP-0232 | 1 | 1 | 0 |
| XEP-0313 | 2 | 2 | 0 |
| XEP-0347 | 3 | 3 | 0 |
| XEP-0352 | 6 | 6 | 0 |
| XEP-0363 | 12 | 12 | 0 |
| XEP-0374 | 2 | 2 | 0 |
| XEP-0384 | 4 | 4 | 0 |
| XEP-0410 | 0 | 3 | 3 |
| XEP-0421 | 0 | 67 | 67 |
| XEP-0433 | 0 | 19 | 19 |
| XEP-0486 | 4 | 4 | 0 |
To be clear: the work doesn’t end here. There is still significant improvement to be made (and we’ve not yet used up all of the grant either!) - we just liked to give you all an update. In the works are additional test implementations, and a couple of new test runners. That should both increase coverage, but also allow our tests to be executed on even more CI/CD platforms!
Please get in touch if you have any ideas for improvement, or other feedback. We’d love to hear from you!
- Splash image courtesy of Imagine Buddy, Unsplash