One of XMPP’s largest challenges (and indeed of all open standards) is interoperability. We want to enable anyone to use the standards on any platform they like, written in the programming language they prefer, focussing design and utility toward any group, from enterprise to egg-cup enthusiasts.
When we came up with our idea for XMPP interop tests for server implementations, we knew we weren’t the first folk to ever have the idea. What was important for us wasn’t about getting the tests right, but about getting the tests used. With so many programming languages in play, we can’t get almost everyone to adopt an additional language into their toolchain. That’s a clear no-go.
Instead, we’re choosing to wrap our tests in the packaging of the common CI systems (a GitHub Action, a Circle CI Orb, etc) so that whatever your regular toolchain, you can slot the tests into the test phase of your build pipeline.
Who’s this for?
The intended target audience of this project consists of XMPP server software developers. The tests that are provided by this project aim to identify problems in the implementation of XMPP server software. We provide tooling that is explicitly designed to be used in the continuous integrations systems that are typically used in server software development.
Our project is not designed verify the configuration of an XMPP server. Although there is overlap, these are two different beasts. For people that administer an XMPP server, this project is probably less suitable. They likely benefit more from tooling like the XMPP Compliance Tester.
Installation & Configuration
If you’d like to learn how to integrate XMPP standards conformance tests in your build pipeline, pick your continuous integration system below:
Usage
After integrating our tests in your build pipeline, you might be interested in the following usage documentation.
What is being tested?
The tests mimic clients that interact with the server under test. All tests are client-to-server based tests. Server-to-server tests aren’t currently included, but those might be added in the future.
The tests verify behavior as specified by the many RFCs and XEPs that govern the XMPP protocol. At the time of writing, tests exist for these specifications:
- RFC 6120: Extensible Messaging and Presence Protocol (XMPP): Core
- RFC 6121: Extensible Messaging and Presence Protocol (XMPP): Instant Messaging and Presence
- XEP-0030: Service Discovery
- XEP-0045: Multi-User Chat
- XEP-0048: Bookmarks
- XEP-0050: Ad-Hoc Commands
- XEP-0054: vcard-temp
- XEP-0060: Publish-Subscribe
- XEP-0080: User Location
- XEP-0085: Chat State Notifications
- XEP-0092: Software Version
- XEP-0096: SI File Transfer
- XEP-0107: User Mood
- XEP-0115: Entity Capabilities
- XEP-0118: User Tune
- XEP-0198: Stream Management
- XEP-0199: XMPP Ping
- XEP-0215: External Service Discovery
- XEP-0232: Software Information
- XEP-0313: Message Archive Management
- XEP-0347: Internet of Things - Discovery
- XEP-0352: Client State Indication
- XEP-0363: HTTP File Upload
- XEP-0374: OpenPGP for XMPP Instant Messaging
- XEP-0384: OMEMO Encryption
- XEP-0486: MUC Avatars
The set of tested specifications changes frequently (and we might not be the best at keeping the list on this page completely up-to-date).
Our tests do not necessarily cover the listed specifications completely, for a couple of reasons:
- We’ve simply not been able to finish all tests. The amount of XMPP-related specifications numbers in the hundreds. On top of that: some specifications are lengthy, lengthy documents, for which more effort is needed to complete test coverage. Feel free to lend us a hand, either by implementing tests yourself, or making available resources for us to do so! If you’re considering this, please reach out to us.
- Secondly, we have opted to cover only the bits of the specification that are defined to be mandatory (‘REQUIRED’, ‘MUST’ and ‘SHALL’ in RFC 2119-speak). Other requirement indicators (‘SHOULD’, ‘MAY’) signal wiggle-room: for those parts of a specification, a server could possibly act differently than what is specified, which can be acceptable in certain contexts. We prefer to have test failures to unequivocally indicate a failure (which is hard enough in itself)! We might add tests for other requirement indicators in the future (guarded by some kind of configuration option), but for now, there are plenty of tests that we can create that do not suffer from this ambiguity. We’ve opted to start with those.
The order in which we’re creating tests is loosely based on the XMPP Compliance Suites, that provide guidance on which specifications are important when implementing an application of a certain kind. We’ve also added tests that happen to be already available, or that were created in an effort to tackle some kind of development issue that one of us was working on in the past.