Testing in CircleCI

Introduction

This guide is intended for XMPP Server developers who are building their server in a Circle CI. It demonstrates how to add XMPP Interoperability and Standards Conformance tests integrated into a build & test pipeline. This assumes that you already have a pipeline established, and wish to add the additional checks.

The checks run as their own action within the pipeline, in which they’ll execute tests and output junit-esque and debugging output. The only prerequisite is that you’ve got a built server and have started it.

Assuming that you have a pre-existing pipeline that build your server and starts it for integration testing, then adding our Circle CI Orb is as easy as adding one step to your pipeline, like this:

- xmpp-interop-tests/test:
    domain: shakespeare.lit
    timeout: 60000
    adminAccountUsername: juliet
    adminAccountPassword: O_Romeo_Romeo!

This example demonstrates one possible way to provision accounts. It assumes your server is running and accessible at shakespeare.lit, and that it already has an administrative account (one that is allowed to create other users, per XEP-0133) that uses the username juliet and the provided password. Other configuration methods are also available, and you can explore the full set of options in the sections below.

A Full Example

Assume that you have a pre-existing build pipeline that looks like this for building your XMPP server:

description: >
  Run an interop tests step as part of an XMPP server build process

usage:
  version: 2.1

  jobs:
    build:
      docker:
        - image: cimg/base:2024.02
      steps:
        - run: echo "Build your server here"
        - persist_to_workspace:
            paths:
              - ./my-server

  workflows:
    build-my-server:
      jobs:
        - build

Here, we are hand-waving around the actual check out of the code and setting up of tooling. We do make available the server artifact as a path, to be used further down the line.

A second job that may already exist uses the build output, and launches a server to be used for integration testing.

    test:
      docker:
        - image: cimg/openjdk:17.0
      steps:
        - run:
            command: ./my-server/run.sh
            background: true
        - run:
            command: ./my-server/execute-my-tests.sh

The workflow is adjusted to run the additional test step after the build step completes:

  workflows:
    build-my-server:
      jobs:
        - build
        - test:
            requires:
              - build

We’ve added a second job to the end of the first which fetches a launch script and the freshly-built binary, runs the launch script to run the server in the background, then calls some script to run whatever tests that you already had in your project.

The above combines into a pipeline that you may already have in your project, or something that you would add in preparation of running our Interoperability Tests. It is all highly specific to your project.

To run our Interoperability Tests, only one step needs to be added to this. This step makes use of our Orb, that you’ll need to define in the top of your configuration:

  orbs:
    xmpp-interop-tests: xmpp-interop-tests/tests@1.6.0

With the Orb defined, a step that uses it can be added to your pre-existing job:

      - xmpp-interop-tests/test:
          domain: shakespeare.lit
          timeout: 60000
          adminAccountUsername: juliet
          adminAccountPassword: O_Romeo_Romeo!

This step will use our Circle CI Orb to run the XMPP Interoperability Tests against your server. Look below for additional configuration options.

For completeness, here is the full pipeline, combining everything above:

description: >
  Run an interop tests step as part of an XMPP server build process

usage:
  version: 2.1

  orbs:
    xmpp-interop-tests: xmpp-interop-tests/tests@1.6.0

  jobs:
    build:
      docker:
        - image: cimg/base:2024.02
      steps:
        - run: echo "Build your server here"
        - persist_to_workspace:
            paths:
              - ./my-server
    test:
      docker:
        - image: cimg/openjdk:17.0
      steps:
        - run:
            command: ./my-server/run.sh
            background: true
        - run:
            command: ./my-server/execute-my-tests.sh
        - xmpp-interop-tests/test:
            domain: shakespeare.lit
            timeout: 60000
            adminAccountUsername: juliet
            adminAccountPassword: O_Romeo_Romeo!

  workflows:
    build-my-server:
      jobs:
        - build
        - test:
            requires:
              - build

Configuration

Various options are available when calling xmpp-interop-tests, and whilst none of them are absolutely required, the defaults are unlikely to be perfect for everyone.

For the latest updates to the documentation of the configuration updates, consult the GitHub repository of the CircleCI orb.

Provisioning Test Accounts

To be able to run the tests, the server that is being tested needs to be provisioned with test accounts. Three different mechanisms can be used for this:

• Admin Account - By configuring the username and password of a pre-existing administrative user, using the adminAccountUsername and adminAccountPassword configuration options, three test accounts will be created using XEP-0133: Service Administration functionality.
• Explicit Test Accounts - You can configure three pre-existing accounts that will be used for testing, using the accountOneUsername, accountOnePassword, accountTwoUsername, accountTwoPassword, accountThreeUsername and accountThreePassword configuration options.
• In-Band Registration - If no admin account and no explicit tests accounts are provided, in-band registration (XEP-0077) will be used to provision accounts.

For more information on provisioning accounts, consult the ‘Test Account Provisioning’ guide.

Next steps

Now that you have integrated the XMPP Interoperability and Standards Conformance tests into your build pipeline, it is time to start working with the tests!

We’ve compiled the following resources to help you with any run-time challenges that you may run into:

• Choosing what test to run - helps you choose how to disable specific tests (useful when you want to ignore a particular test for some reason).
• Diagnose test failures - explains what you can do to find the reason that test is failing. Did we identify an issue in your server software? Find out!

Splash image courtesy of Circle CI Blogs