Testing in Github

Introduction

This guide is intended for XMPP Server developers who are building on Github Actions. It demonstrates how to add XMPP Interoperability and Standards Conformance tests integrated into a build & test pipeline. This assumes that you already have a pipeline established, and wish to add the additional checks.

The checks run as their own action within the pipeline, in which they’ll execute tests and output junit-esque and debugging output. The only prerequisite is that you’ve got a built server and have started it.

Assuming that you have a pre-existing pipeline that build your server and starts it for integration testing, then adding our Github Action is as easy as adding one step to your pipeline, like this:

- name: Run XMPP Interoperability Tests against CI server.
  uses: XMPP-Interop-Testing/xmpp-interop-tests-action@v1.0
  with:
    domain: 'shakespeare.lit'
    adminAccountUsername: 'juliet'
    adminAccountPassword: 'O_Romeo_Romeo!'

The above assumes that your server is running and reachable on the domain shakespeare.lit and that it is provisioned with an administrative account (one that is allowed to create other users, per XEP-0133) that uses the username juliet and the provided password. You’ll find the full range of available configuration options below.

A Full Example

Assume that you have a pre-existing build pipeline that looks like this for building an XMPP server called “eXaMPPle”:

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Java
        uses: actions/setup-java@v4
      - name: Build with Maven
        run: mvn package
      - name: Stash the built artifacts
        uses: actions/upload-artifact@v4
        with:
          name: eXaMPPle
          path: target/example.jar

Here, you check out the code, set up some tooling, then run a process that builds, unit tests and produces a binary output, which we then archive as an artifact.

A second job that may already exist uses the build output, and launches a server to be used for integration testing.

  test:
    name: Compliance Tests
    needs: build # Sequential, not parallel to the build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout convenience scripts
        uses: actions/checkout@v4
        with:
          sparse-checkout: |
            ci-scripts

      - name: Download distribution artifact from build job
        uses: actions/download-artifact@v4
        with:
          name: eXaMPPle
          path: .

      - name: Start server from build job
        id: startServer
        uses: ./ci-scripts/launch-server # Launch the eXaMPPle server with test configs

      - name: Run eXaMMple integration tests against CI server
        run: ./ci-scripts/execute-tests # Run tests that already existed in your pipeline

We’ve added a second job to the end of the first which fetches a launch script and the freshly-built binary, runs the launch script to run the eXaMPPle server in the background, then calls some script to run whatever tests that you already had in your project.

The above combines into a pipeline that you may already have in your project, or something that you would add in preparation of running our Interoperability Tests. It is all highly specific to your project.

To run our Interoperability Tests, only one step needs to be added to this:

      - name: Run XMPP Interoperability Tests against CI server.
        uses: XMPP-Interop-Testing/xmpp-interop-tests-action@v1.2
        with:
          domain: 'shakespeare.lit'
          adminAccountUsername: 'juliet'
          adminAccountPassword: 'O_Romeo_Romeo!'

This step will use our Github Action to run the XMPP Interoperability Tests against your server. Look below for additional configuration options.

For completeness, here is the full pipeline, combining everything above:

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Java
        uses: actions/setup-java@v4
      - name: Build with Maven
        run: mvn package
      - name: Stash the built artifacts
        uses: actions/upload-artifact@v4
        with:
          name: eXaMPPle
          path: target/example.jar

  test:
    name: Compliance Tests
    needs: build # Sequential, not parallel to the build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout convenience scripts
        uses: actions/checkout@v4
        with:
          sparse-checkout: |
            ci-scripts

      - name: Download distribution artifact from build job
        uses: actions/download-artifact@v4
        with:
          name: eXaMPPle
          path: .

      - name: Start server from build job
        id: startServer
        uses: ./ci-scripts/launch-server # Launch the eXaMPPle server with test configs

      - name: Run eXaMMple integration tests against CI server
        run: ./ci-scripts/execute-tests # Run tests that already existed in your pipeline

      - name: Run XMPP Interoperability Tests against CI server
        uses: XMPP-Interop-Testing/xmpp-interop-tests-action@v1.0
        with:
          domain: 'shakespeare.lit'
          adminAccountUsername: 'juliet'
          adminAccountPassword: 'O_Romeo_Romeo!'

Configuration

Various options are available when calling xmpp_interop_tests_action, and whilst none of them are absolutely required, the defaults are unlikely to be perfect for everyone.

For the latest updates to the documentation of the configuration updates, consult the GitHub repository.

Next steps

Now that you have integrated the XMPP Interoperability and Standards Conformance tests into your build pipeline, it is time to start working with the tests!

We’ve compiled the following resources to help you with any run-time challenges that you may run into:

• Choosing what test to run - helps you choose how to disable specific tests (useful when you want to ignore a particular test for some reason).
• Diagnose test failures - explains what you can do to find the reason that test is failing. Did we identify an issue in your server software? Find out!

Splash image courtesy of Roman Synkevych, Unsplash