As a great fan of measuring things, I have recently added code coverage metrics to my API testing utility Rester. Rester is a Swift Package Manager (SPM) project and after researching a few of the available tools it became clear that most, if not all, are designed for Xcode-built iOS and macOS projects and don’t support SPM out of the box.

Eventually, I settled on Codecov and managed to get it to work with my SPM project with a little tweaking. One very nice feature of Codecov is its intergration with Github that not only shows you your current coverage but also how pull requests will change it.

Code coverage report in Github pull requests

Note that I am using Travis as the continuous integration (CI) tool for Rester but this setup should be easily adaptable to other CI providers.

The way Codecov works is the following:

run your tests with code coverage metrics enabled to generate coverage data
run a script provided by Codecov to gather that data and upload it to the service after the test run

The set up of Codecov itself is pretty simple: You sign up to Codecov with your Github account and set the CODECOV_TOKEN API token in your Travis environment variables. This authorises you CI job to write coverage data to your codecov account.

Next, you call the Codecov script and send your coverage data:

bash <(curl -s https://codecov.io/bash)

You can do this manually to test the set up, but eventually you’ll want to add this to your build script for it to happen automatically after each CI build.

If you are running macOS or iOS tests you are already done at this point, because it can find the metrics and the binary they are referring to automatically.

Not quite so for SPM projects, because binaries and frameworks are in different places and you need to guide Codecov a little.

Rather than tweak the Codecov upload script to find the different products, I’ve chosen to simply use an additional build step with xcodebuild for my SPM project to generate the expected results.

So while you normally run the tests with swift test, instead to generate coverage we run them with xcodebuild. This is probably a good idea anyway as it allows you test additional platforms like iOS, tvOS, etc.

The one thing to look out for is to ensure the products are written to a path inside your project. This is where Codecov is expecting to find results. Here’s what this looks like:

test-macos-xcode:
    swift package generate-xcodeproj
    xcodebuild test \
            -scheme Rester \
            -destination platform="macOS" \
            -enableCodeCoverage YES \
            -derivedDataPath .build/derivedData

This generates the derived data inside SPM’s .build directory.

The one caveat is that Codecov will only pick up coverage from app, framework, and xctest products. It’s looking for bundles with those folder name extensions specifically, as you can see if you inspect the script:

swiftcov() {
  _dir=$(dirname "$1" | sed 's/\(Build\).*/\1/g')
  for _type in app framework xctest
  do
    ...
  done
}

Now, as mentioned above, unless you change the script, it will only pick up library targets of your SPM build, because executable targets in SPM are not shipped as app bundles and therefore not found. Libraries, however, are discovered by Codecov.

What this means for your SPM project is that you need to bundle as much of your code into library targets as possible. In the case of Rester, I’ve shrunk the executable target within main.swift to simply import ResterCore – where I’ve placed all of Rester’s functionality. The (uncovered) executable target shrinks down to:

import ResterCore

app.run(ResterVersion)

With these caveats, here is what the complete Travis job looks like:

- name: macos xcode test
  stage: test
  os: osx
  osx_image: xcode10.2
  script: make test-macos-xcode
  after_success:
    # upload coverage data
    - bash <(curl -s https://codecov.io/bash) -J '^ResterCore$' \
        -D .build/derivedData

You can find the full instrumentation of this in Rester’s github project.

Note that the -J parameter allows you to limit coverage reporting to certain modules. Here, I’m ensuring I only get coverage for the library and not the tests themselves.

Did you find this post useful? Do you have questions regarding the setup? Drop me a message on Twitter or via email. I’m happy to help – or just to hear if this helped get you up and running with code coverage in your Swift project.