Github Action: How to parallelize tests dynamically by folder?

#github#action#CI#tutorial
Post available in: Français

Once again, I gave myself a challenge to learn how to use Github Action. I tried to see if it was possible to dynamically launch a job for each folder in a directory.

We often want to run tests in parallel to reduce the time of continuous integration. Often tools like Jest or CucumberJS have parallelization features, but these often face the limits of the power available on the machines on which you want to run them.

tests/
  user/
    foo.feature
    bar.feature
  account/
    example.feature
  live/
    a.feature
    video.feature

I wondered if it wouldn’t be more efficient to run the tests in parallel directly with the CI tool so that only small runners would be needed. In the previous example, I would like to be able to run in parallel on a separate runner for each folder in test/.

A Solution with Github action

name: CI

on:
  push:
    branches: [main]

jobs:
  directories: # Job that list subdirectories
    runs-on: ubuntu-latest
    outputs:
      dir: ${{ steps.set-dirs.outputs.dir }} # generate output name dir by using inner step output
    steps:
      - uses: actions/checkout@v2
      - id: set-dirs # Give it an id to handle to get step outputs in the outputs key above
        run: echo "::set-output name=dir::$(ls -d */ | jq -R -s -c 'split("\n")[:-1]')"
        # Define step output named dir base on ls command transformed to JSON thanks to jq
  loop:
    runs-on: ubuntu-latest
    needs: [directories] # Depends on previous job
    strategy:
      matrix:
        dir: ${{fromJson(needs.directories.outputs.dir)}} # List matrix strategy from directories dynamically
    steps:
      - run: echo ${{matrix.dir}}

It’s up to you to adapt it to your needs!


Written by Antoine Caron who lives and works in Lyon (France) building useful things at Bedrock. You should follow him on Twitter. Take a look at my friends websites. You could also take a look at the conferences I gave. If you want to see my professional background, my resume is available here.

If you like some content of this blog, or it has helped you, please consider donating to the Abbé Pierre Foundation which I personally support.
"We are only ever happy in the happiness we give. To give is to receive."