Using artifacts with gitlab-runner’s SSH executor

GitLab’s runner is used by the CI/CD system to run jobs on remote machines. It has various executors to run the jobs through, with the most common probably being Docker. However, in some situations you can’t run gitlab-runner directly on the machine, so you need to use the SSH executor to allow it to log in remotely.

As an aside, it’s also worth noting that the VirtualBox and Parallels executors also make use of the SSH executor to run jobs.

The problem with the SSH executor is that to make use of build artifacts you also need gitlab-runner on the system on which you are SSHing in to. I suspect in the majority of cases the reason for using SSH in the first place is because you can’t run gitlab-runner directly, so this is a problem. For me, this is because I have a whole bunch of diverse systems inside VirtualBox and it’s not worth the time and effort to get gitlab-runner working on all of them. I also have a SPARC system and I’m not even sure if Go (which gitlab-runner is written in) even works there.

So taking a step back I thought about what’s going on. To download and upload artifacts gitlab-runner is executed remotely over SSH. It only has the information given to it on the command line (and in the environment, although I don’t think it uses it), and it communicates using HTTP to GitLab’s API. This can’t be that hard to replicate with normal tools, can it? It turns out that for the simple case (single artifact down and up), it is:

#!/bin/sh

if [ X"${1}" = X"artifacts-downloader" ]; then
    curl --silent --location --header "JOB-TOKEN: ${5}" "${3}api/v4/jobs/${7}/artifacts" > artifacts.zip
    unzip -q artifacts.zip
    rm artifacts.zip
fi

if [ X"${1}" = X"artifacts-uploader" ]; then
    zip -qr artifacts.zip ${9}
    E=`echo ${11} | tr " " "+"` export E
    curl --silent --location --header "JOB-TOKEN: ${5}" -F 'file=@artifacts.zip' "${3}api/v4/jobs/${7}/artifacts?artifact_format=${13}&artifact_type=${15}&expire_in=${E}" >/dev/null
    rm artifacts.zip
fi

Latest version available on GitHub.

Yes, this is a hacky, and it’s liable to break if the API changes, but it’s solved an irritating problem without too much effort. The only tools it uses that might need installing are curl, unzip and zip. I’d be surprised if these aren’t easily available on pretty much every Unix-like system.

It would be nicer if gitlab-runner included a small shim like this that it could copy over when using the SSH executor. Or maybe it could download and upload artifacts outside of the system and use SCP to copy them over. But in the mean time this hack will suffice.