Add-ons

Each category is named after a CI worker (e.g AppVeyor) or operating system (e.g Windows) and references add-ons designed to be used on the associated continuous integration service or system.

An add-on is a file that could either directly be executed or used as a parameter for an other tool.

Anyci

This a special category containing scripts that could be executed on a broad range of CI services.

ctest_junit_formatter

Add-on converting test results from CTest to JUnit format.

The add-on get the name of the latest build tag by reading the first line of <BUILD_DIR>/Testing/TAG, and then convert the file <BUILD_DIR>/Testing/<LATEST_TAG>/Test.xml. The conversion results is outputted on stdout.

This add-on supports both Python 2 and Python 3 and is based on stackoverlow answer contributed by Calvin1602 and MOnsDaR.

Usage:

ci_addons ctest_junit_formatter BUILD_DIR > JUnit.xml

Example of use from CircleCI:

$ mkdir ${CIRCLE_TEST_REPORTS}/CTest
$ ci_addons ctest_junit_formatter BUILD_DIR > ${CIRCLE_TEST_REPORTS}/CTest/JUnit-${CIRCLE_NODE_INDEX}.xml

Note

CircleCI will automatically aggregate test results generated by different node.

Example of CircleCI test summary with failing tests:

_images/ctest_junit_formatter_circleci_example.png

docker.py

Add-on facilitating docker use on CI services.

It allows to load an image from local cache, pull and save back using a convenience one-liner.

Usage:

ci_addons docker load-pull-save [-h] [--cache-dir CACHE_DIR] [--verbose]
                                     NAME[:TAG|@DIGEST]

Example:

$ ci_addons docker load-pull-save hello-world:latest
[anyci:docker.py] Loading cached image from /home/jcfr/docker/hello-world-latest.tar
[anyci:docker.py]   -> cached image not found
[anyci:docker.py] Pulling image: hello-world:latest
[anyci:docker.py]   -> done
[anyci:docker.py] Reading image ID from current image
[anyci:docker.py]   -> image ID: sha256:c54a2cc56cbb2f04003c1cd4507e118af7c0d340fe7e2720f70976c4b75237dc
[anyci:docker.py] Caching image
[anyci:docker.py]   -> image cached: /home/jcfr/docker/hello-world-latest.tar
[anyci:docker.py] Saving image ID into /home/jcfr/docker/hello-world-latest.image_id
[anyci:docker.py]   -> done

Note

  • Image is saved into the cache only if needed.

    In addition to the image archive (e.g image-name.tar), a file containing the image ID is also saved into the cache directory (e.g image-name.image_id).

    This allows to quickly read back the image ID of the cached image and determine if the current image should be saved into the cache.

noop.py

Display name of script and associated argument (basically the value of sys.argv).

publish_github_release.py

Add-on automating the creation of GitHub releases.

Based on the git branch found in the current working directory, it allows to automatically create a GitHub prerelease and/or release and upload associated packages.

Getting Started

To create a pre-release named latest:

ci_addons publish_github_release --prerelease-packages "dist/*"

To create a release named after the current tag:

ci_addons publish_github_release --release-packages "dist/*"

In both case, packages found in dist directory are uploaded.

Note

Pre-releases are created only if the current commit is NOT a tag (latest tag is automatically ignored). Similarly, releases are created ONLY if current commit is a tag (different from latest).

Terminology

Prerelease: By default, this corresponds to a GitHub prerelease associated with a tag named latest and named Latest (updated on YYYY-MM-DD HH:MM UTC). The prerelease is automatically updated each time the publish_github_release script is executed. Updating the latest prerelease means that (1) the latest tag is updated to point to the current HEAD, (2) the name is updated and (3) latest packages are uploaded to replace the previous ones. GitHub prerelease are basically release with draft option set to False and prerelease option set to True.

Release: This corresponds to a GitHub release automatically created by publish_github_release script only if it found that HEAD was associated with a tag different from latest. It has both draft and prerelease options set to False. Once packages have been associated with such a release, they are not expected to be removed.

Usage

ci_addons publish_github_release [-h]
                                 [--release-packages [PATTERN [PATTERN ...]]]
                                 [--prerelease-packages [PATTERN [PATTERN ...]]]
                                 [--prerelease-packages-clear-pattern PATTERN]
                                 [--prerelease-packages-keep-pattern PATTERN]
                                 [--prerelease-tag PRERELEASE_TAG]
                                 [--prerelease-name PRERELEASE_NAME]
                                 [--prerelease-sha PRERELEASE_SHA]
                                 [--token GITHUB_TOKEN]
                                 [--exit-success-if-missing-token]
                                 [--re-upload]
                                 [--display-python-wheel-platform]
                                 [--dry-run]
                                 ORG/PROJECT

Note

  • Packages to upload can be a list of paths or a list of globbing patterns.

Mini-language for packages selection

To facilitate selection of specific packages, if any of the strings described below are found in arguments associated with with either --prerelease-packages or --release-packages, they will be replaced.

<PYTHON_WHEEL_PLATFORM>: This string is replaced by the current platform as found in python wheel package names (e.g manylinux1, macosx, or win). Executing ci_addons publish_github_release --display-python-wheel-platform returns the same string.

<COMMIT_DATE>: This string is replaced by the YYYYMMDD date as returned by git show -s --format="%ci".

<COMMIT_SHORT_SHA>: This string is replaced by the sha as returned by git rev-parse --short=7 HEAD.

<COMMIT_DISTANCE>: This string is replaced by the distance to the tag specified using --prerelease-tag. If the tag does not exist, it corresponds to the number of commits. This is particularly useful when selecting prerelease packages generated using pep440-pre style implemented in python-versioneer.

Use case: Automatic upload of release packages associated with a tag

In this example, the script automatically detects that the current branch HEAD is associated with the tag 1.0.0 and automatically uploads all packages found in the dist directory.

$ cd PROJECT

$ git describe
1.0.0

$ ci_addons publish_github_release ORG/PROJECT \
  --release-packages "dist/*"
Checking if HEAD is a release tag
Checking if HEAD is a release tag - yes (found 1.0.0: creating release)

created '1.0.0' release
  Tag name      : 1.0.0
  ID            : 5436107
  Created       : 2017-02-13T06:36:29Z
  URL           : https://github.com/ORG/PROJECT/releases/tag/1.0.0
  Author        : USERNAME
  Is published  : True
  Is prerelease : False

uploading '1.0.0' release asset(s) (found 2):
  uploading dist/sandbox-1.0.0-cp27-cp27m-manylinux1.whl
  download_url: https://github.com/ORG/PROJECT/releases/download/1.0.0/sandbox-1.0.0-cp27-cp27m-manylinux1.whl

  uploading dist/sandbox-1.0.0-cp35-cp35m-manylinux1.whl
  download_url: https://github.com/ORG/PROJECT/releases/download/1.0.0/sandbox-1.0.0-cp35-cp35m-manylinux1.whl

Use case: Automatic creation of “nightly” prerelease from different build machines

When building projects using continuous integration services (e.g Appveyor, TravicCI, or CircleCI), the publish_github_release script has the following responsibilities:

  • update the nightly tag reference

  • update the release name

  • keep only the most recent packages. This means that after successfully uploading package generating on a given platform, the older ones will be removed.

To fulfill its requirements, publish_github_release provides two convenient options --prerelease-packages-clear-pattern and --prerelease-packages-keep-pattern.

prerelease-packages-clear-pattern: This option allows to select all packages that should be removed from the prerelease. For example, on a machine responsible to generate windows python wheels, the following pattern can be used :"*win*.whl".

prerelease-packages-keep-pattern: This option allows to keep packages that have been selected by the previous globbing pattern. For example, assuming development package names contain the date of the commit they are built from, specifying a globbing pattern with the date allows to delete older packages while keeping only the new ones built from that commit.

In the following example, we assume a prerelease done on 2017-02-12 with 16 packages (4 linux, 4 macosx, and 8 windows) already exists. The command reported below corresponds to the execution of the script on a linux machine, after one additional commit has been done the next day.

$ cd PROJECT

$ git describe
1.0.0-2-g9d40177

$ commit_date=$(git log -1 --format="%ad" --date=local | date +%Y%m%d)
$ echo $commit_date
20170213

$ ci_addons publish_github_release ORG/PROJECT \
  --prerelease-packages dist/*.dev${commit_date}*manylinux1*.whl \
  --prerelease-packages-clear-pattern "*manylinux1*.whl" \
  --prerelease-packages-keep-pattern "*.dev${commit_date}*.whl"
Checking if HEAD is a release tag
Checking if HEAD is a release tag - no (creating prerelease)

release nightly: already exists

uploading 'nightly' release asset(s) (found 4):
  uploading dist/sandbox-1.0.0.dev20170213-cp27-cp27m-manylinux1_x86_64.whl
  download_url: https://github.com/ORG/PROJECT/releases/download/nightly/sandbox-1.0.0.dev20170213-cp27-cp27m-manylinux1_x86_64.whl

  uploading dist/sandbox-1.0.0.dev20170213-cp34-cp34m-manylinux1_x86_64.whl
  download_url: https://github.com/ORG/PROJECT/releases/download/nightly/sandbox-1.0.0.dev20170213-cp34-cp34m-manylinux1_x86_64.whl

  uploading dist/sandbox-1.0.0.dev20170213-cp35-cp35m-manylinux1_x86_64.whl
  download_url: https://github.com/ORG/PROJECT/releases/download/nightly/sandbox-1.0.0.dev20170213-cp35-cp35m-manylinux1_x86_64.whl

  uploading dist/sandbox-1.0.0.dev20170213-cp36-cp36m-manylinux1_x86_64.whl
  download_url: https://github.com/ORG/PROJECT/releases/download/nightly/sandbox-1.0.0.dev20170213-cp36-cp36m-manylinux1_x86_64.whl

deleting 'nightly' release asset(s) (matched: 8, matched-but-keep: 4, not-matched: 12):
  deleting sandbox-1.0.0.dev20170212-cp27-cp27m-manylinux1_x86_64.whl
  deleting sandbox-1.0.0.dev20170212-cp34-cp34m-manylinux1_x86_64.whl
  deleting sandbox-1.0.0.dev20170212-cp35-cp35m-manylinux1_x86_64.whl
  deleting sandbox-1.0.0.dev20170212-cp36-cp36m-manylinux1_x86_64.whl
  nothing to delete

resolved 'master' to '9d40177e6d3a69890de8ea359de2d02a943d2e10'
updating 'nightly' release:
  target_commitish: '62fe605938ff252e4ddee05b5209299a1aa9a39e' -> '9d40177e6d3a69890de8ea359de2d02a943d2e10'
  tag_name: 'nightly' -> 'nightly-tmp'

deleting reference refs/tags/nightly
updating 'nightly-tmp' release:
  tag_name: 'nightly-tmp' -> 'nightly'

deleting reference refs/tags/nightly-tmp
updating 'nightly' release:
  target_commitish: '62fe605938ff252e4ddee05b5209299a1aa9a39e' -> '9d40177e6d3a69890de8ea359de2d02a943d2e10'

Use case: Automatic creation of GitHub releases and prereleases

This can be done by combining the options --release-packages and --prerelease-packages.

Note also the use of --display-python-wheel-platform to automatically get the current python platform.

For example:

$ commit_date=$(git log -1 --format="%ad" --date=local | date +%Y%m%d)

$ platform=$(ci_addons publish_github_release ORG/PROJECT --display-python-wheel-platform)
$ echo $platform
manylinux1

$ ci_addons publish_github_release ORG/PROJECT \
    --release-packages "dist/*" \
    --prerelease-packages dist/*.dev${commit_date}*${platform}*.whl \
    --prerelease-packages-clear-pattern "*${platform}*.whl" \
    --prerelease-packages-keep-pattern "*.dev${commit_date}*.whl"

The same can also be achieved across platform using the convenient mini-language for package selection:

$ ci_addons publish_github_release ORG/PROJECT \
    --release-packages "dist/*" \
    --prerelease-packages "dist/*.dev<COMMIT_DATE>*<PYTHON_WHEEL_PLATFORM>*.whl" \
    --prerelease-packages-clear-pattern "*<PYTHON_WHEEL_PLATFORM>*.whl" \
    --prerelease-packages-keep-pattern "*.dev<COMMIT_DATE>*.whl"

Testing

Since the add-on tests interact with GitHub API, there are not included in the regular scikit-ci-addons collection of tests executed using pytest. Instead, they needs to be manually executed following these steps:

  1. Generate a personal access token with at least public_repo scope enabled.

  2. Create a test project on GitHub with at least one commit.

  3. Check out sources of your test project.

  4. Create a virtual environment, download scikit-ci-addons source code, and install its requirements.

  5. Execute the test script.

For example:

export GITHUB_TOKEN=...   # Change this with the token generated above in step (1)
TEST_PROJECT=jcfr/sandbox # Change this with the project name created above in step (2)

cd /tmp
git clone https://github.com/scikit-build/scikit-ci-addons
cd scikit-ci-addons/
mkvirtualenv scikit-ci-addons-test
pip install -r requirements.txt
SRC_DIR=$(pwd)

cd /tmp
git clone https://github.com/$TEST_PROJECT test-project
cd test-project

python $SRC_DIR/anyci/tests/test_publish_github_release.py $TEST_PROJECT --no-interactive

run.sh

Wrapper script executing command and arguments passed as parameters.

Appveyor

Warning

These scripts were designed to work on worker from http://appveyor.com/

They have been retired and are available only in scikit-ci-addons <= 0.25.0

See https://scikit-ci-addons.readthedocs.io/en/0.25.0/addons.html#appveyor

enable-worker-remote-access.ps1

install_cmake.py

run-with-visual-studio.cmd

patch_vs2008.py

rolling-build.ps1

tweak_environment.py

Circle

These scripts are designed to work on worker from http://circleci.com/

install_cmake.py

Download and install in the PATH the specified version of CMake binaries.

Usage:

ci_addons circle/install_cmake.py X.Y.Z

Example:

$ ci_addons circle/install_cmake.py 3.6.2

Note

  • The script will skip the download in two cases:

    • if current version matches the selected one.

    • if archive already exist in $HOME/downloads directory.

  • Adding directory $HOME/downloads to the CircleCI cache can speed up the build. For more details, see Caching Dependencies.

Travis

Warning

These scripts were designed to work on worker from http://travis-ci.org/

They have been retired and are available only in scikit-ci-addons <= 0.25.0

See https://scikit-ci-addons.readthedocs.io/en/0.25.0/addons.html#appveyor

install_cmake.py

install_pyenv.py

enable-worker-remote-access.sh

Windows

These scripts are designed to work on any windows workstation running Windows 7 and above and can be directly used from a powershell terminal (or command line terminal) using a simple one-liner.

Content of the scripts can easily be inspected in the associated source repository.

For example, on a new system without python or git installed, they can be installed from a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-python-36-x64.ps1'))
iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-git.ps1'))

Read here to learn about the powershell execution policy.

Details for each install-*.ps1 scripts are reported below.

install-cmake.ps1

Install selected CMake version in C:\cmake-X.Y.Z.

From a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

$cmakeVersion="3.22.2"
iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-cmake.ps1'))

Note

  • CMake is NOT added to the PATH

  • setting $cmakeVersion to “X.Y.Z” before executing the script allows to select a specific CMake version.

  • on AppVeyor, the download and install can be skipped by adding directory C:\cmake-X.Y.Z to the cache. For more details, see https://www.appveyor.com/docs/build-cache/#configuring-cache-items

Note

install-flang.ps1

Install latest flang in a new conda environment named flang-env.

From a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-flang.ps1'))

Flang is a Fortran compiler targeting LLVM, it was announced in 2015.

Source code is hosted on GitHub at https://github.com/flang-compiler/flang, the windows fork is hosted as https://github.com/isuruf/flang

Note

Note

  • this script is intended to work with Miniconda 4.5.4

install-git.ps1

Install Git 2.11.0 (including Git Bash) on the system.

From a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-git.ps1'))

Note

  • Git executables are added to the PATH

Note

install-miniconda3.ps1

Install latest miniconda3 environment into C:\Miniconda3.

From a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-miniconda3.ps1'))

Note

  • miniconda environment is NOT added to the PATH and registry.

Note

install-miniconda3-4-5-4.ps1

Install Miniconda3 4.5.4 environment into C:\Miniconda3.

From a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-miniconda3-4-5-4.ps1'))

Note

  • miniconda environment is NOT added to the PATH and registry.

Note

install-ninja.ps1

Install ninja executable v1.7.2 into C:\ninja-1.7.2.

From a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-ninja.ps1'))

Note

  • ninja executable is NOT added to the PATH

Note

install-nsis.ps1

Install NSIS 3.01 on the system.

From a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-nsis.ps1'))

Note

  • nsis executable is added to the PATH

Note

install-python.ps1

Install Python 3.7, 3.8, 3.9, 3.10, 3.11 and 3.12 (32 and 64-bit) along with pip and virtualenv in the following directories:

C:\Python37-x64
C:\Python37-x86

C:\Python38-x64
C:\Python38-x86

C:\Python39-x64
C:\Python39-x86

C:\Python39-x64
C:\Python39-x86

C:\Python310-x64
C:\Python310-x86

C:\Python311-x64
C:\Python311-x86

C:\Python312-x64
C:\Python312-x86

From a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-python.ps1'))

Note

  • python interpreter is NOT added to the PATH

  • setting $pythonVersion to either “3.7”, “3.8”, “3.9”, “3.10”, “3.11” or “3.12” before executing the script allows to install a specific version. By default, all are installed.

  • setting $pythonArch to either “86”, “32” or “64” before executing the script allows to install python for specific architecture. By default, both are installed. Values “86” and “32” correspond to the same architecture.

  • setting $pythonPrependPath to 1 will add install and Scripts directories the PATH and .PY to PATHEXT. This variable should be set only if $pythonVersion and $pythonArch are set. By default, the value is 0.

Note

Warning

  • The downloaded versions of python may NOT be the latest version including security patches. If running in a production environment (e.g webserver), these versions should be built from source.

install-svn.ps1

Install Slik SVN 1.9.5 in the following directory:

C:\SlikSvn

From a powershell terminal open as administrator:

Set-ExecutionPolicy Unrestricted -Force
[System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48

iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-svn.ps1'))

Note

  • svn executable is added to the PATH

Note

install-utils.ps1

This script is automatically included (and downloaded if needed) by the other addons, it provides convenience functions useful to download and install programs:

Always-Download-File($url, $file):

Systematically download $url into $file.

Download-File($url, $file):

If file is not found, download $url into $file.

Download-URL($url, $downloadDir):

Download $url into $downloadDir. The filename is extracted from $url.

Install-MSI($fileName, $downloadDir, $targetDir):

Programatically install MSI installers $downloadDir$fileName into $targetDir. The package is installed for all users.

Which($progName)

Search for $progName in the PATH and return its full path.

Download-7zip($downloadDir):

If not found, download 7zip executable 7za.exe into $downloadDir. The function returns the full path to the executable.

Always-Extract-Zip($filePath, $destDir):

Systematically extract zip file $filePath into $destDir using 7zip. If 7zip executable 7za.exe is not found in $downloadDir, it is downloaded using function Download-7zip.

Extract-Zip($filePath, $destDir):

Extract zip file into $destDir only if $destDir does not exist.

Frequently Asked Questions

Installing add-on from a Windows command line terminal

This can be using the following syntax:

@powershell -ExecutionPolicy Unrestricted "iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-ninja.ps1'))"

Addressing “The underlying connection was closed” error

PS C:\Users\dashboard> iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/scikit-build/scikit-ci-addons/master/windows/install-python.ps1'))

Error: 0
Description: The underlying connection was closed: An unexpected error occurred on a receive.

As explained the chololatey documentation, this most likely happens because the build script is attempting to download from a server that needs to use TLS 1.1 or TLS 1.2 and has restricted the use of TLS 1.0 and SSL v3.

The first things to try is to use the following snippet replacing https://file/to/download with the appropriate value:

$securityProtocolSettingsOriginal = [System.Net.ServicePointManager]::SecurityProtocol

try {
    # Set TLS 1.2 (3072), then TLS 1.1 (768), then TLS 1.0 (192), finally SSL 3.0 (48)
    # Use integers because the enumeration values for TLS 1.2 and TLS 1.1 won't
    # exist in .NET 4.0, even though they are addressable if .NET 4.5+ is
    # installed (.NET 4.5 is an in-place upgrade).
    [System.Net.ServicePointManager]::SecurityProtocol = 3072 -bor 768 -bor 192 -bor 48
} catch {
    Write-Warning 'Unable to set PowerShell to use TLS 1.2 and TLS 1.1 due to old .NET Framework installed. If you see underlying connection closed or trust errors, you may need to upgrade to .NET Framework 4.5 and PowerShell v3'
}

iex ((new-object net.webclient).DownloadString('https://file/to/download'))

[System.Net.ServicePointManager]::SecurityProtocol = $securityProtocolSettingsOriginal

If that does not address the problem, you should update the version of .NET installed and install a newer version of PowerShell: