Mirror of forum.nim-lang.org

5763 :: Thorough CI Travis and Azure Pipelines scripts

• mratsim (orginal) [2020-01-02T00:29:45+01:00] view original

  While working on Weave and Synthesis I thought for over a day with my continuous integration. After a couple of weeks I have now settled on the following Travis and Azure pipelines script.

  Note that Weave only works on devel but the scripts support CHANNEL=stable.

  Highlights:

  Both scripts can cache Nim-stable and nim-devel which would require over 10 mins of build time otherwise.

  On Travis, Linux and MacOS on x86_64 and ARM64 are supported with the selection of GCC or Clang compiler as well.

  On Azure Pipelines, Linux (32 or 64-bit), Windows (32 or 64-bit) or MacOS are supported with C or C++ backend.

  The Azure pipelines scripts have a separate powershell and a bash path as on Windows, nimble (Nim findExe to be accurate) has trouble with $PATH under a bash shell. It also neatly separates the mingw building, Nim building, nimble dependencies from the actual testing of the binary (passing environment variables in-between tasks is very tricky and had numerous bugs in Azure).

  The caching part is inspired by oprypin's devel caching at https://github.com/nim-lang/Nim/wiki/BuildServices#devel-branches but with fixes I will update the wiki as well.

  Without further ado:

  Travis

  
  # Travis config for Synthesis
  language: c
  
  cache:
    directories:
      - nim-devel
      - nim-stable
  
  matrix:
    include:
      # Weave only works with Nim devel
      # Build and test using both gcc and clang
      # Build and test on both x86-64 and ARM64
      - os: linux
        arch: amd64
        env:
          - ARCH=amd64
          - CHANNEL=devel
        compiler: gcc
      
      - os: linux
        arch: arm64
        env:
          - ARCH=arm64
          - CHANNEL=devel
        compiler: gcc
      
      - os: linux
        arch: amd64
        env:
          - ARCH=amd64
          - CHANNEL=devel
        compiler: clang
      
      # On OSX we only test against clang (gcc is mapped to clang by default)
      - os: osx
        arch: amd64
        env:
          - ARCH=amd64
          - CHANNEL=devel
        compiler: clang
    fast_finish: true
  
  before_install:
    - |
      if [ "${CHANNEL}" = stable ]; then
        BRANCH="v$(curl https://nim-lang.org/channels/stable)"
      else
        BRANCH="${CHANNEL}"
      fi
  install:
    # Detect caching of Nim compiler
    - |
      if [ ! -x "nim-${CHANNEL}/bin/nim" ]; then
        git clone -b "${BRANCH}" https://github.com/nim-lang/nim "nim-${CHANNEL}/"
        pushd "nim-${CHANNEL}"
        git clone --depth 1 https://github.com/nim-lang/csources csources/
        pushd csources
        sh build.sh
        popd
        rm -rf csources
        bin/nim c koch
        ./koch boot -d:release
        ./koch tools
      else
        pushd "nim-${CHANNEL}"
        git fetch origin "${BRANCH}"
        if [ git merge FETCH_HEAD | grep -c "Already up to date." -ne 1 ]; then
          bin/nim c koch
          ./koch boot -d:release
          ./koch tools
        fi
      fi
      popd
  before_script:
      - export PATH="$PWD/nim-${CHANNEL}/bin${PATH:+:$PATH}"
  script:
      - nimble refresh
      - nimble install cligen synthesis
      - nimble test
  branches:
    except:
      - gh-pages
  Run

  Azure Pipelines

  
  
  strategy:
    maxParallel: 10
    matrix:
      # Nim requires enforcing ARCH="x86" and ucpu
      # for 32-bit targets as it seems like Azure machines are 64-bit
      Windows_devel_32bit:
        VM: 'windows-latest'
        ARCH: x86
        ucpu: i686
        PLATFORM: x86
        CHANNEL: devel
        WEAVE_TEST_LANG: c
      Windows_devel_64bit:
        VM: 'windows-latest'
        PLATFORM: x64
        CHANNEL: devel
        WEAVE_TEST_LANG: c
      Windows_cpp_devel_64bit:
        VM: 'windows-latest'
        PLATFORM: x64
        CHANNEL: devel
        WEAVE_TEST_LANG: cpp
      Linux_devel_64bit:
        VM: 'ubuntu-16.04'
        PLATFORM: x64
        CHANNEL: devel
        WEAVE_TEST_LANG: c
      Linux_cpp_devel_64bit:
        VM: 'ubuntu-16.04'
        PLATFORM: x64
        CHANNEL: devel
        WEAVE_TEST_LANG: cpp
      Linux_devel_32bit:
        VM: 'ubuntu-16.04'
        PLATFORM: x86
        ucpu: i686
        CHANNEL: devel
        WEAVE_TEST_LANG: c
      MacOS_devel_64bit:
        VM: 'macOS-10.14'
        PLATFORM: x64
        CHANNEL: devel
        WEAVE_TEST_LANG: c
  
  pool:
    vmImage: $(VM)
  
  steps:
    - task: CacheBeta@1
      displayName: 'cache Nim binaries'
      inputs:
        key: NimBinaries | $(Agent.OS) | $(CHANNEL) | $(PLATFORM)
        path: NimBinaries
    
    - task: CacheBeta@1
      displayName: 'cache MinGW-w64'
      inputs:
        key: mingwCache | 8_1_0 | $(PLATFORM)
        path: mingwCache
      condition: eq(variables['Agent.OS'], 'Windows_NT')
    
    - powershell: |
        Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' -Name 'LongPathsEnabled' -Value 1
      displayName: 'long path support'
      condition: eq(variables['Agent.OS'], 'Windows_NT')
    - bash: |
        echo "PATH=${PATH}"
        set -e
        echo "Installing MinGW-w64"
        if [[ $PLATFORM == "x86" ]]; then
          MINGW_FILE="i686-8.1.0-release-posix-dwarf-rt_v6-rev0.7z"
          MINGW_URL="https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/8.1.0/threads-posix/dwarf/${MINGW_FILE}"
          MINGW_DIR="mingw32"
        else
          MINGW_FILE="x86_64-8.1.0-release-posix-seh-rt_v6-rev0.7z"
          MINGW_URL="https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/8.1.0/threads-posix/seh/${MINGW_FILE}"
          MINGW_DIR="mingw64"
        fi
        mkdir -p mingwCache
        pushd mingwCache
        if [[ ! -e "$MINGW_FILE" ]]; then
          rm -f *.7z
          curl -OLsS "$MINGW_URL"
        fi
        7z x -y -bd "$MINGW_FILE" >/dev/null
        mkdir -p /c/custom
        mv "$MINGW_DIR" /c/custom/
        popd
        echo "##vso[task.prependpath]/c/custom/${MINGW_DIR}/bin"
      displayName: 'Install dependencies (Windows)'
      condition: eq(variables['Agent.OS'], 'Windows_NT')
    - bash: |
        echo "PATH=${PATH}"
        export ncpu=
        case '$(Agent.OS)' in
        'Linux')
          ncpu=$(nproc)
          ;;
        'Darwin')
          ncpu=$(sysctl -n hw.ncpu)
          ;;
        'Windows_NT')
          ncpu=$NUMBER_OF_PROCESSORS
          ;;
        esac
        [[ -z "$ncpu" || $ncpu -le 0 ]] && ncpu=1
        echo "Found ${ncpu} cores"
        echo "##vso[task.setvariable variable=ncpu;]$ncpu"
      displayName: 'Detecting number of cores'
    - bash: |
        echo "PATH=${PATH}"
        if [ "${CHANNEL}" = stable ]; then
          BRANCH="v$(curl https://nim-lang.org/channels/stable)"
        else
          BRANCH="${CHANNEL}"
        fi
        mkdir -p NimBinaries
        pushd NimBinaries
        if [ ! -x "nim-${CHANNEL}/bin/nim" ]; then
          git clone -b "${BRANCH}" https://github.com/nim-lang/nim "nim-${CHANNEL}/"
          pushd "nim-${CHANNEL}"
          git clone --depth 1 https://github.com/nim-lang/csources csources/
          pushd csources
          make -j $ncpu CC=gcc
          popd
          rm -rf csources
          bin/nim c koch
          ./koch boot -d:release
          ./koch tools
        else
          pushd "nim-${CHANNEL}"
          git fetch origin "${BRANCH}"
          if [ git merge FETCH_HEAD | grep -c "Already up to date." -ne 1 ]; then
            bin/nim c koch
            ./koch boot -d:release
            ./koch tools
          fi
        fi
        popd # exit nim-CHANNEL
        popd # exit NimBinaries
      displayName: 'Building Nim'
    # Nimble uses findExe which is broken under bash windows
    # We need to set PATH in the collector for the next tast
    # and also update it within this task with export
    - bash: |
        echo "##vso[task.prependpath]$PWD/NimBinaries/nim-${CHANNEL}/bin"
      displayName: 'Set env variable (Posix)'
      condition: ne(variables['Agent.OS'], 'Windows_NT')
    - bash: |
        echo "PATH=${PATH}"
        nimble refresh
        nimble install cligen synthesis
      displayName: 'Building the package dependencies (Posix)'
      condition: ne(variables['Agent.OS'], 'Windows_NT')
    - powershell: |
        echo "##vso[task.prependpath]$pwd\NimBinaries\nim-$(CHANNEL)\bin"
      displayName: 'Set env variable (Windows)'
      condition: eq(variables['Agent.OS'], 'Windows_NT')
    - powershell: |
        echo $Env:Path
        nimble refresh
        nimble install cligen synthesis
      displayName: 'Building the package dependencies (Windows)'
      condition: eq(variables['Agent.OS'], 'Windows_NT')
    - bash: |
        echo "PATH=${PATH}"
        nimble test
      displayName: 'Testing the package'
  Run

---

• Araq (orginal) [2020-01-02T11:10:02+01:00] view original

  The Azure pipelines scripts have a separate powershell and a bash path as on Windows, nimble (Nim findExe to be accurate) has trouble with $PATH under a bash shell.

  Let me guess, Bash refuses to use ';' as the path separator even though it's standard on Windows and uses ':' instead.

---

• mratsim (orginal) [2020-01-02T12:37:40+01:00] view original

  No idea. Feel free to dive in into this PR: https://github.com/mratsim/weave/pull/59 There are 30 commits to add 180 lines of code for windows support. About 7 actually contain real content and more than 20 are just there to try to appease Azure and/or nimble. I certainly don't want to deal with that again this year :P (year 2020).

---

• timothee (orginal) [2021-03-12T19:31:54+01:00] view original

  Let me guess, Bash refuses to use ';' as the path separator even though it's standard on Windows and uses ':' instead.

  see: findExe and getAppFilename are full of bugs · Issue #14292 · nim-lang/Nim

  (sorry to resurrect this, but i was searching for a solution to https://github.com/nim-lang/Nim/issues/17325, nim CI broken for 32 bit linux, i found this thread but unfortunately it doesn't seem applicable)

---