How the Tableland team has created a release flow for the Validator to allow external collaborators to run their own nodes

Announcing Validator Releases

As part of the January 2023 Launch Week we are excited to announce that the Tableland Network Validator now has official releases!

By @Joe Wagner

The Tableland Validator is one of the core pieces that make up the Tableland Network. This post will cover how the Tableland team has created a release flow for the Validator that lays the ground work to allow external collaborators to run their own nodes and continue building the community that is the future of the Tableland Network.

💡

Releases are available via Github, Docker Hub, and NPM!

Depending on how you want to consume a release it will include the source code, a set of pre-built binaries, or docker images. The binaries are available for download on Github, and also available as a package via npmjs.org. The Docker Images are available on Docker Hub.

If you’re interested in running a Validator node, or want to share how you are running a Validator, get in touch on Discord!

Distributing with Github, Docker, and NPM

The Validator source and build are available for download on GitHub, via docker hub, and published as an NPM package. This makes starting a validator fast and painless, regardless of how you are running things.

In order to streamline the Validator publishing process, the Tableland core team has made use of Github Actions to build, publish, and notify downstream dependencies of releases.

To start digging into how everything works, let’s look at the high-level view of what happens during a Validator release. When a release is created via the Github UI, there are a few interconnected pieces to consider:

The Validator Github repository workflow that builds the binaries, docker images, and triggers downstream workflows.
The NPM Validator package workflow that triggers a pull request anytime a new Validator release is available.
Downstream packages that depend on the NPM Validator packages that make use of dependabot to ensure dependencies are up to date.

The Validator Source

The tablelandnetwork/go-tableland repository is the home for the Validator source. When features and fixes are added to the Validator that indicate a need for a version bump, a member of the Tableland Core team will tag the associated commit, and then use Github to create a Release.

The go-tableland repo has a workflow file called binaries.yml. To understand how this works lets look at the first few lines of the file

name: Generate binaries
on:
  release:
    types:
      - created
# when a release is created all jobs in this file will be run...

The on: directive tells Github Actions that when a release has been created this workflow should be run. The specifics of each job in the workflow are declared in the jobs: section. First let’s look at what the three jobs this workflow handles are. Then we can go into detail on what each job does.

jobs:
  binaries:
    # Inside this section the details of how the binaries
		# should built are declared.
  docker:
    # Inside this section the details of building the docker
		# images are declared.
  js-release:
    # Inside this section we will trigger a workflow in the
		# tablelandnetwork/js-validator repository, which contains
		# everything we need to publish to npm

The `binaries` job

This is the job responsible for building the actual binaries that are going to be distributed. In order to build the binaries for a few of the most common systems and architectures, we will make use of the very handy Github Actions Matrix Strategy. This makes building the Validator binary on multiple systems easy. Let’s take a look at that part of the workflow file.

jobs:
  binaries:
    strategy:
      matrix:
        include:
					# an array of the different operating systems we want to use, and
					# the commands needed to build and compress the resulting file.
          - os: ubuntu-latest
            asset_name: api-linux-amd64.tar.gz
            compress_cmd: tar -czvf
            build_cmd: docker run -v $PWD:/data golang:1.19 bash -c "cd /data && go build ./cmd/api"
          - os: windows-latest
            asset_name: api-windows-amd64.zip
            compress_cmd: tar.exe -a -c -f
            build_cmd: go build -o api ./cmd/api
          - os: macos-latest
            asset_name: api-darwin-amd64.tar.gz
            compress_cmd: tar -czvf
            build_cmd: go build -o api ./cmd/api
    runs-on: ${{ matrix.os }}
		# Here are the steps that will be run on each of the operating systems
    steps:
      - name: Checkout repo
        uses: actions/[email protected]
      - name: Install Go
        uses: actions/[email protected]
        with:
          go-version: v1.19.x
      - name: Build binary
        run: ${{ matrix.build_cmd }}
      - name: Pack output
        run: ${{ matrix.compress_cmd }} ${{ matrix.asset_name }} api
      - name: Upload binary
        uses: svenstaro/[email protected]
        with:
          repo_token: ${{ secrets.GITHUB_TOKEN }}
          file: ${{ matrix.asset_name }}
          asset_name: ${{ matrix.asset_name }}
          tag: ${{ github.ref_name }}
          overwrite: true

As you can see we are currently building for linux amd64, windows amd64, and mac amd64.

💡

Github doesn’t yet support mac M1 os, but when that’s available we will include that here as well. If you are developing on an M1 chip, don’t fret. Until the Github matrix strategy supports M1 chips, the Tableland team will be manually building a binary for that system.

The steps for each system are fairly straightforward.

checkout the repo
install go 1.19.x
build the binary
compress the binary
Finally upload the compressed file to the tablelandnetwork/go-tableland repository

At this point we have binaries available for download on Github

The `docker` job

The next job in the workflow we’ll look at is the creation and publishing of the Docker images to Docker Hub.

jobs:
  binaries:
		# ...
  docker:
    runs-on: ubuntu-latest
    steps:
			# Docker in combination with QEMU gives us more flexibility
			# than the Github matrix strategy. More on this below...
      - name: Set up QEMU
        uses: docker/[email protected]
      - name: Set up Docker Buildx
        uses: docker/[email protected]
			# Login to Docker Hub using repository level secrets
      - name: Login to Docker Hub
        uses: docker/[email protected]
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
			# Docker makes building and pushing to Docker Hub simple
      - name: Build and push
        uses: docker/[email protected]
        with:
          file: ./cmd/api/Dockerfile
          push: true
          tags: textile/tableland:latest,textile/tableland:${{ github.ref_name }}
          platforms: linux/amd64, linux/arm64

One difference to note here is that instead of using the Matrix Strategy, we are taking advantage of Docker in combination with QEMU.

💡

If you aren’t familiar with QEMU, definitely check it out!

Since Docker Images let us run on Linux, Windows, and Mac. All we need to do is build an image for amd64 and arm64, and then we have support for quite a few os+arch combinations. The way that we are able to get the arm64 images is by using QEMU and the convenient docker/setup-qemu-action. I won’t go into much detail, but QEMU is capable of emulating a complete machine fully in software regardless of the hardware in use.

The `js-release` job

The last job to talk about in this workflow is the js-release job. There are a few interesting things to point out about this job. It is responsible for triggering a workflow in a different repository, and because it relies on the output of the binaries job, it must run after the binaries job is finished.

Let’s look at the relevant parts of the workflow file.

jobs:
  binaries:
		# ...
  docker:
		# ...
  js-release:
    runs-on: ubuntu-latest
		# by using the `if` and `needs` directives we ensure that this job
		# only runs after the binaries job has successfully finished
    if: ${{ success() }}
    needs: [binaries]
    steps:
      - run: echo "validator_version ${{ github.ref_name }}"

      - name: PR to publish this release via the npm package
        uses: actions/[email protected]
        with:
          github-token: ${{ secrets.TEXTILEIO_MACHINE_ACCESS_TOKEN }}
          script: |
            try {
              const ownerOrg = 'tablelandnetwork';

              // if the tag/release has a preceeding "v" we want to remove
              // it and match standard symantics in the js ecosystem
              let version = '${{ github.ref_name }}';
              if (/^v[0-9]/.test(version)) {
                version = version.slice(1);
              }

              const options = {
                owner: ownerOrg,
                repo: 'js-validator',
                workflow_id: 'validator-update-pr.yml',
                ref: 'main',
                inputs: {
                  validator_version: version
                }
              };

              console.log(options);
							// At this point we trigger the `validator-update-pr.yml` workflow
							// in the tablelandnetwork/js-validator repository
              const response = await github.rest.actions.createWorkflowDispatch(options);

              if (response.status !== 204) {
                core.setFailed(`create workflow_dispatch received status code ${response.status}`);
              }
            } catch(err) {
              console.log(err);
              core.setFailed(err.message);
            }

After taking a closer look at how this works, you’ll notice that it’s leveraging the powerful actions/github-script which lets you use the Github API in a JS like environment. If you want to dig into what you can do with this action have a look at the Octokit REST SDK.

In this case we are using it to ensure that the release version doesn’t start with a “v”, and then to trigger a Workflow in the tablelandnetwork/js-validator repository. Something to note in regard to triggering workflows. Since the ability to trigger a workflow from a workflow introduces the possibility to create an endless recursive loop of triggering workflows, we can’t authenticate with the familiar secrets.GITHUB_TOKEN to trigger the next workflow. In its place we need to create an access token that both is allowed to trigger workflows and has access to both of the repositories involved. In this case the Tableland and Textile team has setup a one of these tokens and named it TEXTILEIO_MACHINE_ACCESS_TOKEN. There are a few ways to configure this kind of token, but for a good primer checkout the GitHub docs on personal access tokens.

The NPM Validator Package

To recap our journey so far through a Validator release, we have built assets and made them available to download from Github or Docker Hub. The next piece we want to look at is how we can get the binaries available on npm, and make it easy for folks in the node.js ecosystem to use these binaries.

This is where the tablelandnetwork/js-validator repository comes into play. It is the home to the npmjs package that is published to npm for each Validator release. This package is relatively small, and really only has a few parts.

A script that downloads the binaries from Github, unpacks them, and puts them in the bin/ directory.
A bin/ directory that holds the available binaries.
A workflow file that runs the above script and creates a pull request with those changes.

At the end of the binaries.yml workflow above we saw that the validator-update-pr.yml was triggered. Let’s take a look at what that file does.

To start out you’ll notice on the first few lines the on: directive tells Github when this workflow should run.

name: Validator Update PR
on:
  workflow_dispatch:
    inputs:
      validator_version:
        description: "Validator version"
        required: true

The main thing to note here is the only input is validator_version, and that corresponds to the Validator release, minus the leading “v”.

Next you’ll see that there’s only one job in this workflow.

jobs:
  validator-update-pr:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎️
        uses: actions/[email protected]

      - name: Cache 📦
        uses: actions/[email protected]
        with:
          path: ~/.npm # npm cache files are stored in `~/.npm` on Linux/macOS
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-

      - name: Setup Node Environment ⬢
        uses: actions/[email protected]
        with:
          node-version: 18

      - name: Update validatorVersion in package.json
        id: validator-version-bump
        uses: jaywcjlove/[email protected]
        with:
          data: |
            {
              "validatorVersion": "${{ github.event.inputs.validator_version }}"
            }

      - run: echo "validator version - ${{ steps.info.outputs.version }} - ${{ github.event.inputs.validator_version }}"

      - name: Install 🔧
        run: npm install

      - name: Build 🛠
        run: npm run build

      - name: Commit files from the build
        run: |
          git config --local user.name  ${{ github.actor }}
          npm run prettier:fix
          git add --all
          git commit -m "Validator version update for ${{ github.event.inputs.validator_version }}"

      - name: Push changes
        uses: ad-m/[email protected]
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          branch: validator-update-pr-${{ github.event.inputs.validator_version }}

      - uses: actions/[email protected]
        id: create-validator-update-pr
        name: Create Validator Update PR
        with:
          # NOTE: This uses ${{secrets.TEXTILEIO_MACHINE_ACCESS_TOKEN}}
          #       so that the workflow can create a pull request.
          github-token: ${{secrets.TEXTILEIO_MACHINE_ACCESS_TOKEN}}
          script: |
            try {
              const version = '${{ github.event.inputs.validator_version }}';
              const branchName = 'validator-update-pr-' + version;
              const baseBranch = 'main';
              const prBody = '✅ This PR was created by the Validator Update PR action for version ' + version;

              // Create a pull request
              await github.rest.pulls.create({
                owner: context.repo.owner,
                repo: context.repo.repo,
                title: 'Validator Update PR ' + version,
                head: branchName,
                base: baseBranch,
                body: prBody
              });
            } catch (err) {
              console.log(err);
              core.setFailed('Failed to create release branch: ' + err.message);
              return;
            }

The validator-update-pr: job has 10 steps. That’s kind of a lot so let’s look at what each one does.

The first three steps are the somewhat standard checkout the repo, use cache files if possible, and install node.js.

The fourth step is the first interesting piece. In this step we update a custom field in the package.json file. The field is called validatorVersion and this tracks the name of the validator release. When the build step downloads the release it will use this value to ensure it gets the right version.

Next up we run npm install and then run the build script. After these steps are done we can commit the updates, push the commit to a new branch, and create a pull request to merge the new release into the main branch.

When this pull request is created anyone subscribed to the repo will get a notification. One of the members of the Tableland team can then do a manual inspection of the proposed changes, merge the pull request, and publish the changes to npm in the @tableland/validator package.

Downstream Dependents

The last piece to mention in this release flow is how other Tableland npm packages that depend on the @tableland/validator package manage this dependency.

As an example lets look at the @tableland/local package. The repository for this package is found at tablelandnetwork/local-tableland. This repository uses the awesome builtin Github tool, Dependabot. Once Dependabot has been setup it runs in the background and automatically creates pull requests anytime a dependency has a new version available. Since @tableland/validtor is a normal npm dependency, Dependabot handles creating pull requests. The Tableland team just needs to look at the pull request decide if the version update should be merged.

Conclusion

At this point you should have a good understanding of how Validator releases propagate through the community and make a Tableland Validator a dependency of your project. Thanks for reading!

Please connect with us on Discord https://discord.com/invite/dc8EBEhGbg, give us a star us on Github https://github.com/tablelandnetwork/, and follow us on Twitter https://twitter.com/tableland__!

← Blog Home

Announcing Validator Releases
Distributing with Github, Docker, and NPM
The Validator Source
The binaries job
The docker job
The js-release job
The NPM Validator Package
Downstream Dependents
Conclusion

Announcing Validator Releases

Announcing Validator Releases

Distributing with Github, Docker, and NPM

The Validator Source

The binaries job

The docker job

The js-release job

The NPM Validator Package

Downstream Dependents

Conclusion

The `binaries` job

The `docker` job

The `js-release` job