⚙️ Monorepo scripts strategies & naming conventions2019-01-29

⚙️ Monorepo scripts strategies & naming conventions

The vast majority of everyday monorepo management tasks boil down to running package scripts in a certain order and under certain conditions, both locally and on CI. This article explains one of many possible approaches. This approach worked fairly well in some of RingCentral’s monorepo projects. The size of projects varies: from less than a hundred files and a handful of devs up to thousands of files and 50+ developers.

In this article, I will use Lerna as a tool that helps to manage such mono repositories. It allows you to run scripts and execute code in packages, as well as takes care of some publishing activities. You may read more about Lerna concepts here: https://github.com/lerna/lerna#concepts.

Main ideas

  • Scripts should eliminate the possibility of error in critical tasks like test coverage collection and publishing
  • Default scripts (like install, test and publish) should be no-brainers: do all the things automatically with the least astonishment
  • Full control mode when needed: ability to run scripts granularly or in a quicker fashion

Phases

Overall, any interaction with a repository, both locally and on the CI, can be presented as scenarios that can be split into some granular phases (or stages in Gitlab terminology). Scenarios may skip phases or have some phases substituted with other different phases. Scenarios usually have certain goals and expectations — here are the most common ones:

  • A developer checks out the repo and runs npm install — the goal is to have a fully functional and ready-to-use dev setup at this point, the expectation is that right after this command the developer may run other commands with no additional actions related to getting things ready to work
  • A developer runs the npm start script — the goal is to run start in all packages, the expectation is that it just starts working, with no need to pre-build anything
  • CI determines that a tag has been pushed, checks out the repo, runs linter, runs build scripts, runs tests, and collects coverage, and if all is good — publishes to NPM. The goal is the delivery of high-quality code. The expectation is that if any phase breaks it stops the process and there is zero possibility of any unwanted publishes of incomplete/broken code

Let’s dive deeper into those phases.

Install

In this phase, CI or the developer sets up the environment to run other scripts.

  • CI should run npm install --ignore-scripts so that we can control when and how to do bootstrapping in the CI environment, for example, the lint task does not require packages to be bootstrapped, but the test task does (more about this in the next section)
  • Developers should use regular npm install, which must run full bootstrap

Lint

This is the cheapest check that makes sure the codebase is in good condition.

  • CI should run lint right after install as the first cheap check
  • CI must stop all further expensive checks if the cheap lint check failed

Packages bootstrapping

In this phase, dependencies of underlying packages are installed — this is a post install on the dev machine or a post lint phase in the CI environment, but it depends on the setup.

  • In general the postinstall script should run lerna bootstrap --hoist --no-ci so that devs will get a fully working repo right after running npm install, the -no-ci flag is needed due to a bug https://github.com/lerna/lerna/issues/1324
  • CI may not require demos to be built/tested and thus neither to be bootstrapped unless you use demos for e2e tests. In this case, we should bootstrap only the truly used packages by scoping the bootstrap script: npm run bootstrap -- --scope=@ringcentral/* (we usually have libraries in this scope). This is a nice optimization that saves CI time.

Building

A phase needed to obtain an artifact, something that later will be published. Also, it may be a prerequisite for tests.

  • The build script on package level should run the clean script beforehand to make sure no previous artifacts exist on subsequent builds
  • The build:quick can be used to build all libs that will be used in demos

Tests

A phase that does the majority of code quality verification and makes sure things work as expected, not just written as expected (linter phase).

Sometimes code has to be built before running tests. If there are multiple libs in monorepo which depend on each other it is safer to run tests once all dependencies are built instead of using magic to run tests using only sources.

  • CI runs tests by calling the test script, which is a maximum set of all tests with e2e tests and coverage collection
  • Locally devs may run test:quick ****which does not collect coverage or does not run e2e tests
  • CI should also run test:coverage script to upload coverages after test
  • Optionally there could also be a test:watch script which will run quick tests in watch mode

Committing

This is a phase that occurs on dev machines when developers commit and push the code, some minor quick checks should happen on this phase.

  • Obviously, we don’t want to make CI run potentially invalid code so we can use test:quick and lint:staged (or lint:quick)scripts in pre-commit hooks

Publishing

This delivers artifacts to some package management system that distributes them to end users. Code must be built and tested before publishing, which is CI responsibility.

  • Publish scripts actually do the publishing (usually on CI)
  • Canary means a pre-release (something you can run nightly)
  • Release means a regular versioned release
  • CI may add -yes flag to publish:* scripts: it’s better to add them here in order to prevent unwanted local publishes

Alternatively, on publish, CI can skip regular flow (build, test, coverage) and only run publish in packages, which should do build, test, coverage to make sure that if publish is run locally from dev machine it still goes through all phases, but we recommend to always publish from CI only as it is the only way to ensure all proper checks.

Starting/watching

Developer-specific phase, day to day activities like website or library/demo/storybook development, on this phase codebase is constantly watched for changes and re-built once changes occur.

Assume the following setup:

packages/
- demo (depends on lib1 and lib2)
- lib1
- lib2 (depends on lib1)
  • Demos that use libraries often need those libs to be pre-built in order to run start correctly without errors about missing files so root’s start script must run build script scoped for libs and then run start:quick script (see examples below), unfortunately, this brings overhead until https://github.com/webpack/webpack/issues/4991 is fixed (previously TS was prone too: bug 12996, see update below)
  • start:quick simply runs start in all packages which starts Webpack and Babel watchers
  • UPDATE In TypeScript 3.4 new incremental option has been introduced: it will produce a build and a cache, so no matter how often you restart watchers it will be ready much faster. Unfortunately it’s not yet supported by TS Loader for Webpack It still does not eliminate the necessity to pre-build libraries.

Cleaning

Developer-specific phase before builds or when switching branches.

  • Root’s clean should first run clean scripts in packages (e.g. clean artifacts), then remove node_modules in packages, then remove node_modules in the root, this will bring repository to ground zero
  • Cleaning is useful when devs switch branches, then command npm run clean && npm install makes sure they have working setup after branch switch

Examples

Main package scripts

{
  "postinstall": "npm run bootstrap",
  "bootstrap": "lerna bootstrap --no-ci --hoist",
  "bootstrap:quick": "npm run bootstrap -- --scope=@ringcentral/*",

  "clean": "npm run clean:artifacts && npm run clean:packages && npm run clean:root",
  "clean:artifacts": "lerna run clean --parallel",
  "clean:packages": "lerna clean --yes",
  "clean:root": "rimraf node_modules",
  "start": "npm run build:quick && npm run start:quick",
  "start:quick": "dotenv lerna run start -- --parallel",
  "build": "lerna run build --concurrency=1 --stream",
  "build:quick": "npm run build -- --scope=@ringcentral/*",
  "test": "lerna run test --concurrency=1 --stream",
  "test:quick": "lerna run test:quick --concurrency=1 --stream",
  "test:coverage": "lerna run test:coverage --parallel",
  "test:watch": "lerna run test:watch --parallel",
  "publish:release": "lerna publish --force-publish=* --no-push --no-git-tag-version",
  "lint": "eslint --cache --cache-location node_modules/.cache/eslint --fix",
  "lint:all": "npm run lint 'packages/*/src/**/*.ts*'",
  "lint:staged": "lint-staged"
}

We prefer to run tests one by one in topological order (maintained by Lerna) to see the nice structured output. Technically --concurrency=1 --stream can be replaced with --parallel which disregards topology and runs everything in parallel.

Notes:

  • You can publish all packages together using -force-publish=* flag for simplicity.

Scripts test:quick and lint:staged should be part of pre-commit hook.

Package scripts

{
  "mocha": "mocha --opts mocha.opts",
  "karma": "karma start --singleRun",
  "nyc": "nyc mocha --opts mocha.opts",

  "build": "npm run clean && npm run build:tsc && npm run build:wp",
  "build:tsc": "tsc",
  "build:wp": "webpack --progress",

  "start": "npm-run-all -p watch:tsc watch:webpack",
  "start:tsc": "npm run build:tsc -- --watch --preserveWatchOutput",
  "start:webpack": "npm run build:webpack -- --watch"
}

GitLab CI config

image: node:lts

stages:
  - validation
  - bootstrap
  - test
  - publish

variables:
  BRANCH: $(echo ${CI_COMMIT_REF_NAME} | sed -e 's/\(.*\)/\L\1/' | sed -r 's/[^a-z0-9-]/-/g')

before_script:
  - npm config set //registry.npmjs.org/:_authToken=${NPM_TOKEN}
  # or
  - npm config set repository$PRIVATE_NPM
  - npm config set _auth$PRIVATE_NPM_AUTH
  - npm config set email$PRIVATE_NPM_EMAIL
  - npm install --progress=false --ignore-scripts

cache:
  key: '$CI_COMMIT_REF_NAME'
  paths:
    - node_modules/
    - packages/*/node_modules

job_lint:
  stage: validation
  script: DEBUG=eslint:cli-engine npm runlint:all

job_bootstrap:
  stage: bootstrap
  script:
    - npm runbootstrap:quick
    - npm runbuild:quick
  artifacts:
    paths: -"*/es"
      -"*/lib"
    expire_in: 1 day

job_test:
  stage: test
  script:
    - npm test
    - npm run test:coverage

job_canary:
  stage: publish
  script: npm runpublish:release-- --canary --preid=$BRANCH.$CI_PIPELINE_IID --dist-tag=$BRANCH --yes
  only:
    - master
    - feature/*
    - release/*

job_publish:
  stage: publish
  script: npm runpublish:release --$CI_COMMIT_TAG--yes
  only: -tags

This setup assumes you have NPM_TOKEN (and others) in GitLab ENV variables. Also note the --yes flags for publish scripts.

Due to https://github.com/lerna/lerna/issues/2171 we have to explicitly set the pipeline ID in the canary preid.

Travis CI config

language: node_js
node_js:
  - stable

cache:
  directories:
    - $HOME/.npm
    - node_modules
    - packages/*/node_modules

before_install:
  - BRANCH=$(echo ${TRAVIS_BRANCH} | sed -e 's/\(.*\)/\L\1/' | sed -r 's/[^a-z0-9-]/-/g')
  - npm config set //registry.npmjs.org/:_authToken=${NPM_TOKEN}before_script:
  - DEBUG=eslint:cli-engine npm runlint:all
  - npm run build:quick

deploy:
  - provider: script
    script: npm runpublish:release --$TRAVIS_TAG--yes
    skip_cleanup: true
    on:
      branch: master
    tags: false
    repo: xxx
  - provider: script
    script: npm runpublish:release-- --canary --preid=$BRANCH.$TRAVIS_JOB_NUMBER --dist-tag=$BRANCH --yes
    skip_cleanup: true
    on:
      tags: true
      repo: xxx
  - provider: releases
    api_key: $GITHUB_TOKEN
    skip_cleanup: true
    file:
      - packages/xxx/dist/xxx.js
      - packages/yyy/dist/yyy.js
    on:
      tags: true
      repo: xxx

after_success:
  - npm runtest:coverage

This setup also assumes you have NPM_TOKEN and GITHUB_TOKEN in Travis ENV variables. Also, note the --yes flags for publish scripts. Travis by default will do npm install which will do npm bootstrap, so there’s some room for optimization like we did in GitLab example.

Due to https://github.com/lerna/lerna/issues/2171 we have to explicitly set the Travis build number in the canary preid.

Packages used

Since we mention a lot of packages it does make sense to tell what they are doing.

  • Lerna — manages JavaScript projects with multiple packages
  • Dotenv-cli — loads .env files that stores environment variables
  • ESLint — a fully pluggable tool for identifying and reporting on patterns in JavaScript
  • Mocha — simple &flexible javascript test framework for Node.js & the browser
  • NYC — command line interface for Istanbul, a JS code coverage tool that computes statement, line, function and branch coverage
  • Karma — test runner for JavaScript, which is able to run Mocha and others
  • Coveralls — code coverage tracking system
  • TSC — TypeScript compiler
  • Webpack — module bundler, bundles JavaScript files for usage in a browser
  • Husky — runs NPM scripts from git hooks
  • Lint-staged — runs linters on git staged files
  • Jest — JavaScript testing framework with a focus on simplicity, can be used as replacement for Karma+Mocha+Istanbul

Conclusion

This approach with minor variations (mostly alternate phases) was successfully implemented and tested in different projects and proven to be consistent and complete. I hope it will help you to build, test and interact with your projects more efficiently.

This whole thing definitely requires some library to make things easier :)