MEP 3 - CI / CD

Overview

This proposal applies to core and web.

Both core and web should have a master and a staging branch.

The tip of staging should be deployed to the staging environment.

The tip of master should be deployed to the production and demo environments.

Thus, all PRs should have staging as the merge target.

staging should be merged into master at least once a week, this is called a release.

If a change is required to be directly to master, this is called a hotfix.

Commits directly to master and staging without a pull request will be blocked via branch protections.

Upon merging into master, a backmerge can be created to keep staging up to date.

All PRs to staging should use squash commits.

All PRs to master should use merge commits. Backmerges should use merge commits.

A changelog is automatically generated on master excluding all PRs with label release or backmerge.

A release should automatically tag the version of master with <YEAR>.<MONTH>.<DAY>.<SHORT_HASH>.

This tag should propagate through to Sentry and ECR.

Tagging images in ECR allows for quick rollbacks if necessary.

A release is a PR from staging to master and should be automatically created for review on Wednesday evenings.

A release is automatically labeled with "release".

A release has auto merge enabled, meaning once someone approves it, it will be merged with the correct merge strategy automatically.

In emergencies, a PR can be made directly against master. This should automatically trigger a release.

Ensure master is back-merged to staging to prevent divergent heads.

These should be labeled with "hotfix".

After either a hotfix or a release PR is merged to master, an autogenerated PR should be created to back merge master to staging.

Backmerge PRs are, by script, auto-approved by Rodda and auto-merge enabled.

These PRs should be labeled with "backmerge".

As the system of record for equipment sale and rental businesses, Moab is mission critical and cannot afford down time - especially during business hours.
Frequently deploying the master branch to production without rigorous automated and manual testing significantly increases the likelihood of causing an outage. Shepherding changes through a staging environment for QA can help catch bugs before they land in production.

There are three environments:
- prod: used by customers
- demo: used by sales (and potentially customers e.g. during an RFP process)
- staging: used by internal teams for testing
SLAs
- prod and demo should be "up" and "stable"
- staging has no guarantees, but should aim for stability and uptime such that it's usable for testing
The tip of the master branch is what is deployed to prod and demo
demo is a separate tier to allow for:
- intensive migration runs on its database
- integration with testing SaaS account, e.g. Stripe Connect
The tip of the staging branch is what is deployed to staging
All feature branch pull requests should select staging as their merge target
staging shall be merged into master to create a "release"
- "releases" should be tagged following a <YEAR>.<MONTH>.<DAY>.<SHORT-HASH> versioning scheme
Hotfixes can be applied via merges directly to master if prod experiences downtime or a severe bug
- Once the the hotfix has been deployed, master should be back-merged to staging

Commits to master and staging without a pull request will be blocked via branch protections
Pull requests to master must be auto-tagged with a release version label
- The current build version should flow through to any relevant other monitors, e.g. Sentry
Release pull requests are opened automatically by a cron job for review once a week on Wednesday evening
The build image should be tagged in ECR allowing for trivial rollbacks