Getting started

Terraform examples are available for different use-cases for example multiple runners, ephemeral runners, and windows. For more details see the examples.

The module supports two main scenarios for creating runners. Repository level runners will be dedicated to only one repository, no other repository can use the runner. At the organization level you can use the runner(s) for all repositories within the organization. See GitHub self-hosted runner instructions for more information. Before starting the deployment you have to choose one option.

The setup guide below is a generic direction. There are many choices you can make, and there is no right way. For example we deploy ephemeral runners for both Linux and WIndows with packer pre-built AMI's that are automatically updated. Deployment is done with GitHub actions, Terragrunt and terraform. The lambda's we sync to AWS S3. For the major fleet we have a tiny pool to let start jobs quickly.

Required tools

The following tools are a minimum requirement. We advise to deploy the stack via a CI/CD pipeline.

Terraform
Bash shell or compatible
Docker (optional, to build lambdas without node).
AWS cli (optional)
Node and yarn to build the Lambda's (or download via Release).

Setup guide

The setup consists of running Terraform to create all AWS resources and manually configuring the GitHub App. The Terraform module requires configuration from the GitHub App and the GitHub app requires output from Terraform. Therefore you first create the GitHub App and configure the basics, then run Terraform, and afterwards finalize the configuration of the GitHub App.

Setup GitHub App (part 1)

Go to GitHub and create a new app. Be aware you can create apps for your organization or for a user. For now we only support organization level apps.

Create an app in Github
Choose a name
Choose a website (mandatory, not required for the module).
Disable the webhook for now (we will configure this later or create an alternative webhook).
Permissions for all runners:
- Repository:
- Actions: Read-only (check for queued jobs)
- Checks: Read-only (receive events for new builds)
- Metadata: Read-only (default/required)
Permissions for repo level runners only:
Repository:
- Administration: Read & write (to register runner)
Permissions for organization level runners only:
Organization
- Self-hosted runners: Read & write (to register runner)
Save the new app.
On the General page, make a note of the "App ID" and "Client ID" parameters.
Generate a new private key and save the app.private-key.pem file.

Setup terraform module

Download lambdas

To apply the terraform module, the compiled lambdas (.zip files) need to be available either locally or in an S3 bucket. They can either be downloaded from the GitHub release page or built locally.

To read the files from S3, set the lambda_s3_bucket variable and the specific object key for each lambda.

The lambdas can be downloaded manually from the release page or using the download-lambda terraform module (requires curl to be installed on your machine). In the download-lambda directory, run terraform init && terraform apply. The lambdas will be saved to the same directory.

For local development you can build all the lambdas at once using .ci/build.sh or individually using yarn dist.

Service-linked role

To create spot instances the AWSServiceRoleForEC2Spot role needs to be added to your account. You can do that manually by following the AWS docs. To use terraform for creating the role, either add the following resource or let the module manage the service linked role by setting create_service_linked_role_spot to true. Be aware this is an account global role, so maybe you don't want to manage it via a specific deployment.

resource "aws_iam_service_linked_role" "spot" {
  aws_service_name = "spot.amazonaws.com"
}

Terraform module

Next create a second terraform workspace and initiate the module, or adapt one of the examples.

Note that github_app.key_base64 needs to be a base64-encoded string of the .pem file i.e. the output of base64 app.private-key.pem. The decoded string can either be a multiline value or a single line value with new lines represented with literal \n characters.

module "github-runner" {
  source  = "philips-labs/github-runner/aws"
  version = "REPLACE_WITH_VERSION"

  aws_region = "eu-west-1"
  vpc_id     = "vpc-123"
  subnet_ids = ["subnet-123", "subnet-456"]

  prefix = "gh-ci"

  github_app = {
    key_base64     = "base64string"
    id             = "1"
    webhook_secret = "webhook_secret"
  }

  webhook_lambda_zip                = "lambdas-download/webhook.zip"
  runner_binaries_syncer_lambda_zip = "lambdas-download/runner-binaries-syncer.zip"
  runners_lambda_zip                = "lambdas-download/runners.zip"
  enable_organization_runners = true
}

Run terraform by using the following commands

terraform init
terraform apply

The terraform output displays the API gateway url (endpoint) and secret, which you need in the next step.

The lambda for syncing the GitHub distribution to S3 is triggered via CloudWatch (by default once per hour). After deployment the function is triggered via S3 to ensure the distribution is cached.

Setup the webhook / GitHub App (part 2)

At this point you have two options. Either create a separate webhook (enterprise, org, or repo), or create a webhook in the App.

Option 1: Webhook

Create a new webhook at the repo level for repo level runners, or org (or enterprise level) for org level runners.
Provide the webhook url, which should be part of the output of terraform.
Provide the webhook secret (terraform output -raw <NAME_OUTPUT_VAR>).
Ensure the content type is application/json.
In the "Permissions & Events" section and then "Subscribe to Events" subsection, check either "Workflow Job" or "Check Run" (choose only one option!!!).
In the "Install App" section, install the App in your organization, either in all or in selected repositories.

Option 2: App

Go back to the GitHub App and update the following settings.

Enable the webhook.
Provide the webhook url, should be part of the output of terraform.
Provide the webhook secret (terraform output -raw <NAME_OUTPUT_VAR>).
In the "Permissions & Events" section and then "Subscribe to Events" subsection, check either "Workflow Job" or "Check Run" (choose only one option!!!).

Install GitHub app

Finally you need to ensure the app is installed to all or selected repositories.

Go back to the GitHub App and update the following settings.

In the "Install App" section, install the App in your organization, either in all or in selected repositories.

Debugging

In case the setup does not work as intended follow the trace of events:

In the GitHub App configuration, the Advanced page displays all webhook events that were sent.
In AWS CloudWatch, every lambda has a log group. Look at the logs of the webhook and scale-up lambdas.
In AWS SQS you can see messages available or in flight.
Once an EC2 instance is running, you can connect to it in the EC2 user interface using Session Manager (use enable_ssm_on_runners = true). Check the user data script using cat /var/log/user-data.log. By default several log files of the instances are streamed to AWS CloudWatch, look for a log group named <environment>/runners. In the log group you should see at least the log streams for the user data installation and runner agent.
Registered instances should show up in the Settings - Actions page of the repository or organization (depending on the installation mode).