GitHub runners for a few cents

Scott Guymer / Niek Palm

Niek Palm

Philips | Principal Engineer

@niekos77

040code.github.io

Travel | Home Cook | Triathlon

Scott Guymer

Philips | Principal Engineer

@mr_scottguymer

www.scottguymer.co.uk

How do you picture Philips?

Probably this?

Maybe this?

Not this

Philips is a health technology company improving people's health and well-being through meaningful innovation

Our purpose is to improve people’s health and well-being. We aim to improve 2.5 billion lives per year by 2030

Software in Philips

• Global Organisation

• Embedded | Cloud | Web | Mobile

• 6500+ Software Professionals

• 100s Millions lines of code

• Regulated Medical Software

InnerSource Journey

InnerSource is a development methodology where engineers build proprietary software using best practices from large-scale open source projects.

How we started?

March 2020

InnerSource the new default

Driven by GitHub

Empower everyone with CI/CD

101 - GitHub Actions

• Actions == GitHub CI/CD ++
• Actions == CI/CD Lego bricks
• Jobs are triggered by an event
• Jobs require a runner to run

on: [push]
jobs:
  check-bats-version:
    runs-on: [self-hosted]
    container: node:16
    steps:
      - uses: actions/checkout@v3
      - run: npx bats -v

Why do we need

self hosted GitHub runners?

Connectivity

Costs

Security

Requirements

• Run on standard Linux VMs
• Option to tailor (OS / Arch)
• Scale up / down / zero
• Connect to private services
• Only pay for what's used

Event based

Scale based on workflow jobs

Serverless

low cost / low maintenance control plane

Treat as Cattle

Secure and no fire fighting

Networking

Bring your own connection

Cloud Architecture

• GitHub App for events
• AWS API gateway to get events
• AWS Lambda for event handling
• AWS SQS for decoupling
• AWS Lambda to scale up
• GitHub App for API access
• AWS EC2 (Spot) to run jobs
• AWS Direct connect for networking
• AWS Lambda for scaling down

DEMO

• Create cloud resources
• Connect cloud with GitHub
• Run 40 jobs

Security

• Ephemeral
• Encrypted secrets
• Minimal privileges
• Permission boundaries

Open Source

900+ stars

75+ contributors

330+ Pull requests

Recommended by GitHub

Contribution

• Support windows
• Support ARM
• Support GHES
• Better docs
• Security improvements
• Upgrades

Running at Scale

in Philips

Deployment

• Deploy runners with the runners
• Terragrunt to keep our Terraform dry
• Connect to Philips with AWS Direct Connect
• Network rules controlled via PR and CODEOWNERS

Self service via IssueOps

Scaling in and out

10K instances on a average day

Problems

CI DOS

Rate Limits

Costs

How much?

30 to 40 cents per developer per month

Questions

# Resources

resource "website" "github" {
  url = "github.com/philips-labs/terraform-aws-github-runner"
}

resource "website" "slides" {
  url = "github.com/philips-labs/2022-07-07_scaling-github-runners"
}

resource "contact" "niek" {
  github   = "@npalm"
  linkedin = "in/niekpalm/"
  twitter  = "@niekos77"
}

resource "contact" "scott" {
  github   = "@ScottGuyme"
  linkedin = "in/scottguymer/"
  twitter  = "@mr_scottguymer"
}