What are Github Actions?

Up until recently I hadn’t had a ton of opportunities to really dig my fingers into all that Github had to offer. I had only been hosting some of my random and test repos there. Essentially just a remote location to push | pull to/from. My general workflow would often be some form of development in a dev branch, maybe opening up a pull request and eventuanlly merging my changes into main. The “basics”. Any downstream CI/CD steps (if what I was working on required any) would often be handled outside of Github. For instance, I used to have an entire Jenkins server setup to handle testing and linting of a repo. Although not THAT much of a hassle, working with Jenkins did add additional server maintenance steps that I often didn’t have time for (read: I just plain didn’t want to do that). 😭

So, when I began to hear about what was possible with Github Actions my interest was peaked. I had seen the “Actions” tab in all of my repositories, but I never took the time to really get to the bottom of what was inside there.

tab_bar

Github defines Actions as the following:

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can create workflows that build and test every pull request to your repository, or deploy merged pull requests to production.

To overly simplify things. Github Actions allows you, the developer, to write your own custom “workflows” where you define step-by-step what actions should happen after your workflow has been triggered. Even better, is that you can leverage already built workflow steps by looking through the Github Marketplace. I’m going to take the next few posts to break down into more detail one particular Github Actions Workflow I put together and use a lot. Brush up on working in YAML if you want to follow along!

name: Sample GH Action Workflow


on:
  workflow_dispatch:
    inputs:
      SUB_DIR:
        required: true
        type: string
      ENVIRONMENT:
        required: true
        type: string
      PROJECT:
        required: true
        type: string
      REGION:
        required: true
        type: string
      ZONE:
        required: true
        type: string
      GCP_AR_REPO:
        required: true
        type: string


env:
  SUB_DIR: ${{ inputs.SUB_DIR }}
  WORKINGDIR: ints/${{ inputs.SUB_DIR }}
  WORK_QUEUE: ${{ inputs.ENVIRONMENT }}
  ENVIRONMENT: ${{ inputs.ENVIRONMENT }}
  PROJECT: ${{ inputs.PROJECT }}
  REGION: ${{ inputs.REGION }}
  ZONE: ${{ inputs.ZONE }}
  GCP_AR_REPO: ${{ inputs.GCP_AR_REPO }}
  GCE_INSTANCE: prefect-docker-${{ inputs.ENVIRONMENT }}-vm
  GCE_INSTANCE_ZONE: ${{ inputs.REGION }}-${{ inputs.ZONE }}


jobs:
  changes:
    name: Code & dependency changes
    runs-on: ubuntu-latest
    outputs:
      prefect_flows: ${{ steps.filter.outputs.flows_files }}
      prefect_flows_changed: ${{ steps.filter.outputs.flows }}
      code_dependencies_changed: ${{ steps.filter.outputs.docker_code }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Generate Markdown Summary
        run: echo "Starting CI/CD for flows and dependencies added/modified with commit $GITHUB_SHA" >> $GITHUB_STEP_SUMMARY

      - uses: dorny/paths-filter@v2
        id: filter
        with:
          list-files: json
          filters: |
            flows:
              - added|modified: "${{ env.WORKINGDIR }}/flows/**/*"
              - added|modified: "${{ env.WORKINGDIR }}/blocks.py"
              - added|modified: "${{ env.WORKINGDIR }}/deployments.py"
            docker_code:
              - added|modified: "${{ env.WORKINGDIR }}/flows/**/*"
              - added|modified: "${{ env.WORKINGDIR }}/requirements.txt"
              - added|modified: "${{ env.WORKINGDIR }}/Dockerfile"
              - added|modified: "ints/Dockerfile.agent"

      - name: Generate Markdown Summary
        run: |
          echo Flows: ${{ steps.filter.outputs.flows_files }} >> $GITHUB_STEP_SUMMARY
          echo Code dependency changes: ${{ steps.filter.outputs.code_files }} >> $GITHUB_STEP_SUMMARY


  prefect_deploy:
    needs: changes
    if: needs.changes.outputs.prefect_flows_changed == 'true'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Set up Python 3.10
        uses: actions/setup-python@v4
        with:
          python-version: '3.10.4'
          architecture: 'x64'

      - name: Python dependencies
        working-directory: ${{ env.WORKINGDIR }}
        run: |
          pip install .

      - name: Prefect Cloud login
        run: |
          prefect config set PREFECT_API_KEY=${{ secrets.PREFECT_API_KEY }}
          prefect config set PREFECT_API_URL=https://FAKE_URL.COM

      - name: Deploy Blocks to Prefect Cloud
        id: build-blocks
        working-directory: ${{ env.WORKINGDIR }}
        run: |
          cat <<EOF > gcp_cred_block.py
          from prefect_gcp import GcpCredentials

          service_account_info = ${{ secrets.GCP_CREDENTIALS }}
          gcp_creds = GcpCredentials(service_account_info=service_account_info)
          gcp_creds.save("my-srvc-usr", overwrite=True)
          EOF
          python gcp_cred_block.py
          python blocks.py --env $ENVIRONMENT

      - name: Deploy Flows to Prefect Cloud
        id: build-deployments
        working-directory: ${{ env.WORKINGDIR }}
        run: |
          python deployments.py --deployments-ver $GITHUB_SHA --env $ENVIRONMENT

      - name: Upload YAML deployment manifest as artifact
        uses: actions/upload-artifact@v3
        with:
          name: Deployment YAML manifests
          path: ./ints/**/*.yaml


  gcp_deploy:
    needs: changes
    if: needs.changes.outputs.code_dependencies_changed == 'true'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    # id-token: write is used by Google auth to request an OpenID Connect JWT Token https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#adding-permissions-settings
    outputs:
      image: ${{ steps.build-image.outputs.image }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Login to GAR
        uses: docker/login-action@v2
        with:
          registry: "${{ env.REGION }}-docker.pkg.dev"
          username: _json_key
          password: ${{ secrets.GCP_CREDENTIALS }}

      - name: Set docker image URI
        run: |
          echo "AGENT_IMG_URI=$REGION-docker.pkg.dev/$PROJECT/$GCP_AR_REPO/my-project-prefect-agent-$ENVIRONMENT:latest" >> $GITHUB_ENV
          echo "FLOWS_IMG_URI=$REGION-docker.pkg.dev/$PROJECT/$GCP_AR_REPO/$SUB_DIR-flows-$ENVIRONMENT:latest" >> $GITHUB_ENV

      - name: Build and Push Agent Docker Image
        id: build-agent-image
        working-directory: ints
        run: |
          docker build \
          --build-arg PREFECT_API_KEY=${{ secrets.PREFECT_API_KEY }} \
          --build-arg PREFECT_API_URL=https://FAKE_URL.COM \
          --build-arg PREFECT_WORK_QUEUE="$WORK_QUEUE" \
          -t "${{ env.AGENT_IMG_URI }}" \
          -f Dockerfile.agent .
          docker push "${{ env.AGENT_IMG_URI }}"

      - name: Build and Push Flows Docker Image
        id: build-flows-image
        working-directory: ${{ env.WORKINGDIR }}
        run: |
          docker build --build-arg ENVIRONMENT=$ENVIRONMENT -t "${{ env.FLOWS_IMG_URI }}" .
          docker push "${{ env.FLOWS_IMG_URI }}"

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v1
        with:
          credentials_json: "${{ secrets.GCP_CREDENTIALS }}"

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v1

      - name: Delete VM if exists
        continue-on-error: true
        run: gcloud compute instances delete "$GCE_INSTANCE" --zone "$GCE_INSTANCE_ZONE" --quiet

      # Scopes defines which services VM needs, SA sets permissions for those
      - name: Deploy Prefect Agent VM
        run: |
          gcloud compute instances create-with-container "$GCE_INSTANCE" \
            --zone "$GCE_INSTANCE_ZONE" \
            --machine-type "e2-micro" \
            --scopes "cloud-platform" \
            --service-account "[email protected]" \
            --container-image "${{ env.AGENT_IMG_URI }}"

I’d like to just quickly summarize what it actually does. When triggered, the workflow looks for modifications made to specific files in the repo. If any of those files were, indeed, touched the workflow proceeds to deploy changes to two targets: Prefect Cloud, and Google Cloud Platform. This workflow handles all of the CI/CD needed to add and maintain mission critical ELT pipelines. All without ever leaving Github.

#TODO data engineering

What are Github Actions?