What are Github Actions? - Part 4
Last week I alluded to how our first job’s output dictates whether or not downstream workflow jobs even need to be ran. Declaring this dependency is faily simple in the GitHub Action
syntax. Similar to my previous post, I am going to walk us through the interesting bits of the second jobs entry. See below for the snippet we’ll be working through today.
prefect_deploy:
needs: changes
if: needs.changes.outputs.prefect_flows_changed == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: '3.10.4'
architecture: 'x64'
- name: Python dependencies
working-directory: ${{ env.WORKINGDIR }}
run: |
pip install .
- name: Prefect Cloud login
run: |
prefect config set PREFECT_API_KEY=${{ secrets.PREFECT_API_KEY }}
prefect config set PREFECT_API_URL=https://FAKE_URL.COM
- name: Deploy Blocks to Prefect Cloud
id: build-blocks
working-directory: ${{ env.WORKINGDIR }}
run: |
cat <<EOF > gcp_cred_block.py
from prefect_gcp import GcpCredentials
service_account_info = ${{ secrets.GCP_CREDENTIALS }}
gcp_creds = GcpCredentials(service_account_info=service_account_info)
gcp_creds.save("my-srvc-usr", overwrite=True)
EOF
python gcp_cred_block.py
python blocks.py --env $ENVIRONMENT
- name: Deploy Flows to Prefect Cloud
id: build-deployments
working-directory: ${{ env.WORKINGDIR }}
run: |
python deployments.py --deployments-ver $GITHUB_SHA --env $ENVIRONMENT
- name: Upload YAML deployment manifest as artifact
uses: actions/upload-artifact@v3
with:
name: Deployment YAML manifests
path: ./ints/**/*.yaml
needs
With needs you are able to quickly list any jobs that need to be successfully completed prior to this job kicking off. Potential values here can either be a string (if
you only depend on a single job) or an array of stings (to list multiple jobs that need to successfully complete).
if
🥹 Our dear friend the if-statement! This works exactly how you probably are already imagining. The conditional will only allow a job to run if the logic in the if evaluates to
True.
steps
Our first three steps here can be grouped together and probably don’t require that much of a detailed explanation. These can be thought of as setting up our ubuntu runner.
As a matter of fact, you could probably glean all you needed to know from just reading the name of each of the three steps. If our changes job successfully runs, then
with our first three steps of the prefect_deploy our workflow runner gather’s the state of our repo at the time of the workflow trigger, ensures that python is available
for us to use, and lastly we perform a good ole pip install because we are going to need some of our python modules as we’ll soon see. Notice our use of the previously
covered env when we set the working directory we want to be in when we run our pip install.
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: '3.10.4'
architecture: 'x64'
- name: Python dependencies
working-directory: ${{ env.WORKINGDIR }}
run: |
pip install .
The remaining steps in this prefect_deploy job directly relate to deploying new or modified files that need to be synced to my Prefect Cloud workspace.
Some of these steps may not necessarily make the most sense (especially if you are not familiar with Prefect), but they are still good
examples of how you can use your workflow steps to perform some simple cli commands to aid with your deployments.
- name: Prefect Cloud login
run: |
prefect config set PREFECT_API_KEY=${{ secrets.PREFECT_API_KEY }}
prefect config set PREFECT_API_URL=https://FAKE_URL.COM
- name: Deploy Blocks to Prefect Cloud
id: build-blocks
working-directory: ${{ env.WORKINGDIR }}
run: |
cat <<EOF > gcp_cred_block.py
from prefect_gcp import GcpCredentials
service_account_info = ${{ secrets.GCP_CREDENTIALS }}
gcp_creds = GcpCredentials(service_account_info=service_account_info)
gcp_creds.save("my-srvc-usr", overwrite=True)
EOF
python gcp_cred_block.py
python blocks.py --env $ENVIRONMENT
- name: Deploy Flows to Prefect Cloud
id: build-deployments
working-directory: ${{ env.WORKINGDIR }}
run: |
python deployments.py --deployments-ver $GITHUB_SHA --env $ENVIRONMENT
Lastly, remember that if syntax at the start of this job?
if: needs.changes.outputs.prefect_flows_changed == 'true'
I configured this job to only run if files that match the flows filter in my changes job are either added or modified (see below for a quick refresher).
jobs:
changes:
name: Code & dependency changes
runs-on: ubuntu-latest
outputs:
prefect_flows: ${{ steps.filter.outputs.flows_files }}
prefect_flows_changed: ${{ steps.filter.outputs.flows }}
code_dependencies_changed: ${{ steps.filter.outputs.docker_code }}
...
- uses: dorny/paths-filter@v2
id: filter
with:
list-files: json
filters: |
flows:
- added|modified: "${{ env.WORKINGDIR }}/blocks.py"
- added|modified: "${{ env.WORKINGDIR }}/deployments.py"
So, if my new commits to the repo haven’t touched any of the files listed in that filter this job has no reason to even run! For this workflow this shaves about 1 minute off the workflow run time. This type of conditional job setup can definitely be a good tool to keep in mind for other workflows.
There is one more job that I shared in the first post of this series. I am going to leave for you readers to grok.
gcp_deploy:
needs: changes
if: needs.changes.outputs.code_dependencies_changed == 'true'
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
# id-token: write is used by Google auth to request an OpenID Connect JWT Token https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#adding-permissions-settings
outputs:
image: ${{ steps.build-image.outputs.image }}
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Login to GAR
uses: docker/login-action@v2
with:
registry: "${{ env.REGION }}-docker.pkg.dev"
username: _json_key
password: ${{ secrets.GCP_CREDENTIALS }}
- name: Set docker image URI
run: |
echo "AGENT_IMG_URI=$REGION-docker.pkg.dev/$PROJECT/$GCP_AR_REPO/my-project-prefect-agent-$ENVIRONMENT:latest" >> $GITHUB_ENV
echo "FLOWS_IMG_URI=$REGION-docker.pkg.dev/$PROJECT/$GCP_AR_REPO/$SUB_DIR-flows-$ENVIRONMENT:latest" >> $GITHUB_ENV
- name: Build and Push Agent Docker Image
id: build-agent-image
working-directory: ints
run: |
docker build \
--build-arg PREFECT_API_KEY=${{ secrets.PREFECT_API_KEY }} \
--build-arg PREFECT_API_URL=https://FAKE_URL.COM \
--build-arg PREFECT_WORK_QUEUE="$WORK_QUEUE" \
-t "${{ env.AGENT_IMG_URI }}" \
-f Dockerfile.agent .
docker push "${{ env.AGENT_IMG_URI }}"
- name: Build and Push Flows Docker Image
id: build-flows-image
working-directory: ${{ env.WORKINGDIR }}
run: |
docker build --build-arg ENVIRONMENT=$ENVIRONMENT -t "${{ env.FLOWS_IMG_URI }}" .
docker push "${{ env.FLOWS_IMG_URI }}"
- name: Authenticate to Google Cloud
uses: google-github-actions/auth@v1
with:
credentials_json: "${{ secrets.GCP_CREDENTIALS }}"
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v1
- name: Delete VM if exists
continue-on-error: true
run: gcloud compute instances delete "$GCE_INSTANCE" --zone "$GCE_INSTANCE_ZONE" --quiet
# Scopes defines which services VM needs, SA sets permissions for those
- name: Deploy Prefect Agent VM
run: |
gcloud compute instances create-with-container "$GCE_INSTANCE" \
--zone "$GCE_INSTANCE_ZONE" \
--machine-type "e2-micro" \
--scopes "cloud-platform" \
--service-account "[email protected]" \
--container-image "${{ env.AGENT_IMG_URI }}"