Testing End-to-End Locally with Waldo Scripting

In my previous article, I described step by step how I took an existing iOS app and added support for end-to-end testing with Waldo Scripting by using a GitHub Actions workflow.

While end-to-end testing from my CI is great when I am ready to open a pull request, for day-to-day development work I need something a bit more interactive and responsive. When I make a change to my app’s source code, I want to be able to quickly run my end-to-end test scripts against my changed app without having to go through a full CI cycle.

Since testing with Waldo Scripting requires that I have uploaded a build of my app, I chose to set up my local environment such that I can build and upload to Waldo on demand and run my test scripts locally with ease. (And because I am lazy, I made it as dead simple as possible.)

If you have similar needs, then read on. The rest of this article details how I incorporated Waldo Scripting into my daily development cycle by writing a straightforward Bash script.

Preliminaries

Before writing the script, I first took care of a couple of preliminary tasks:

1. I followed the instructions here to install Waldo CLI and authenticate access to Waldo with my API token.
2. I then ran brew install jq, because I strongly suspected that I’d need to parse a bit of JSON from the command line.

At that point, since I already had my app uploaded and had Waldo Scripting integrated, I was ready to buckle down and start “Bash”-ing out some code!

Writing the Bash Script

Essentially I picked up right where I left off in the previous article. I changed nothing in the TravelSpot app itself, nor did I touch any part of the Waldo Scripting integration from last time: nothing at all in the waldo subfolder — not even the test script in tests/onboarding.ts.

To keep things simple, I decided to write a Bash script that

1. builds the app,
2. uploads it to Waldo, and
3. runs the end-to-end tests against it.

Sound familiar? These are the same exact three tasks I needed to implement in my GitHub Actions workflow. However, the details are somewhat different, so I will walk you through pretty much the whole script and explain everything, a little bit at a time.

Locating the build artifact

Just as in the GitHub Actions workflow, I used xcodebuild to build the TravelSpot app, once again specifying an explicit derived data path:

xcodebuild -project TravelSpot.xcodeproj            \
           -scheme TravelSpot                       \
           -configuration Release                   \
           -sdk iphonesimulator                     \
           -derivedDataPath "$DERIVED_DATA_PATH"    \
           build

I located the build artifact relative to the derived data path, and then saved that location in an environment variable named APP_DIR_PATH so I could use it later on:

APP_DIR_PATH=${DERIVED_DATA_PATH}/Build/Products/Release-iphonesimulator/TravelSpot.app

‍

No secrets

Uploading the TravelSpot.app build artifact required a little bit of cleverness since, unlike last time, I couldn’t rely on Waldo’s custom upload action — it isn’t available outside of GitHub Actions. Instead, I made use of Waldo CLI:

waldo upload --app_id $APP_ID "$APP_DIR_PATH" | tee "$UPLOAD_OUT_PATH"

‍

There are a couple of interesting points to make about that preceding line of code:

1. Notice that I did not specify the --upload_token option, whereas I did specify the --app_id option.

Waldo requires an “upload token” when uploading a build artifact. Rather than use the CI token associated with TravelSpot like in the GitHub Actions workflow, I chose to supply my personal API token instead. Since I already provided it when I installed Waldo (see the Preliminaries section above), waldo upload can simply use it without requiring me to specify the --upload_token option. However, because an API token is not associated with a particular app, I am required to specify the --app_id option with the appropriate ID for the TravelSpot app. I was able to go to the “General” tab of the “Configuration” for the app to fetch this value.

2. If you are not familiar with the tee command, it simply duplicates standard input. By piping the output of waldo upload into tee, I am basically saving a copy of the output into a temporary file ($UPLOAD_OUT_PATH) while still writing to standard output as usual.

Why would I want to do that? As it turns out, the very last line of output from a successful call to waldo upload is a short blob of JSON that contains info about the upload — including the build ID that the Waldo backend assigned to that particular build artifact. You can see an example in the following excerpt from the output of a successful upload to Waldo:

The build ID is contained in the field named appVersionID.

Painless extraction

Now that I had a copy of the output stashed in a temporary file ($UPLOAD_OUT_PATH), it was just a matter of employing a bit of Bash jujitsu to extract the build ID:

BUILD_ID=$(tail -n 1 "$UPLOAD_OUT_PATH" | jq -r .appVersionID)

Let’s break this bad boy down:

• As you probably know, the tail command displays the last part of a file, while the -n option (shorthand for --lines) restricts the number of lines. Therefore, tail -n 1 "$UPLOAD_OUT_PATH" displays the last line of the temporary file, thus isolating the JSON upload info.
• The JSON upload info is then fed into the jq command. The .appVersionID filter (shorthand for .["appVersionID"]) extracts the value (if any) of the field named appVersionID from the top-level JSON object. The -r option (shorthand for --raw-output) returns the string value “as is” (meaning, without the surrounding double-quotes or any escaping).
• The resulting value is then saved into an environment variable named BUILD_ID for subsequent use.

Running the test scripts

With this last bit of info, I was able to run the test scripts using our old friend npm run wdio:

cd waldo && VERSION_ID=$BUILD_ID SHOW_SESSION=1 npm run wdio

‍

(You may recall that this command needs to be run in the context of the waldo folder, hence the initial cd waldo invocation.)

Avoiding unnecessary work

Now that I had the makings of a basic Bash script that accomplished my initial three tasks, I considered how I might avoid unnecessary work. Specifically, I wanted to avoid build and uploading the app if nothing had changed in the source code. After all, I could very well add a new test, or tweak an existing one, and merely want to run my end-to-end tests against the exact same build of the app that I had already uploaded earlier.

(Granted, this demo app takes less than 15 seconds to build from source and upload to Waldo from my laptop, but a full-featured app can easily take several minutes to build from source and another minute or two to upload to Waldo. Life is too short to waste it on that kind of nonsense.)

With that goal in mind, I determined that I needed to add a bit more logic to my Bash script. The high-level flow would look like this (in pseudocode):

checksum = computeChecksum()

if !findBuildInfo(checksum, &buildID) {
    artifact = buildApp()

    buildID = uploadApp(artifact)

    save(checksum, buildID)
}

runTestScripts(buildID)

‍

Computing the checksum

Computing a single checksum to uniquely represent the combined source code was the trickiest bit to get right. I opted to encapsulate this logic in a shell function named compute_source_md5:

function compute_source_md5() {
    find BuildSettings                          \
         Resources                              \
         Sources                                \
         TravelSpot.xcodeproj/project.pbxproj   \
         -type f                                \
         -not -name '.*'                        \
         -exec md5 -r {} + |                    \
    LC_ALL=C sort | md5
}

‍

Taking this piece by piece:

• Thefind invocation searches the directory trees of the specified paths looking for regular files (-type f) that are not hidden (-not -name '.*'). Notice that I did not include the waldo folder as it does not contain any files that should trigger a new build if they were to change. Each file that meets the criteria has its checksum calculated (-exec md5 -r {} +). That -r option on the md5 command reverses the formt of the output, so that the checksum ends up on left and the file name on the right, which makes for easier sorting.
• The aggregation of all these file checksums is then sorted in a locale-neutral fashion (LC_ALL=C sort) to maximize stability across multiple invocations.
• Finally, the checksum of this sorted list of file checksums is calculated. Thus, the end result is a single checksum uniquely representing the combined source code.

Finding previous build info

For simplicity, I made it so that every successfully build (and upload) appends a line to a single text file ($BUILD_INFO_PATH). Each line is in the format:

"${SOURCE_MD5}=${BUILD_ID}"

‍

Therefore, it is relatively straightforward to determine if a checksum ($SOURCE_MD5) has seen before, and, if so, what the associated build ID is.

I discovered it was easier to encapsulate this logic in a shell function. I named it find_build_info:

function find_build_info() {
    if [[ -r $BUILD_INFO_PATH ]]; then
        local _found=$(fgrep "${SOURCE_MD5}=" "$BUILD_INFO_PATH")

        if [[ -n $_found ]]; then
            BUILD_ID=${_found##*=}

            echo "Found saved build info -- will use build ${BUILD_ID} of app"

            return
        fi
    fi

    echo "No saved build info found -- will build app"

    BUILD_NEEDED=true
}

‍

It is pretty much self-explanatory as to what is going on here. The only obscure part is the line BUILD_ID=${_found##*=}, which is just a Bash incantation to extract everything following the last equal sign (=) contained in $_found.

All together now

Putting it all together, the final Bash script (which I named test-e2e-local.sh) weighed in at 90+ lines (and I am fairly generous in my use of blank lines to improve readability). I even added a couple of bells and whistles:

1. Force a fresh build (and upload), even if the source code has not changed.
2. Turn on verbose mode for both xcodebuild and waldo upload.

Check it out:

#!/bin/bash

set -eu -o pipefail

APP_ID="app-f9005a40b20d5559"
DERIVED_DATA_PATH="/tmp/TravelSpot-$(uuidgen)"

BUILD_ID=
BUILD_NEEDED=false
UPLOAD_OPTIONS=
XCODEBUILD_ACTIONS="build"
XCODEBUILD_OPTIONS="-quiet"

APP_DIR_PATH="${DERIVED_DATA_PATH}/Build/Products/Release-iphonesimulator/TravelSpot.app"
BUILD_INFO_PATH=".build-info.txt"
UPLOAD_OUT_PATH="${DERIVED_DATA_PATH}/upload.out"

function compute_source_md5() {
    find BuildSettings                          \
         Resources                              \
         Sources                                \
         TravelSpot.xcodeproj/project.pbxproj   \
         -type f                                \
         -not -name '.*'                        \
         -exec md5 -r {} + |                    \
    LC_ALL=C sort | md5
}

function find_build_info() {
    if [[ -r $BUILD_INFO_PATH ]]; then
        local _found=$(fgrep "${SOURCE_MD5}=" "$BUILD_INFO_PATH")

        if [[ -n $_found ]]; then
            BUILD_ID=${_found##*=}

            echo "Found saved build info -- will use build ${BUILD_ID} of app"

            return
        fi
    fi

    echo "No saved build info found -- will build app"

    BUILD_NEEDED=true
}

while (( $# )); do
    case $1 in
        force)
            BUILD_NEEDED=true
            XCODEBUILD_ACTIONS="clean build"
            ;;

        verbose)
            UPLOAD_OPTIONS="--verbose"
            XCODEBUILD_OPTIONS="-verbose"
            ;;
    esac

    shift
done

SOURCE_MD5=$(compute_source_md5)

if [[ $BUILD_NEEDED != true ]]; then
    find_build_info
fi

if [[ $BUILD_NEEDED == true ]]; then
    echo "Building app"

    xcodebuild -project TravelSpot.xcodeproj            \
               -scheme TravelSpot                       \
               -configuration Release                   \
               -sdk iphonesimulator                     \
               -derivedDataPath "$DERIVED_DATA_PATH"    \
               $XCODEBUILD_OPTIONS                      \
               $XCODEBUILD_ACTIONS

    waldo upload --app_id $APP_ID   \
                 $UPLOAD_OPTIONS    \
                 "$APP_DIR_PATH" | tee "$UPLOAD_OUT_PATH"

    BUILD_ID=$(tail -n 1 "$UPLOAD_OUT_PATH" | jq -r .appVersionID)

    echo "Saving build info"

    echo "${SOURCE_MD5}=${BUILD_ID}" >> "$BUILD_INFO_PATH"
fi

cd waldo && VERSION_ID=$BUILD_ID SHOW_SESSION=1 npm run wdio

exit

‍

You will likely be able to copy my work without too much trouble and use it as a launching pad for accelerating development on your own apps.

Happy scripting!

‍

‍