At Remotion we aim to release fast as part of our strategy to maintain quality. But quality and speed are normally thought of as two ends of a spectrum, so how do we do that exactly? What does our process look like that lets us both release fast and maintain quality?
Today we’ll cover how we use Xcode Cloud to help us ensure quality in our PR process, while keeping developer friction to a minimum. In later posts we’ll cover other topics like how we use CD to release 5-30 internal builds for testing each day, how we validate builds before we release them, and what our distribution process looks like.
If you’d just like to see a quick tutorial on how to use Xcode Cloud with GitHub to check PRs build before you merge them, skip to the end.
Low friction, high value PRs
We believe that code review is very important to help ensure quality, but we only require one reviewer to approve a PR since there is a point of dimensioning returns when you go past one. We then dogfood what was just merged in internally, allowing us to quickly find issues. (I plan on covering the automatic build process for our internal testing in another post.)
There is one thing that no number of human reviewers are likely to find in a review though, and that is if a given change is actually going to build. We can discuss logic and argue about style, but only the compiler can tell us if our code is legal. And if we merge broken code then our CD system can’t provide us with builds for us to test.
The most common issue we ran into was forgetting to wrap some debug code in an #if DEBUG check, which causes release builds to fail while debug builds succeed. That means the build will work when you hit Run in Xcode, but fail once you try to build for production.
That is where Xcode Cloud’s integration with GitHub comes in. We have some branch protection rules turned on for the master branch, including "Require status checks to pass before merging”. Xcode Cloud in turn is configured to automatically build Remotion in release mode for all new PRs that are going to merge to master. That gives us a low cost, high value addition to the manual code review, and it has saved us a considerable amount of time since turning it on.
We first actually started investigating Xcode Cloud for deployment builds, but when I discovered how easy it was to integrate with GitHub I turned that on first, and arguably that has been much more valuable given how easy it was to enable.
Why Xcode Cloud instead of another tool?
Xcode Cloud wasn’t the first tool we considered migrating to. We also considered Bitrise, and even almost got that working for our builds. So why did we pick Xcode Cloud instead?
Some of the reasons we chose Xcode Cloud:
macOS is a first class citizen! (some CI platforms are mobile/web focused, like Bitrise)
Integration with Xcode (and feels familiar to someone used to configuring projects in Xcode)
New versions of Xcode and macOS are usually available the same day as they are released (including betas)
Super configurable, yet easy to understand, build settings (check out some great examples of the flexibility of Xcode Cloud’s settings in the slides from Pol Piella Abadia’s great NYSwifty presentation)
Workflows don’t have to be tied to a single branch, you can start one off builds on any branch (looking at you AppCenter…)
Can be used for all of our CI needs (YMMV)
Slack integration for build updates
Some of these features are available on other platforms of course, but since Remotion is a macOS native application the Xcode integration and first class macOS support are key.
Xcode Cloud vs AppCenter
Platform support
Xcode Cloud only supports Apple’s platforms. Basically, if you can build it in Xcode, you can build it in Xcode Cloud. For some people that is a limitation, but for us it means our macOS application is actually a first class citizen! No need to try and make an iOS focused tool work for us.
iOS apps are only distributed via TestFlight and the AppStore, neither of which we use, so iOS tools don’t always fill the needs for macOS distribution.
While AppCenter is Microsoft owned, some of you might remember AppCenter was actually HockeyApp before Microsoft bought it. That heritage means it has support for all the major platforms, not just Microsoft’s. And (by some miracle) that list includes explicit macOS support! But that long list of support is a double edge sword, however, as we’ll see in the next point.
Xcode version support
Since Xcode Cloud is basically just Xcode in the cloud, and is provided by the same company that makes Xcode, new Xcode and macOS releases are almost always available on the same day of their release on Xcode Cloud.
In AppCenter we’ve had to wait weeks before we can update to the latest version of Xcode for our CD builds. That’s tradeoff you get for supporting so many platforms. Since we only build for macOS, it seemed obvious which of the two is best for us in this regard.
Obviously you shouldn’t immediately jump on the newest version of Xcode on day one for your release builds, but it is important to have the freedom and flexibility to make that jump on your terms. And it is very useful to have access to the newest Xcode for test and validation builds so you can decide when you do want to move forward for your release builds.
This issue was especially painful when Xcode 13.3 came out and it changed the format of the package.resolved file. Apple should have made that a backwards compatible change (especially as a minor update), but they didn’t. That meant when the first developer at Remotion started using Xcode 13.3 locally while AppCenter was still on 13.2 the AppCenter builds stopped working because of the incompatible package.resolved file.
Fixing our AppCenter build required us to revert the package.resolved file, but Xcode 13.3 kept updating the file and if we weren’t careful the file would get merged in again and break AppCenter again. So we decided to all downgrade Xcode to avoid updating the file until AppCenter finally made 13.3 available.
Check out the excellent Xcodes app if you have found yourself juggling multiple versions of Xcode as well! https://github.com/XcodesOrg/XcodesApp And there is a command line version too. https://github.com/XcodesOrg/xcodes
With Xcode Cloud, that is no longer a problem. No matter what version of Xcode you use locally, that same version will be available in Xcode Cloud as well. This also means no waiting for the latest Swift updates!
Other tools promise faster turnaround than AppCenter, so you don’t have to use Xcode Cloud just to get the latest Xcode faster.
Build Configuration
I love Xcode Cloud workflows. No, I haven’t used all the competitors out there, but I haven’t yet found something I couldn’t do with an Xcode Cloud workflow that I wish it could.
One of my favorite differences things in Xcode Cloud is how workflows aren’t directly tied to an individual branch like build settings are in AppCenter. In AppCenter you first have to clone the settings from one branch to another, which is an extra step, and means you’ve now fractured your build settings.
In fact, in Xcode Cloud you can start a build using on an open PR or tag as well as branches! No need to make any changes to the workflow.
Not only is cloning build settings to each branch more work to start the build, it also means that any time you need to update one setting, like updating the Xcode version after it the latest is finally made available, or replacing your provisioning profile, you have to manually go through all the branches and apps you have set up. This is super painful and can definitely lead to broken builds and headaches. In Xcode Cloud, just update your short list of workflows.
If someone knows how to bulk edit build settings in AppCenter, please let me know!
Of course, just because you can easily start manual builds in Xcode Cloud doesn’t mean that is the only way to do so. There are a four options for when the build should start automatically:
Branch Changes
Pull Request Changes
Tag Changes
On a Schedule for a Branch
AppCenter’s only auto option is On Push (same idea as Branch Changes). To set up a nightly build we had to set up a timed GitHub action to push changes to a specific branch set up to build On Push in AppCenter.
You can even set up multiple triggers for a workflow so you don’t have to duplicate the workflow to get the same output out of different triggers, further reducing the fractured settings problem AppCenter has.
You can have multiple Actions and Post-Actions as well, letting you automate a swath of use cases in a single workflow, which AFAIK is only possible in AppCenter by creating multiple branches or multiple apps each with different build settings for the same branch.
All of the options except On a Schedule for a Branch (the Changes options) also have a setting where you can provide a list of files that should trigger or prevent the build from starting when they are changed. This is useful for preventing automated tests from running when you just changed a readme, for example. Or you can start a certain group of tests only when the pertinent files changed, which pairs real nicely with using Swift Packages since you can set the Test Action to run the tests from one package and then only select the Source folder of that package as the build trigger. 🤯
Drawbacks
Some of the challenges we’ve had with Xcode Cloud include:
It has gone down a few times in the past year, preventing us from merging PRs
There is an option that surfaces a ‘skip check’ checkbox in the PR in GitHub which mitigates this issue, but we have that off to avoid accidents. For a handful of important PRs though we have turned that on long enough to merge and then turned it off again though, but only when we need something merged ASAP and we are really sure it builds correctly and won’t break anything.
A few mysterious build errors that were cleared by restarting the build
And that’s really it.
Fun fact: A teammate messaged me to inform me of one of these rare ‘mysterious build errors’ in the exact moment I was writing that bullet point. 🤣
Auto PR build Gotchas (that aren’t Xcode’s fault)
Occasionally we’ll run into an issue where the PR source branch builds, but then after the PR merges to master the AppCenter build fails. Most of the time that is due to us using Xcode Cloud for the PR builds and AppCenter for our release builds. That will be resolved once we migrate our distribution builds to Xcode Cloud, but while we are in the middle of the transition these are things we have to watch out for:
I once added a package dependency using the git@ url scheme instead of an https url and Xcode on my machine and Xcode Cloud both handled that fine, but then AppCenter failed to retrieve the package and the build failed.
Fix: only use https urls for Swift Package Manager dependencies
We have a build step that runs SwiftLint which may output warnings and errors, and for some reason some SwiftLint errors don’t break the build when built locally or in Xcode Cloud, but they do break builds in AppCenter?!? (This happened recently, so I haven’t had time to figure out why exactly yet.)
Fix: Correct the SwiftLint errors OR disable SwiftLint in AppCenter builds (we did the first)
And there are a couple issues that our PR builds can’t resolve no matter the CI/CD platform used:
We don’t notarize in PR builds, so if there is an issue with notarization we won’t catch that before merging
This has come up when we add helper applications within our binary. We’ve added three of our own, each with a different signing issue, and one time we added a vendor provided framework that they had already signed and we had to re-sign.
In these cases we can test notarization is working before we merge by manually building in AppCenter
It is possible for two independent PRs to build properly but then cause the build to break after they are both merged.
This has only happened to us once, with two branches made by two different people that merged within an hour of each other. One branch removed some code, and the other added some code, which resulted in an orphaned line of code that broke the build.
Fix: Enable "Require branches to be up to date before merging” in GitHub branch protection rules. OR also automatically build the target branch after merging and monitor that for build failures (we have Slack integration turned on that tells us when a build fails).
We use the second method since it allows us to merge faster and use much less build hours. Enabling the setting requiring each PR to be up to date with the target branch would mean each open PR would need to be rebuilt each time one is merged, effectively rate limiting how often you can PR, and burning through your build time. We merge at least a dozen, sometimes two, PRs a day, so the impact would add up extremely quickly. Also, we already have automatic builds for each merge anyway (again, I plan on discussing that in a future post).
Setting up Xcode Cloud for GitHub PR checks
If it looks like Xcode Cloud will work for you I hope you give it a shot. I’ll share our Xcode Cloud workflow and GitHub branch protection rules here so you can see how to try it for yourself. Let us know what you think if you try it out!
Our Xcode Cloud workflow:
The pertinent GitHub branch protection rules:
At the moment this is all we are using Xcode Cloud for, but I plan on turning on some automatic unit tests with it soon. We are considering moving our CD builds to Xcode Cloud too, but at the moment AppCenter is handling that part just fine so we’ve left it alone for now. “If it ain’t broke, don’t fix it”, after all.
If we do make that move I’ll be sure to document that process and share that with you here as well, so watch out for that!
So, what do you think? Want to give Xcode Cloud a shot with us? 🤔
