I recently had the opportunity to try migrating a large commercial project to the Xcode Cloud for testing purposes. In this post, I will share my thoughts and the problems I encountered.
I assume that you have a general idea of how Xcode Cloud works and what it offers. If not please at least take a look at my post: Xcode Cloud – overview & setup.
Why Xcode Cloud is worth trying out
All-in-one
As we all know, it is very comfortable to have everything built into one ecosystem. And this is what Apple gives us with Xcode Cloud.
Now you can see your CI/CD system directly in Xcode. You can check your builds, change workflows, check which test failed and navigate directly to its source code. You can even see snapshots from your tests and rebuild branch.
This is a really significant advantage, it provides a smooth experience without leaving Xcode.
Familiar experience
Xcode Cloud gives also a similar experience to our daily work with Xcode. Test results are displayed in the same way, we can see logs and warnings as if we run the build locally. We don’t need extra tools to format test results etc. This is very convenient.
Simplicity
Another important thing is the simplicity of the configuration. Now you just add Xcode Cloud to your project and it gets automatically access to all provisioning profiles and certificates. You can instantly build your project and release it to TestFlight and App Store.
You don’t have to generate API keys, export certificates, go from door to door to find the person who’s got a distribution certificate, etc. If you have ever set up CI/CD, you know what I am talking about :D.
Team-wide access
Once you set up Xcode Cloud, all your team members added to the project will automatically get access to see and run builds in Xcode. One place less to give & revoke access.
Day one support
We can quite reasonably assume that Xcode Cloud will get a day one support for every new platform feature, beta system, and beta simulator. This is another significant advantage.
Future development
For now, Xcode Cloud is very simple, but I assume that Apple will present new features soon. Also, the environment is dedicated to Apple apps, so we can expect that build performance will be increasing with time.
Price
For now, Xcode Cloud has very competitive plans. First of all, you get 25 hours/month for free until December 2023. However, let’s assume that our large project is using 250 hours/month. Let’s compare it to Bitrise which is another great and mature solution for mobile CI/CD.
Xcode Cloud 100 hours plan costs $44.99/month. On Bitrise we will need 100*60*4 credits = 24,000 credits, assuming the average machine “8vCPU @3.2GHz 35 GB RAM running on Intel Mac Minis”. We will need to buy a 30,000 credits package having a Teams plan, in total $1,350.00! It’s 30x more than Xcode Cloud! This is a huge difference!
Last comparison. Xcode Cloud biggest plan is available @ $399.99 for 1000 hours! On Bitrise we would need to contact support to get pricing for more than 50,000 credits, but based on known prices we could assume that it would cost around $8,000!
I think in the future this could kill other CI/CDs or at least force them to lower prices.
Why Xcode Cloud is not ready yet
If you used to work with Bitrise, Jenkins, or any other mature CI/CD, you will instantly notice Xcode Cloud limitations. Then it depends on your project if you can live with them or not :). Below you can read a list of problems and limitations that I noticed.
Xcode Cloud is still unstable
Xcode Cloud is officially no longer in beta. However, builds are very unstable at least if you are using external dependencies. I encounter build failure every ~5 builds because of some issue with network connection while downloading dependencies using CocoaPods, Bundler, or RubyGems.
You will also encounter some issues with Xcode. I experienced multiple crashes, not loading logs, not saving configurations, etc.
Sometimes there are also problems with the repository connection. I encountered this issue both with GitHub and Bitbucket. Even when writing this post, Xcode Cloud just lost the connection to Bitbucket and can’t clone my repository. Even though the repository is working perfectly fine. I can’t even connect it again. Most likely I will need to wait like the last time.
This is quite a problem if you can’t rely on something that is supposed to increase your reliability, I think.
No custom scheduled work
This is an interesting issue I noticed. You can schedule builds but you can’t schedule some work without at least one action like Build, Archive, Test, or Analyze. You can always create an empty project to make it fast, but still, it’s strange. I think it will be improved in the future.
Scheduled work without build might be useful to update your dsyms on Firebase, backup your repository, or something like that.
No certificates management
Workflow is limited to the account that you selected which means that you can’t release an app to your in-house TestFlight (common in software houses) and then to your client’s TestFlight from a single Xcode Cloud.
This is not a big deal, because you can set up two workflows and two Xcode Clouds on both accounts. The only disadvantage is that you will need to pay for two subscriptions. However, taking into account current pricing, it would be still very cheap.
Logs are really annoying
If you want to track build logs in Xcode it is not really efficient. Every single time you leave the “Logs” tab and go back again, you have to wait and watch how logs are loading from the oldest to the newest entry.
This is really annoying. I’ve just checked it on my project and to see the last line of logs it takes around 90 seconds using a really fast internet connection.
No predefined actions
If you need to do something more than Build, Analyze, Archive, and Test, you are going to write your scripts from scratch. If you are used to Bitrise, you are going to miss predefined open-source steps to do some extra work.
No sudo access
This could be a problem if you need some extra work in scripts for example using a tool that requires sudo permission. Currently, sudo access is not allowed on Xcode Cloud.
No versioning for Xcode Cloud
Xcode Cloud doesn’t support configuration stored in a repository. Therefore, you can’t easily share your config or track its changes in git. Also if someone breaks or deletes a workflow, you won’t be able to easily restore it or see what changed.
Hours limit
Currently, the maximum Xcode Cloud plan gives you 1000 hours. I just wonder what if you are working in a software house with a single account and 1000 hours is not enough for a whole organization :D.
Xcode Cloud decides what is the lowest iOS version
If you need to support older iOS versions and you want to run some UI/Unit Tests on such simulators, it might not be possible with Xcode Cloud. Currently, the lowest supported iOS is 13.7 and most likely it will be abandoned once iOS 16 is released.
No cache control
Using typical CI/CD you can usually do a lot of things to optimize your build times. One of the possible improvements is to cache things like CocoaPods. Unfortunately, on Xcode Cloud you don’t have a control over the cache.
No custom artifacts
I couldn’t find anything about storing custom artifacts on Xcode Cloud. The only thing I found is a 1-year-old post from an Apple employee saying that it is not possible.
No remote access for debugging
From time to time in large projects, you will encounter some issue that happens only on CI/CD. It is really useful then to connect to the machine while running a build. Usually, you can use a remote desktop or at least SSH.
Unfortunately, it is not possible with Xcode Cloud.
No M1/M2 CPU? 😢
Finally, I was setting up Xcode Cloud really with the hope that build-time will be much faster than on our previous CI/CD because of the Apple Silicon. Unfortunately, it turns out that Xcode Cloud, for now, is not using M1 nor M2 most likely, because build times are the same as on Intel machines.
Hopefully, they will soon migrate to Apple Silicon.
UPDATE
I managed to run the following commands on Xcode Cloud:
|
1 2 |
uname -m system_profiler SPSoftwareDataType SPHardwareDataType |
And I received the following information about the machine:
Architecture: x86_64
Software:
System Software Overview:
System Version: macOS 12.4 (21F79)
Kernel Version: Darwin 21.5.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Computer Name: 13807fd0-1535-412d-907f-cb923ed54ec4-6489796869-d4dwf-vm
User Name: local (local)
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled
Time since boot: 24 minutes
Hardware:
Hardware Overview:
Model Name: Mac
Model Identifier: MacVM1,1
Processor Name: Unknown
Processor Speed: 2 GHz
Number of Processors: 4
Total Number of Cores: 4
L2 Cache (per Processor): 4 MB
L3 Cache (per Processor): 16 MB
Memory: 16 GB
System Firmware Version: AVM11.88Z.0003.D00.2110230656
OS Loader Version: 540.120.3~6
SMC Version (system): 1.13f3
Serial Number (system): XCCGXAP7PWPC
Summary
As you can see, there are still a lot of problems and limitations of Xcode Cloud. However, I think it has potential and it could be a great built-in CI/CD once the listed problems are addressed. In my opinion, it was too early to get rid of the “beta” suffix.
I think, for now, it is also too early to use Xcode Cloud in large and mature commercial projects. However, if you have a very limited budget or if you are just starting a new project you should give it a try. At least to see how it works. Remember you have 25 hours for free!
Apple – keep up the good work and make it our first-choice CI/CD :)!
