Apple had acquired TestFlight in 2014, and the much-anticipated announcement at WWDC gave many in the industry hope that TestFlight would spell the end for the numerous headaches associated with development builds and beta distributions. So where does TestFlight stand a year later? Has it lived up to these hopes?
UDIDs & Provisioning Profiles
One of the biggest pain points that TestFlight sought to solve was developers having to fuss with unique device identifiers, or UDIDs. When you deal with distributing apps outside of the App Store, TestFlight, or enterprise environments, the UDID for each test device must be in the app's provisioning profile.
There are some headaches that go along with this. It can be a pain for user's to get the UDID of their devices if they're not familiar; developers are limited to 100 devices per developer account; you can only remove devices once per year; and developers have to update the provisioning profile every time a device is added. The entire process of dealing with provisioning profiles can also be error prone and difficult to troubleshoot. Many developers were excited about TestFlight because it potentially spelled the end of these struggles. A year later, I think it's safe to say that Apple has delivered on this.
A year later, I think it's safe to say that Apple has delivered on this.
Now, to add a user to your TestFlight beta, you need only enter their email address and send them an invitation. Once they've accepted, they will see your app show up in Apple's TestFlight app. No dealing with UDIDs or provisioning profiles, and the tester limit increased dramatically from 100 devices to 1,000 Apple IDs (regardless of how many devices are associated to it).
The major caveat to all this is that your app must go through an Apple review first. If you wish to give testers access prior to an Apple review, you'll need to add them as an internal tester in TestFlight. Apps are limited to having 25 internal testers. That means, unless your testers are using more than four devices each, you'll be more limited opting for TestFlight than a third-party service in this scenario.
The major caveat to all this is that your app must go through an Apple review first.
There are a few other limitations of TestFlight that are worth noting. Most of these aren't necessarily things Apple failed to deliver on, they are limits that we knew TestFlight would have from the beginning.
- iOS is the only platform that is supported. If you develop cross-platform, you'll need an additional or alternative system.
- Only iOS 8 and later are supported. This will become less relevant over time, but developers wishing to support iOS 7 or earlier are out of luck with TestFlight.
- Only the latest build is available for download. This means you can't install old builds to compare builds, isolate when an issue was introduced, or perform upgrade testing from a previous major version.
- Only three (I think) builds can be uploaded each day.
- Limited support. If you run into an issue with TestFlight, your options for getting support will be more limited with Apple than they would be many of the other third-party solutions.
- Builds are only good for 30 days. After 30 days, the app will fail to launch and you'll either need to issue an update to your testers or they will need to install an App Store version of the app.
I don't have much experience with TestFlight from administration side, so there may be some key benefits and drawbacks that I've missed. I know there have been other complaints, like limited flexibility in roles & permissions you can assign to testers, but there's likely more too it than that as well.
While TestFlight is completely free and many third-party services cost money, even with App Store review, for most developers, TestFlight's convenience over dealing with UDIDs and profiles makes it an easy choice of which testing platform to go with.
Crash Reporting
Apple's crash reporting services have long lagged behind that of third-party services. TestFlight had good crash reporting before it was acquired by Apple, so it made sense that Apple would polish it and add it to their suite of tools for developers.
It's worth noting that while crash reporting was announced during WWDC last year, it was only launched a few months ago, so there are likely still some kinks being worked out. That said, Apple's crash reporting seems more limited and less useful than other third-party crash reporting services (I have a personal bias toward HockeyApp).
Apple only gives crash reports for users who opt-in to sharing diagnostic information with app developers.
Apple only gives crash reports for users who opt-in to sharing diagnostic information with app developers. All TestFlight users automatically agree to share this information, but for App Store users, the choice is left up to them. This is certainly nice from a privacy standpoint, but from the perspective of a developer who's trying to monitor the health of their apps and address crashes, only seeing crash logs for users who opted-in to sharing them may be limiting.
Most developers may want to consider a third-party crash reporting service for production apps for this reason alone. (It was also pointed out to me by Andreas Linde that developers can see how many of their users have opted in to sharing this information. This will be a helpful bit of info for developers trying to decide if Apple's crash reporting will be sufficient for them.) Interested developers can find this percentage in iTunes Connect by navigating to App-Analytics, clicking the app they're interested in, then clicking "About App Analytics Data" in the top right.
Overall, the new crash reporting that has come with TestFlight is an improvement.
Overall, the new crash reporting that has come with TestFlight is an improvement. The old iTunesConnect crash reporting was quite bad and only useful in a couple of scenarios. This new crash reporting seems like a potentially viable solution for indie developers, those who are unwilling to pay for a third-party service, or as a service that's supplemental to using a third-party service.
As stated previously, this crash reporting is still new. If Apple is interested in making it the best, we could see the company make some improvements over the next few months that have third-party crash reporting services sweating. If not, if Apple is only interested in providing an entry-level service, then developers may need to continue using third-party services for anything beyond the basic functionality currently offered by Apple.
I give Apple credit here for the improvement, but third-party services still offer the most features and greatest flexibility.
What it all means for developers
I see two strong use cases for TestFlight: smaller development studios that want to run hassle-free betas, and anybody wanting to perform user-acceptance testing with a large pool of testers. Personally, I have access to eight apps in TestFlight right now. Of those eight, I don't believe any of them have more than two developers on them, and most of them have one. For smaller dev shops, TestFlight offers a free, streamlined beta distribution mechanism that removes many of the headaches of dealing with UDIDs and provisioning profiles. If any of the shortcomings covered above aren't a deal-breaker for a developer, then TestFlight may be the perfect solution.
Even for larger dev shops that may employ another solution for internal distribution during development, TestFlight is an appealing option for user acceptance testing near the end of a project. Being able to add 1,000 users with nothing more than an email address makes TestFlight the easiest (if not only, outside of an enterprise account) solution for getting your app in the hands of testers and stakeholders. TestFlight's lack of UDIDs and provisioning profiles means that nobody else can compete with its simplicity.
What it means for testers
Once again, it's impossible to beat TestFlight's simplicity. As a TestFlight user, I don't have to worry about keeping the devices in my account up-to-date, or trying to help a developer troubleshoot provisioning profile problems. Using TestFlight is as simple as tapping a link in an email, then installing from the TestFlight app.
I've experienced a few minor frustrations with as a user though. For example, I can't accept an invitation from my computer — I have to accept an invitation from the device I want to test on. Also, TestFlight emails don't contain any release notes. With other services like HockeyApp, developer release notes are included in the email, so you can decide from the email if you care about the update or not. Lack of these release notes from TestFlight means you'll have to tap through to the app and view on your testing device to see if you want the update or not.
The biggest downside that I've experienced as a tester is builds expiring after 30-days. I've had betas expire where I then have to go install the App Store version to use the app. If a developer has introduced data model changes, or other significant modifications, installing the App Store version on top of a beta may result in data-loss or needing to delete the app and do a clean installation. If your developers keep fresh betas coming, this isn't an issue, but on multiple occasions I've found myself cursing the 30-day expiration of TestFlight builds.
TestFlight wins for both developers and testers in terms of convenience and simplicity, but it's important to be aware of all of the gotchas mentioned above.
The bottom line
TestFlight is a nice addition to Apple's tools for developers. A year later, I think it's safe to say the pros of having an Apple-owned TestFlight available to developers easily outweighs any of its cons. TestFlight comes with many clear limitations, and developers will need to evaluate for themselves if TestFlight will work for them, or if they need to go with a more robust third-party solution.
If you're a developer or tester who uses or has tried TestFlight, share your experience and comments with us below.
