Platforms State of the Union: Important highlights from the event

Apple doubles down on user privacy and security

This year's Platforms State of the Union kicked off with a look at Apple's commitment to user privacy and security. The company reviewed its data minimization techniques, reminded developers of its choice to use on-device intelligence for things like search and Photos, detailed its data protection techniques (including Face ID, automatic on-device encryption, App Transport Security, CloudKit, etc.), revealed that two-thirds of its users have Apple ID accounts protected by two-factor authentication, reviewed its transparency and control policies, and — most importantly — gave a closer look at iOS 12's new password management features.

Along with the option to create, store, and autofill strong and unique passwords that sync across devices using iCloud, passwords will not only autofill in Safari, they'll also be available to users in third-party apps. Apple also showed off a nifty new feature involving two-factor authentication. Instead of hopping into Messages to copy and paste a two-factor authentication code, iOS will automatically reveal the authentication code in the QuickType bar. What's more, third-party password managers will be able to make use of iOS's new password autofill extension, so you needn't bounce back and forth between apps to gain access to your accounts. These new features work on iOS, macOS, and tvOS.

Xcode 10 gets several updates and improvements

Xcode 10 got a lot of attention at this year's State of the Union. Between loads of stability and performance improvements, code editing features, and debugging updates, there's quite a bit to love about Xcode 10.

Xcode 10 for macOS Mojave quite obviously supports the operating system's new Dark Mode, but along with offering that support, it also makes it much easier for developers to add Dark Mode support to their own macOS apps. Developers can easily add dark variants to their app's asset catalog and specify dark variants for UI elements. When debugging an app, developers will be able to switch between Light and Dark modes to get a better idea of how their app will look in both environments.

You can now create custom packages for use within Instruments to eek out all the more data from your logs. When you run an app within Instruments, you simply open up a custom package and it'll run within Instruments, logging anything and everything you might need to see in order to improve your app's performance.

Along with performance improvements to Interface Builder (40% faster document opening and 30% faster canvas performance), Apple introduced a flexible new Library. Instead of remaining stuck within Inspectors on either side of your project, the new Library is separate, serving as a floating panel that can be positioned anywhere on your screen and resized as needed.

There's also a new class for AppKit called NSGridView. NSGridView is a container that lets you lay out views like you would in Numbers or another spreadsheet app. You can quickly align your views in a flexible grid without much fuss.

There were quite a few updates for code editing in Xcode 10 and many of them were met with emphatic applause. For one, Xcode 10 brings back the code folding ribbon and improves upon code folding support. The code editor supports overscroll and multi-cursor editing which, as you might imagine, lets you make quick changes to multiple portions of code. Code completion is getting better, too. It now provides more targeted results and limits completions to only the high-confidence results.

Xcode 10 also includes a new source control change bar that gives developers a convenient way to track changes and address conflicts within their code. It also brings support for Bitbucket Cloud, Bitbucket Server, and GitLab.

Given that this year's keynote focused pretty heavily on stability improvements, it's not surprise the State of the Union focused on it, as well. Debugging is getting better thanks to enhanced memory debug tools with an all-new compact layout and improved LLDB performance. Summoning quite the cheer, Apple has improved the speed at which debug symbols are downloaded. It'll take "seconds, not minutes," to download symbols. As for testing, Apple announced new testing options that will automatically include new tests, randomize execution of said tests, and execute tests in parallel. It's a more efficient, more thorough way to perform testing.

The new build system, which is completely written in Swift, is now on for all projects. It uses 20% less memory, results in two-times faster rebuilds, and reduces code size by up to 30%.

Swift 5 is fast approaching. When it hits in 2019, Apple says it will be delivered as part of the OS, resulting in smaller downloads and faster launches. It's a huge performance improvement waiting to happen!

macOS gets safer and will soon play nice with iOS

Between Dark Mode, improvements to Gatekeeper, and iOS apps on the Mac, there was a lot of macOS for Apple to talk about at this year's State of the Union. Before we get there, though, we have to start things off with the beginning of the end: macOS Mojave is the last macOS release that will offer any support of 32-bit apps. Apple said 32-bit frameworks will also be deprecated.

OK on to the new stuff. Firstly, Gatekeeper is getting three improvements:

1. Extending User Consent: Much like iOS, macOS will now require apps to receive user consent before accessing most user data. That includes things like location, contacts, photos, Time Machine backups, Safari data, Reminders, and more.
2. Runtime protections: SIP protection, strong code validation, and protection from code injection combine to offer better runtime protections for third-party apps.
3. Notarized apps: This new protection is an extension of the developer ID program. Non-Mac-App-Store apps will be notarized by Apple and offer better protections for the end user. This will help block malware before it gets distributed and help revoke app versions that are compromised. Apple was very clear that this is not app review, it's simply used to analyze apps for security purposes. Although this isn't currently a factor, future versions of macOS will require all apps to be notarized before they can be installed.

Dark Mode for macOS is going to take a little work. Apple says it won't be an automatic opt-in feature, because it requires design decisions to produce the best experience for Dark Mode on the Mac. Apple's Dark Mode APIs, however, make it simple to add support — if you're following best practices for macOS, Apple says it could take as little as one day of work to add support.

This is, perhaps, one of the coolest features buried in the announcements. Quick Actions let you add shortcuts to Finder's Preview pane. Different files will surface different Quick Actions based on context and best of all, end users can create Quick Actions using Automator, AppleScript, and more.

Although we won't see this until 2019, developers can start getting ready for porting iOS apps to macOS. Apple is currently experimenting with News, Stocks, Voice Memos, and the Home app for iOS, which are all available in macOS Mojave. Although iOS and macOS are fundamentally different, Apple says they're built atop the same environments: Media, Core OS, and Core Services. They've drifted, but Apple is working on bringing back that connection (hence the delay).

In the implementation mouse events are mapped to UI events, so developers will be able to use much of the same functionality despite the change in platform.

Machine Learning was the star of the show

Machine Learning got a lot of attention at the State of the Union this year. Apple has made several improvements to vision and natural language processing, improved Core ML 2, and introduced Create ML.

• Vision: New APIs support object detection, face detection, facial landmark detection (like eyes, nose, and mouth), barcode detection, and people segmentation (gives you the ability to remove a person from a photo or switch out the background of a photo).
• Natural language: A new Swift-focused API can be used to identify the language of a particular string of text, tokenize the sentence (split it up into pieces), convert the string into its parts of speech (noun, adjective, verb, etc.), and recognize named entities (like map locations). What's more, it works in several languages — not just English.

Core ML 2 introduces smaller, faster, customizable models to improve performance and reduce app size. A new batch API makes use of the CPU and GPU to get better performance and quantization helps reduce the size of the model. Essentially, Core ML 2 focuses on making things run better, faster, and smaller.

Create ML is very cool. This machine learning training framework supports feature-level training (image classification and natural language processing), algorithms, and data processing. You can customize the natural language model and perform text classification and domain analysis.

Apple's image classification training framework was demoed at the State of the Union and it was incredible! An open-source model took up nearly 100 MB of data but using a new feature called Transfer Learning, developers can save time and reduce their app's size — the final model was mere kilobytes of data. Transfer Learning bundles in Apple's image classification model, allowing you to augment it with your own tests. By doing so, you end up saving a bunch of space and, in many cases, get a more accurate model.

Augmented reality

It got a lot of attention during the keynote and it got a lot of attention during the State of the Union — it's safe to say Apple is all-in on augmented reality. Created in cooperation with Adobe and Pixar, USDZ is a new file format that makes sharing 3D assets for augmented reality simpler and more efficient. App developers can insert USDZ files in their third-party apps, web developers can embed them on their sites, and folks can pass them around to their friends. Adobe also announced a new tool that it currently calls Project Aero. Leveraging Project Aero, designers and developers can create and share USDZ files for apps and the web.

ARKit 2 introduces some pretty powerful new tracking functionality. It's better at tracking your face, meaning it'll map to your gaze and your tongue. It also leverages Machine Learning for a pretty incredible environment texturing feature. When you place a reflective AR object into a scene, it will reflect what's shown in the scene (a red cup next to a metallic AR object will leave a red reflection in the AR object) and it will reflect what's not shown in the scene. Apple created a Machine Learning model by feeding in a bunch of common environments. ARKit 2 uses this to create textures on the object based on what it thinks exists outside of the scene. SO. COOL. It also gets a lot better at 2D image detection and tracking. In the demo, a 3D object (a motorcycle) was placed atop a real-world, 2D image of a motorcycle. When the person lifted up the 2D image, the 3D object moved with it, both up and down as well as left and right. As was mentioned during the keynote, ARKit 2 supports multiplayer augmented reality experiences and support for persistent experiences based on mapped locations.

What's got you excited?

There were plenty of other, smaller mentions at the Platforms State of the Union, but it was jam-packed with lots of information. Did we miss anything that you found particularly awesome? Sound off in the comments with your picks!