What you need to know
- Apple's Craig Federighi sat down with Popular Mechanics to talk about Scribble.
- The feature converts handwriting into text on iPadOS 14.
- The senior VP talks about how it was created and how it's able to run on device.
In an interview with Popular Mechanics, Craig Federighi, Apple's senior vice president of software engineering, sat down to talk about how Scribble, the company's newest feature for the Apple Pencil and iPad in iPadOS 14, came to be.
When it came to understanding how people wrote, Apple literally had tons of people physically write. Doing so allowed the company to understand all the variations that go into how someone writes fast, slow, at an angle, and more.
"When it comes to understanding (handwriting) strokes, we do data-gathering. We find people all over the world, and have them write things," says Craig Federighi, senior vice president of software engineering at Apple. "We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation." That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple's tech, static examples weren't enough. They needed to see the strokes that formed each letter. "If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written."
Apple's machine learning, which understands not only what you are writing but what you may write next, is calculating everything on the iPad itself. Federighi says that this is required in order to protect user privacy and perform at the speed that people need.
That dynamic understanding of how people write means Apple's software can reliably know what you're writing as you're writing it, but combined with data on a language's syntax, the iPad can also predict what stroke or character or word you'll write next. The massive amount of statistical calculations needed to do this are happening on the iPad itself, rather than at a data center. "It's gotta be happening in real time, right now, on the device that you're holding," Federighi says. "Which means that the computational power of the device has to be such that it can do that level of processing locally."
The Scribble feature, which allows users to convert writing into text with an Apple Pencil, launched with iPadOS 14.
