Guests & Links
- macOS High Sierra 'root' security bug
- Why Gets You Root
- About the security content of Security Update 2017-001
- Repair file sharing after Security Update 2017-001 for macOS High Sierra 10.13.1
- Jonathan Deutsch: Twitter
- Tumult
- Hype 3
- Kendo
Sponsors:
- Mint Mobile: Voice, data, and text for less. Get free first-class shipping with code VTFREESHIP.
- Thrifter.com: All the best deals from Amazon, Best Buy, and more, fussily curated and constantly updated.
- Interested in sponsoring VECTOR? Contact sponsor@mobilenations.com
Transcript
[music]
Rene Ritchie: Joining me today, we have Jonathan Deutsch. Jonathan, if people haven't, somehow, heard the "Debug" episode that you were on, or they haven't been following your career the way I have, or maybe heard you talk at Çingleton the way I have, with, I believe you had your kendo shinai with you at the time. [laughs] Could you give us a quick summary of your background?
Jonathan Deutsch: Yeah. Rarely does a talk go by where I don't mention kendo, which is Japanese fencing.
[laughter]
Jonathan Deutsch: Which I am still practicing to this day.
Rene: Awesome.
Jonathan: I am the founder and the developer of an application called Tumult Hype, and it is HTML 5 animation software. It lets graphic designers make animated content for the Web.
Rene: You saved us from Flash, basically.
Jonathan: Pretty much.
Rene: Prior to that, you did some Mac OS shenanigans at Apple?
Jonathan: Yeah, prior to that, I was an engineering manager at Apple, so I worked on mail for Mac OS X, and I worked on software updates for Mac OS X, as well.
Rene: You got into this whole crazy computer biz just because you figured out you could type things into a keyboard and it would make magic?
Jonathan: It is pretty amazing how you don't need much to create something of value that helps other people. The first time I made a little JavaScript that would help people improve their jobs, and I saw people using it, I was like, "Yep, that's exactly what I want to do."
Rene: [laughs] And then you saved the Internet from Flash, you know, cause and effect.
I wanted to chat with you, because when we did chat on Debug, you had a lot of really interesting perspectives. You have worked in the largest company in the world on one of the most important pieces of software in the world, and you've also worked as an indie on software that is just as important to the people who wanted to deploy rich animation on the Web.
That gives you, I think, a really unique perspective on what it's like to ship product at both large companies, at enormous scale, but also to take personal responsibility for that on your own scale.
Jonathan: I think one of the interesting perspectives is being at Apple, while there's a lot of responsibility on every individual, you're still a little bit of a piece of a machine, and so there's lots of checks and balances.
You're really responsible for your piece. Seeing a little bit of the perspective above and a little bit of the perspective kind of below in the org chart, per se, but you're responsible for your piece. Whereas as a independent developer, you really have to make sure you own the whole piece, and you see everything from the little details to the big details.
That, being in the after working at Apple, now I feel like I have this interesting perspective on Apple, how decisions were made, and how the company was structured as well.
Rene: Right now, as we record this, the updated news on the root, < > exploit, that has been filling the news cycle for the last two days. I think it's beyond the bug itself, which bugs should never happen, but bugs happen. It's led to a lot of discussions, and a lot of them are classic or repetitive. We hear them over and over again any time a bug ships from any vendor.
I thought it would be super interesting, because you had that perspective, to talk about them with you. I guess the first place to start is that people are always shocked -- people who even aren't developers -- are always shocked when there are bugs in code.
Jonathan: I think the thing that would shock the most is not that there are bugs in code, but how many more bugs there are that they don't even see. There are literally millions of bugs on Mac OS and iOS, and oftentimes the release notes will even say, "We fixed maybe 100 bugs," but in reality, there was probably over 1,000 bugs fixed in any given update.
I would first say the magnitude is an order off on how buggy software is, and at the same time, I think there's the interesting perspective that manual QA can't catch everything. You like to think that an organization takes responsibility, and "the buck stops here," and no bugs should go, but the reality is millions of bugs, in fact, do get through. Some are not very important. Some are extremely important, and some are critical security issues like the root access bug.
Rene: Prior to when I worked in media, I worked in product marketing, and it was for a software company. We had developers, and we had QA engineers, and they ran all sorts of tests, automation tests, regression tests, performance tests, but inevitably, the product would ship, and it was a database product. There's very few bugs that are as -- what's the right word? Calamitous to an end user as data loss or data destruction.
No matter how much QA you did, or how much you invested in it, there was nothing that equaled dozens, hundreds, thousands, and when you get to Apple, Microsoft, or Google scale, millions, tens of millions, hundreds of millions, approaching billions at Google, Facebook, Apple scale of people hitting your code.
Jonathan: Yeah. I think for any piece of software, you have to consider, as a company, what's the right strategy to test the software. If it's something like an operating system, you have an extremely difficult test matrix to deal with, because you have lots of interactions with other hardware, with other software, and so that really means that to develop a wide matrix, that can't be all handled internally.
If you have different software, let's say a video game or apps that only read and write their own format, that's something that probably could be handled really well by internal QA, but when you deal with so many users and so many configurations on something like an operating system, it is literally impossible to test every bit of the matrix.
Rene: This is shifting gears a little bit. Is that why you do product beta? Apple's not traditionally known for being open, but even they have started doing public betas for iOS and Mac OS over the years.
Jonathan: Definitely there is, I think, a trend in how you roll out software to users, and so the notion of getting more users and more configurations will actually only make the software better.
There's a different part on how to collect feedback that we can discuss later, but I would say Apple originally started only with the developer seating program way back when to have developers do testing, and I think it was maybe about five years ago, they started doing public betas.
I think this was probably in response to known issues on Apple's quality, perhaps, not being where they wanted it to be, which I think is always good when you see a problem and you're proactive addressing it. I think you also have to look at how an operating system is developed, and how Apple releases their operating system. Apple hasn't been able to take advantage of a lot of the kind of newer strategies towards testing software.
If you look at a website like Facebook, they will very slowly roll out features to certain percentages of their audience. They can do this rollout where they can give a feature to maybe some small percent. If it goes well, they can do it to a larger and larger percentage.
With how Apple releases software, they can't really do it the same way. You can say maybe they should be releasing software differently, which I think is a fair assessment.
You can get a little more of that granularity by first releasing just to developers, then doing the public beta, and then eventually, once those have gone through, do a full GM release. It gives, at least Apple, more users, more beta testers, and better granularity as far as a rollout's concerned.
Rene: When you have, for example Mac OS High Sierra, it went through the beta period. In hindsight, we've gone back and seen that. Someone made a video a week ago, and someone put this on the Apple developer forum two weeks ago.
You never know who may have found this earlier, and decided to keep it to themselves. You go through these processes, but now, yesterday, three weeks from now, six months from now -- I'm not going to make any core audio jokes -- but you find these things all the time.
Jonathan: There's always going to be something that gets missed that's important. I think, going back a couple of years, there was a bug in OpenSSL where it was just a very silly programming mistake based on not using curly brackets, I think, was my recollection. These types of things, unfortunately happen, because code is written by humans, and humans make mistakes.
Rene: There's been stage fright on Android. There's been Windows XP, famously. Microsoft learned huge lessons from that. There used to be this joke that only NASA could afford to write perfect code, but then they got feet and meters mixed up, [laughs] and lost a spaceship.
Jonathan: I'd be curious to do a cost analysis on what some of these security issues could be costing versus NASA losing a Martian probe.
Rene: [laughs] A couple of the things that come up whenever these things happen, and regardless of the company...I don't want to take the focus off Apple, because this was, again, a horrendous bug.
One of the things you hear is these companies are rich. Apple is the richest company in the world. Why can't they just throw more programmers at it? Why can't they throw more QA engineers at it?
Jonathan: From my perspective, there is a few things at play, and I think, ultimately, every organization has different curves, where you start to throw more and more people, and you get less and less work done. There is organizational issues that have to do with how humans are even being managed.
There's also software issues that have a very similar curve, where you can throw more eyes at a problem, but it doesn't necessarily mean that more things will be caught. An issue like this root password issue, it kind of almost would require a happy accident or someone being very clever.
You make some arguments that you should always, in the author dialog, you should test invalid input such as "empty," which I think is also completely valid. I think there's different software where the edge case, that edge happens at some different level where you may go from 10 people to 100 people, but you still haven't really hit that edge, where you hit it much earlier.
Even adding so many people, you might not really be getting any bang for your buck as far as adding those people to do, at least, manual QA on a problem like this.
Rene: Then you have the mythical man-month, where as you add people, you add complexity and managerial overhead, and the inability...The same way massive parallelism in computing, it took a long time to figure out massive parallelism in humans, is an even bigger problem.
[laughter]
Jonathan: I think the other thing is, too, when you have an organization, and you have so many people, communication often becomes an issue, where an issue like this might even have been reported, but there wasn't enough bandwidth or a good enough feedback loop that it went to the right people at the right time and you were able to ship with it.
When you add more people, you add a lot of these secondary effects such as communication overhead, and sometimes things get missed even if they are known and reported. With the root access bug, for example, this is kind of outside the notion of inside of Apple, but outside of Apple it was reported.
It was reported on Apple's forums, and Apple does monitor those forums. I don't think this one was one that they were expecting specific feedback on, but if they were looking, they would have seen this, and hopefully someone would have said this was a legitimate issue.
You have to consider how the entire feedback loop works as well, and so with someone actually monitoring these forums, and if someone was monitoring them, did they think to report this? If it was reported, did it go through radar with a high enough priority, or did it get triaged into some bucket where people haven't even looked at it yet?
There are so many steps along the way, and the more people you add to an organization, the more process and steps you also have to add. Each of those steps is something where something can slip through the cracks.
Rene: It's interesting. Every organization does things entirely differently, but Apple, to the best of my recollection, uses a scale. For example, P1, I believe is...I don't know if there's a P0. I think P1 is the highest urgency of bugs, and it goes down to 2 and 3, and a system of screeners who will look at their radars and sort of make a judgment call on them before they get escalated or passed through. Am I phrasing that right?
Jonathan: Yeah. At least when I was there, there were four priorities of bugs, and certain teams had different ways to prioritize within those when you knew that scale. It always requires a human, or even a group of humans, looking at the bugs to determine what is the priority.
Ultimately, someone's going to read this issue, and if the issue report happens to be worded incorrectly, or difficult to parse, or someone just happens to hit the wrong P1, P2 on the pop-up menu where it exists, then it can get filed incorrectly and not looked at and filtered appropriately.
Rene: It's interesting to me, too, because the other issue is that when bugs get filed early on in the development process, there's a lot of time for people to look at them and sort of fix things that aren't show stoppers, that are annoying, for example, or frustrating, or inelegant, but as you get closer and closer to release, that window gets narrower and narrower, and you start to be constrained.
Again, I think people have trouble with that, because you should be able to do everything. I think regardless of your company's size, as your ship date approaches, unless you make the conscious decision to punt the ship date, you have to focus on the most critical and urgent bugs in order to get the product out the door.
Jonathan: There is a pretty famous triangle. I feel like so many different categories have a triangle of, "Here are three options. Choose two," or three that some taste, speed, and price. For software, it's quality, features, and schedule, and so you've got to choose two of quality, features, or the time in which it's released. If you decide you're going to have hard deadlines, then that means either quality or features are going to hurt from that notion of having a schedule.
Rene: That's one of the things that people also push back on, and I don't know if it's right or wrong. It certainly is interesting, is that Apple has increasingly gone to yearly release schedules. I know indie developers have things like that, too. We've spoken to other developers about, "You have to have enough features to warrant it being an update, otherwise people aren't going to feel compelled to upgrade."
Regardless of your size, there's always tensions around what you release as a product, but being on a schedule when you have dependencies, for example, of the iPhone, can copy/paste a clipboard to a Mac, that Mac update has to be available that lets you copy and paste it. Otherwise, that function is essentially broken, and you sort of get yourself on those rhythms.
Jonathan: Apple's in a very interesting situation right now with the iPhone. I feel like the competitive landscape of the iPhone is really what may be drives that as being a yearly schedule, not to mention hardware and software updates related to the hardware.
I feel like maybe it's the competition that makes Apple think iOS needs to be updated once a year, and then there are so many complimentary features in Mac OS nowadays that it makes some degree of sense for them to be on the same schedule.
I think this also goes back to the notion of how other software companies do it, and if you look at a lot of modern Web-based companies, they don't really do a yearly release. They tend to do a feature release, that when a feature is ready, it goes out, and it goes out to a smaller set of users. Apple doesn't have the flexibility with how they build and ship software to really do that model.
Rene: It's interesting, when you look across the landscape, like Chrome OS, Chromium, and Chrome in general feel like they're continuously trickle updated, where Android has a yearly release cycle. They go through the letters of the alphabet, and they're desserts, pretty much on a once-a-year cadence.
Microsoft's essentially made Windows a service where there are components. I guess Google Android has both the Google Play layer and the Android kernel layer, but Microsoft's essentially made it a service, where they're trying to do half and half, maybe, half software, half Web-based updates, and it is updated on a more continuous basis.
All of these have their pros and cons, but they're interesting ways of tackling the same problem.
Jonathan: Yeah.
[laughter]
Jonathan: I think that's also a matter of "the grass is always greener on the other side" when you're doing software development, that as someone who does do specific releases, and for hype, we do paid updates, there is definitely the "We need to have enough features to make the update worthwhile," and so for me, personally, the schedule is a very artificial thing. It's really just a personal deadline, but there aren't too many factors influencing my schedule as an independent developer.
When Apple makes specific promises or reveals software earlier, and things need to all strike at the same time, that makes it more difficult.
Rene: That gets interesting, too, because there's this thing that happens where -- and I don't know if this is a human thing. It feels like a human thing to me, where, at least as long as I've been covering technology, every release is the worst one ever, and it's possible that it really is.
It's possible that it really is, that as complexity and dependencies grow, and as product lines expand, and as the realities of, again, functional organizations versus the types of organizations set it, that it does put real pressure on these releases.
I think it's also possible, because every time there's a problem, and every time you see something, I go back and I look, like, "What was last year like? What was the year before like? What was the year before that like?" Almost all the time, you see the same things, like, "This is the worst release ever."
I wonder if there's something deep in the human psyche that makes us forget past pain, but acutely feel present pain. There's that joke that if you remembered going through childbirth, we would never have any children. You know this from kendo. In martial arts matches, if you remember the pain of your previous match, you would never want to do the next one, but it sort of fades away, and you get eager to do it again.
Jonathan: [laughs] I do think humans fear change to some degree. I think that's true, but I think also we might be comparing against kind of the wrong memory. If you look at the best version of Mac OS ever, undisputedly, it is 10.6.8. I don't think that's a controversial opinion.
Rene: Why 10.6.8? Because you shipped it?
[laughter]
Jonathan: Yeah, funny how I left Apple shortly thereafter.
No, 10.6.8 was Snow Leopard. This was before iOS had really crept into Mac OS, and I think if you think of Snow Leopard, it was similar to High Sierra, where the notion of the update was to make the last one better, to fix bugs, to increase performance, to really have a refinement. That was the idea behind 10.6 Snow Leopard.
I think 10.5 had a legitimate number of issues, and I think it was a good call to do 10.6 in that manner, but very specifically, I said 10.6.8, 10.6 had massive issues when it shipped, and when you think about the fact that 10.6.8 was a great update, you had to get through 10.6.1, 2, 3, 4, all the way to 8, and that was a long period of time. Apple wasn't on the yearly release schedule.
I think 10.6.8 probably went out with two years of refinement over 10.6, which was, I think, another two years of refinement over the 10.5 update. The 10.6.8 had been begging to get to that point for almost four years, whereas now I think Apple has a similar philosophy on what it means to do a major operating system, but the schedule's a lot shorter, so no operating system goes through that period of incremental bug fix updates to get to that quality point, because they are on the one-year release schedule.
Rene: I think that's fair. Snow Leopard was such an interesting release for me, and again, I don't want to get off on a tangent, but it didn't have a code name. There's no y-name attached to it. It's just Snow Leopard. It had new features, like Grand Central and Exchange, but you can't market Grand Central Exchange so it's smarter to this market, no new features.
It sort of set this precedent where all the time, I'm sure we'll hear it as soon as the smoke settles here, that Apple needs a Snow Leopard moment, despite them basically having a Snow Leopard moment with High Sierra to begin with.
Jonathan: I think High Sierra is one of those interesting cases where it was marketed as an improvement, just by its name alone, but I think people are very hard pressed to find what's unfortunately improved, and the root access bug is not really helping its reputation.
Rene: Oh, god. I'm so tangenting here. It seems like Apple's going through this slow metamorphis where they went through a massive change when they went from the old Mac OS to the next-based technologies, and OS X.
It doesn't look like in the immediate future, they're going to do another reboot, but they are, step by step, introducing Swift, introducing APFS. They tried to introduce Discovery (D). It didn't work so well, rolled that back, but they are, bit by bit, replacing all these aging layers or limited layers, with things that will allow them to project the technology further forward.
Jonathan: I think a lot of the direction's generally a good direction. I think when you look at an operating system, from my perspective, you can have an operating system that's different levels.
There's applications at the very top like Mail, but then there's the low-level core fundamental technologies, and those are things that you want to be extremely stable, because they're on a very foundational layer. If you get those wrong, everything above in the stack will be not stable.
At the same time, you do need to change to keep up with the times, because if you don't improve down there, then also the higher level items, the applications on top, can't improve as well and are limited. It's always this juggling act of introducing change versus not introducing change, and trying to be stable at the low levels versus offering new capabilities.
Rene: I think that's absolutely true, and when you look at the...One of the things that interests me as well, because I think that essentially many of the problems...I think, arguably, Apple's biggest problem right now is a perception issue. It doesn't really matter how buggy or how solid the software is this year relative to previous years.
If the narrative becomes that it is really bad, then the truth is that it's really bad, because that's the customer sentiment, and its customer...What's the right way to phrase this? It's like a currency, that it's really hard to earn and incredibly easy to spend, and if you have the good faith of your customer base, you can do a lot of things, but if that erodes, then everything becomes more difficult.
It's that old cliché about "It's easier to keep a customer than to get a new customer." I think that's one of the things you have to be careful about, but also, I believe it was Phil Schiller -- it might have been Craig Federighi, but it was one of the two, when they were on John Gruber's talk show after WWDC, not this year, but the year before -- where they were talking about Marco Arment's functional high ground piece.
He was talking about the quality of Apple's software slowly slipping, in his opinion, and one of the things I think that they mentioned -- and I'm just doing this out of memory, so I might get this wrong, so please bear with me if I do -- was that they were monitoring certain things. For example, they were looking at crash numbers, and crash numbers were way down, and those are fatal problems.
The number of small things -- like, I think Craig ultimately called it death by 1,000 cuts -- the number of crashes were down, but the number of annoyances, at least perceptively, were up, and when taken as a whole, that ended up bothering people as much, if not more, than just an app crashing every once in a while.
Jonathan: I think you can get lulled into a sense that if you're telemetry is improving, that you think your product is improving, but you have to pay attention to both the automated telemetry of the things like the crash traces, the spin logs, the exceptions, the errors, but also the user sentiment and what the users are actually hitting.
Some, like data loss, is clearly a priority one issue, but when users just aren't happy using the product, that's a very significant problem. You can tell me what you think of this, and it's a little bit silly, but my definition of software that people love is quite simply software that people don't hate.
What I mean by that is we've all hit some sort of issue where software didn't work in a manner that frustrated us, and it's usually some defines of expectations, or it's something random that you don't expect. That causes you to get out of your flow, take pause, scratch your head, and get frustrated like you're getting frustrated at another human being.
Those frustrations really add up, so even if the software does a great job in certain ways, you've got this frustration. You just don't like the software anymore. If you reduce this level of frustration, even if the software does less, if it's frustrating less, I think people will love it more than if the software does a lot more, but causes frustration.
Rene: I think you nailed it there. I think as you increase the surface area of software, and as operating systems mature -- and we certainly see this in iOS, because it's gone from 0-10, literally, in 10 years, 11 now. As the surface area increases, the opportunity to discover bugs increases.
When it does very few things, you can focus on those things, and you can polish those things, but as it does more and more, there's so much surface area to cover, and the likelihood of you encountering something that was missed increases. There's just more opportunity for it.
Jonathan: I think that's always a question. You may be adding specific new features that customers hopefully will love, but by adding so many more features, if you don't do a good job, you can very significantly hamper and take away from that experience.
Rene: I think there's also something to it, like for example, you make Hype, but if you suddenly decided that Tumult was going to make three products, even if you hired three more people to do it, there's a level of complexity there that increases.
I think we've seen this a lot, too, and this is, again, another trope, that this would never happen under Steve Jobs, regardless of the fact that MobileMe happened under Steve Jobs, Antennagate happened under Steve Jobs.
I forget which version it was, but there was a bug so bad Apple had to push, had to figure out a way to force an update onto Springboard to update the phone because of the bug. [laughs]
There was so much that went wrong, that we either forget or didn't know about under Steve Jobs, or again, if Scott Forstall was still there. Back then, there was Mac and iPod, and then iPhone was starting. Now there's Mac, iPhone, Watch, TV, and special projects, and they're not all lumped together.
Craig Federighi, yes, they merged iOS and Mac OS [inaudible 34:45] , but Kevin Lynch runs Watch, and Apple TV is still under Eddie Cue. Special projects are under various. Bob Mansfield has some, and other people have others. Dan Riccio has some. There are different organizations tackling these issues.
I think the level of complexity, when you have all of those arrows that have to hit the same target at the same time means that the company isn't what it was before, can't be what it was before, and the things that worked back then, you can't simply take them and slap them back on and expect them to work now.
Jonathan: Are you trying to make have a choice between my Apple Watch and this root access bug? Because that might be a difficult choice to make.
Rene: No. Maybe. Maybe that's it. I guess this all comes down to this core thing where you're damned if you don't, because if Apple has an event, and they don't have 300 new amazing features, the event was boring and Apple's not innovating any more. They're falling behind. That doomed narrative comes on so hard.
If there is an event where Apple introduces a new product and a bunch of new features, then Apple is losing focus, and they're not working on the foundation. They're abandoning what came before. I think that is a really hard balancing act.
Jonathan: I think it's a hard balancing act for most people, but I think at the same time, Apple gets to define their own destiny and how they want to be represented, not by news organizations, but by their customers.
That, to me, was always more powerful, that Apple might do things, and they will think differently about how they want to be represented and how they want to be thought of. If it means that...
There has always been shots at Apple being a beleaguered company on the verge of death. Apple always had to put up with that, and usually they ignored it and kept doing what they thought they were doing best, and grew a following that way.
Rene: Here's my chance to ask you something. This is a pet theory I have, and you can tell me if you think it has any merit, or if it's just zany. I believe that any company sufficiently large is indistinguishable from evil to a certain percentage of their user base simply because they can never be everything to every user at every occasion.
If you are so passionate about the Mac, the mere fact that Apple has grown to do iPhone, Watch, TV, and other things means that they're not putting all their attention on the Mac, and that become aggravating, frustrating, and maybe even alienating to you as someone who grew up loving the Mac. Or if you love the iPhone, now that they're going the Apple Watch, or something else.
There is such a high chance that they're not working on whatever matters to you the most that it starts to create negative sentiment.
Jonathan: I think that's absolutely true, and you can see that in other industries, like musicians, for example, I think, always get stuck in this. All the fans want an album that was like their past album, but if you give them one that's too similar, then it won't be interesting enough to hold their interests, and if you give them something that's different and maybe more up your alley of what you want, as a musician, to experiment with, then you've lost your fans. Maybe you'll get new fans, at that point.
I would definitely agree that that's a situation.
Rene: It's a trick with movie sequels, too. You want the same, but different. Again, by any means, I don't want to defend or apologize, or in any way make up for it. This kind of bug should never ever ship. One of the things, though, that I believe is that every company makes mistakes. When you're dealing with sophisticated, complicated software, every company makes mistakes.
There's two things I look for. One is, "Was it malicious? Did you do something that was deliberately against the interests of your customers?" I don't mean this in sort of like negligence. You can make an absolute case that negligence is malicious, or that incompetence sufficiently repeated is maliciousness.
There have been other vendors -- and you can accuse me of whatever, false equivalencies, or, "What about other company's things?" whatever.
There have been companies that have put root kits on their computers, that have put man-in-the-middle attacks on their computers. There have been companies that have acted in direct opposition to the best interests of their customers, and that, I think, is inexcusable.
I think when an accident happens -- it could be a battery that burns, or it could be root access, or it could be anything -- those things happen, and all you can do is judge a company by their response to it. If they ignore it, if they pretend it doesn't exist, it would take forever to patch it, that is bad. Then the accident becomes malicious because of the failure to act upon it.
If the company responds to it well, fully with humility and with competence, then I think that's just a process that we go through.
Jonathan: I think if you look at Apple's track record on issues of security, they generally are very proactive on security versus reactive, and proactive is where you want to be. I would say not always, and Apple has definitely improved over time to be in that state, but I think they've become more and more aware of all the different vectors of attack, and have been working to improve that. I would say definitely Apple's putting users as number one, and this was clearly an accident.
Rene: Yeah. I don't think anybody on the any of the security teams or the core layer, I don't think anybody slept last night. That's my guess.
[laughter]
[crosstalk]
Jonathan: ...people on the file sharing team didn't sleep, either.
Rene: [laughs] I'll make sure that that's included in the show notes. You can pseudo. Let's say you can go to Terminal, and you can repair that yourself.
Again, these things just shouldn't happen, but there are so many things that you hit from problems in the user interface to problems with all these different services. Again, I put it down to complexity, but I'm not sure how you solve for it.
Some people say your organization has to change, that you have to go from a functional organization to something else, that you simply can't scale functionality. Other people say that Apple can't keep expanding. They have to settle down to some core competencies.
Meanwhile, there's rumors they're starting a movie...Not even rumors. They're spending billions of dollars on movie content, or video content now. Everybody has a theory on what fixes this, but I don't think it's that easy.
I don't think when you get to Apple scale, Microsoft scale, or Google scale that solving these problems are at all easy, and I think that's why we've seen IBM lose relevance, and why we've seen Microsoft sort of teetering on the brink of losing relevance, and why you see Facebook.
They've grown through acquisitions, but they've pretty much left Instagram, WhatsApp, and Oculus are pretty much still independent teams. I think these are problems that you wrestle with as you scale, and as the dynamics of your leadership change.
Jonathan: I would say also, it's not like this has been a string of security issues week after week after week. We're looking at one thing on the face right now, but I don't think it really can imply too much organizationally about what needs to change, or what organizationally was an issue.
There's clearly things organizations can do to reduce probabilities of events like this from happening, whether it's having more security code reviews, educating developers on security issues, having more security testers.
A lot of those do also have tradeoffs that we mentioned before, and we don't know that it was a lack of that that caused this one particular issue. There's always going to be some probability of some issue getting out and getting beyond an organization, unfortunately.
Rene: Bringing this back down to scale again, you work on Hype. [laughs] How small is your team right now?
Jonathan: I'm doing most of the development, and sometimes I'll have someone do some engineering, or do contract work.
Rene: All of that falls on you at that point, then.
Jonathan: Yeah. There is also someone doing support, so that's another part, kind of, of the entire feedback cycle as well, but yeah, pretty much, it's falling on me, and regardless, the buck does stop with me for everyone's code. I think as the person owning the organization, I also need to own how the application runs, and what instructions it's running.
Rene: What does it feel like to you when you go from the massive Apple scale down to the indie scale, when you encounter bugs or your users encounter bugs?
Jonathan: [sighs] You take it personally, and it hurts a lot more, because you know that just because someone's hit a bug, you know one, it could be your fault entirely, and two, you may not be able to fix it.
You have relationships with the people using your software, because oftentimes I'm the one reading the feedback. I'm the one who goes, "Oh, I can't believe I did this," and then also, "Well, there are so many more issues, and if I was to be disciplined, I wouldn't fix this issue on it."
That can hurt. It can also be extremely rewarding, where someone does report an issue, and you say, "That's silly." You fix it, and then two hours later, you say, "Why don't try out this beta?" and it solves it for them, which is one of the most incredible feelings in the world, that you can have that type of relationship with people, and you're that close to the code and the users.
Rene: It's this interesting dichotomy, because from the outside, as someone who does not code, but will use a software, any problem seems like it should be easily fixed when you're not the one in charge of fixing it.
Jonathan: [laughs]
Rene: It's like, "These bugs should just never happen," and that's my attitude. I hit them, too, and it's frustrating. I'm like, "Why did this even ship?" But on the other side, you have what you just mentioned, and that is whether you're an individual contributor with a specific assignment, or it's your responsibility for the entire app or company, that generally you hit physical limits. You want to do more than you are actually able to do.
Jonathan: Correct, and I think there's, for me, quality is a really important aspect of how I like to run my business, and so there is a lot of process that I put in place, especially when we had a few more employees, around the notion of quality, and beta feedback is definitely one of the biggest things.
Beta users are like the best users in the world, that they take time out of their day to report issues to you. I've often felt that the feedback loop on beta testers was like this very precious gem to help.
I should say maybe it's more like a plant, to grow and foster, where if you treat your beta testers really well, you get so much back in return as far as not only bug reports, but as them being some of your biggest evangelists for the product as well.
I've often thought that making it very easy to send us good, accurate feedback, collecting the feedback, acting on it, and then having that open channel and communicating, letting them know how valuable their feedback was, and kind of closing the loop has been very important to how I do development, just from even a QA standpoint.
I can't test everything. Hype is one of those very large test metrics types of applications, because we deal with the Web, we deal with different browsers, different servers, CMS systems, ad systems, everything under the sun. I really rely on having great beta testers.
At one point, I was even calling out when I would do beta release notes, and I would say, "This bug was fixed." I'd even call out a user's first name and last initial in the beta release note, just to give them that shout out and let them know how valuable they were.
Rene: It's interesting, too, because -- and I keep going back to this, because I find the comparison, the juxtaposition fascinating. You look at an organization like Apple, and you have the engineer who could hit issues. You have whoever their engineering manager is, or the engineering program manager, who could potentially discover issues.
You have code reviews. You have execs running betas. You have people within the company running, whether they're running internal builds of the product, who could run into issues. You have that whole layer of feedback, and then when things do go into developer public betas, you have the beta feedback loop, whether it's [inaudible 48:02] or...I forget what the app is called, the feedback app or...
Jonathan: I think it's just called the app feedback. [laughs]
Rene: Yeah, feedback, on the public betas. You have that level, and then you have everybody who hits it when it goes into wide release. You have reviewers who sometimes find things, too, like famously, Lauren Good and Joanna Stern found the LTE bug, or the captive WiFi portal bug on Apple Watch Series 3 during their review period.
You have all of these levels of feedback to go through, and of course, you have Radar, the screeners, and all of those things, those tools around it, and then you have what you just described, which is an owner/developer with almost completely direct access to a beta group and a customer pool with very few...
You have a direct relationship, but you also don't have the entirety of those stakeholders looking at it every second. [laughs]
Jonathan: I would say, one of the really important issues is, especially with a company like Apple, where they're at such a large scale they get so much feedback, is being able to sort through the good and the bad feedback, make sense of it, and get it to the right place at the right time. That's a very difficult organizational problem, to some degree.
If you look at the bug reporter interface, that's also something that clearly could be improved, and I think it turns into a very virtuous cycle when if people giving feedback feel rewarded by the feedback, they're going to give more feedback. Of course, now you have more feedback to deal with, and you have to figure out how to manage that.
Rene: I'm checking, on the mobile view of Bug Reporter, it no longer has pinstripes.
Jonathan: [laughs]
Rene: That, it lingered. Pinstripes lingered on the mobile version of Bug Reporter, of Radar for so long.
Jonathan: Whether the pinstripe's vertical, horizontal...I don't mind the pinstripes. I just mind the communication.
Rene: I always...
[crosstalk]
Jonathan: It's in the content that's king.
Rene: I always joke with mutual friend Ryan that his legacy is green felt. Green felt's not bad.
[laughter]
Rene: So many textures. Sort of to sum up, because I want to talk to you a little bit more about Hype before I let you go. To sum this up, bugs happen, and they're terrible, and some bugs are catastrophically terrible, but I don't think any company sets out to have those bugs, and I think there are legitimate reasons why they happen. Those absolutely have to get fixed.
I think we're going to keep seeing bugs, even I think, if we went back to an Apple that only made Macs, we would keep seeing bugs. The law of averages would just mean that once in a while, we'd still have catastrophic bugs.
As people now, because both you and I are -- you have joined the outside of Apple [laughs] -- people who use this software, where do you think -- and I know this will depend a lot on the individual. How do you think that we should react to this stuff? There are some people who get super angry and super salty, and there are some people who would just say, "It happens," and they're very blasé and laissez-faire about it.
What do you think our responsibility is as customers and consumers when we encounter stuff like this? Pitchforks, a condolence package?
Jonathan: I think the biggest thing is making sure that accurate information gets out there on what the issue, in fact, is and how to protect yourself from the issue. I think one way or another, maybe this is a larger discussion on the Internet in general, but rage spreads really quickly.
When a mistake has been made, it's very hard, at some level, to not be raged and that to not come across. I would love to say we should all be very level-headed about the issue but I know in reality we're not going to be.
I think the most important thing is that, especially, maybe, from your role, that you make sure, as a journalist, that the right information is communicated. I think the sooner you can get accurate information out, the better a community reaction can be and the better everyone can do to protect themselves until Apple has a fix.
Rene: It's interesting because the Internet, to your point, it tends to reward you for extremist behavior. If you are the "everything is doomed" person and "apple is absolute trash," you're rewarded by people who think that that's cool.
If you were the person "Apple can do no wrong," and you're a jerk if you point out that they are doing something wrong you're rewarded by people who believe that you have to have an absolutely loyal fan base on this kind of stuff.
If you exhibit any median behavior, and I would also suggest that if you remain level-headed you make angry people even more angry which, to me, is always an interesting dynamic.
[laughter]
Jonathan: I think there's also other questions that software companies can ask themselves, as well, on how do we improve quality knowing that not every bug is going to get fixed before it goes public. Hopefully this also spurs the discussion at Apple on how to improve quality and make sure that even more security issues get fixed sooner.
I think that everyone has the responsibility to help improve the world.
Rene: One of the issues, it's almost like cry wolf syndrome. It's dual sided. Your greatest strength is always your greatest weakness. Apple's culture is one of their greatest strengths, but it's also one of their greatest weaknesses.
If you hear, year after year, that it's the worst year ever, or if you hear this product is terrible, only for it to sell incredibly well, AirPods is a recent example of that or the original iPhone. If you hear that stuff all the time, you start to think, "Well, people are always upset when we introduce something, but later on, we prove to them that we're right."
Then, when you hear people being upset, the response ends up becoming, "Well, they're upset now, but when we get to version two or when they've had the product in their hands for a week, they'll come around. They'll see it." The danger there is that when you do ship a dud or a lemon, you still tend to think that.
You get the feedback, "Oh, people hate it. Well, you know, wait a week, wait a month, wait a year. They, they'll find out that we're right." It blinds you to real problems, that your success hides real problems. I think that's the danger, that's the complacency that you can fall into if you don't always rigorously...
I keep going back to kendo with you. [laughs] If you stop matching, you stop realizing what skills are real and what skills aren't. It becomes a theoretical exercise where, "Oh, I would have won if I'd done..."
You know what I mean? If you stop always testing the reality of your universe and your fact base, you can very easily slip into a deluded state.
Jonathan: Yeah, I think there is an expression that a person's true personality comes out in their kendo match. I think not only is that true...
Kendo being a martial art, you try to not have ego, but I find ego does come out. People think, "Oh, you know, I, I can beat that person," or we'll talk, "Oh, you could definitely take that person on," but you never know until you enter into the ring with them.
Rene: No, it's the same thing in Brazilian jiu-jitsu. On the mats, there's no lies. [laughs] There's no stories. It all comes out, and I think that's the attitude that you have to have, no matter how big or successful you are.
Any time that you see this narrative, you see this meme, you have to ask yourself is this one of those cases where they are wrong, and they'll love the iPhone, they'll love AirPods? Is it one of those cases where they are right, and it's like the new Mac Pro, like we went the wrong way, and we've got to fix this?
Jonathan: Let me ask you this question, Rene. You asked me about what is our responsibility as users. Do you think as users we should maybe be holding off on updating a little bit?
Rene: I think that's an incredibly valid question, and you see this now. You see people who stayed on in Sierra say, "Ha, ha! You know, we didn't get bit by the High Sierra bug." You see people who, "Apple is doing a forced update with this." Some people are disabling forced updates, and they ended up not getting hit by the file sharing bug.
It really is a complicated issue now. It was complicated for Microsoft when they started doing the monthly updates, is that you have this window. Most updates, yes, there are bug fixes and performance enhancements, and those are important, but there are security fixes.
When those updates come out, those security fixes are disclosed, at least to some extent. That means that from that moment on, you're a target. Some people have really very minimal target profiles. They have very little danger of anything happening to them.
Other people have much larger target profiles. For examples, if there's something to do with malware, you're on the Web, and you click the wrong link, then your not updating has left you vulnerable to that attack. If you did update, maybe you were left vulnerable because of this High Sierra bug.
I think we are really stuck between a rock and a hard place now where there is absolute, valid reasons that everybody should be updating, but we're not at the software quality standard where everybody can confidently update yet.
I think that is one of the biggest problems we face in software right now. As a user, I don't know what to do about it yet. I update almost all the time, anyway, because I feel like I have to take it on the chin for people that I write for. I don't know what I would recommend to my parents at this point, for example.
Jonathan: It's like getting a new toy that you always want to update to the latest and greatest, but in some cases, maybe it's not advisable. I don't know.
Rene: I think your earlier point is very apt here, and that is there are new strategies that companies...I've heard rumors that Apple has been looking into these, too, in part to solve the issue of people running out of space during updates.
They've done things like app thinning to solve that. Another way of solving that is to continuously stream bits over the life of the product, which is like what Chrome does, and what Microsoft is starting to do.
There are different ways of handling software updates. You can stream bits to people in small amounts for smaller changes. You can also do what you mentioned earlier, which is what I believe the Google Play Store does.
Developers can sample 0.1 percent or 1 percent, I forget the exact number. If there's any adverse effects, they can stop that update, so the other 99-plus percent don't get by that problem.
I think those sort of mitigations are what every software company large and small, because everything is just so interconnected and dependent now, are going to have to start exploring as we move forward.
Jonathan: I think that's really, modern software development is a direction that Apple and other companies need to go in and look at. Maybe it's not do everything that Facebook does, because you are an operating system, which is a very low level component, but there's new strategies to pull.
Rene: Google famously took a lot of the apps out of the operating system and put them in Google Play Services. Now, they have political reasons for doing that, too, but it does mean that all of those apps and services can be updating out of band with the baseline operating system.
That has certain advantages as well. It's not a panacea. I think podcasts.app was actually updated more when it was part of the OS build than it was when it was put into the app store. It has a big recent update, yeah, but I think when I measured the amount of updates, they were less, because there was no drive to get it into the update.
Definitely a mixed blessing, but I think there are all those options that I'm sure Apple is exploring them, but at least my personal opinion is I would like to see them.
Jonathan: I think when you look at Mac OS nowadays, it's also in a very weird state, because Mac OS started not with so many apps. I think they kept increasing the number of apps to provide value to the operating system as an incentive to upgrade, but also as a way for Apple to gain revenue.
Mac OS used to cost money, and it doesn't anymore. I think unbundling some of the applications on the Mac side also might make some degree of sense.
Rene: You can delete apps and re-download them now, but still, as much as I say I want less apps in Mac OS, where is my news app in Mac OS? I want to be able to have all of the stuff that I've set up in iOS news just mirrored on my Mac when I'm sitting at my Mac. Again, there's these tensions.
Jonathan: Yeah, there's no winning them all. I think that goes back to your earlier point.
Rene: Before we wrap, how is Hype doing these days?
Jonathan: Hype's doing pretty good, beta testing a brand new version that I'm super excited about. I don't want to reveal all the details here, but I've seen some documents that beta testers have sent. I'm just blown away with the creative capability, which is what I always like doing.
When I can make a feature that improves someone's creative capability, when they can make an animation that couldn't have been made before, and then I see that back in a professional and a useful way, that just makes my day. I'm seeing that. Hopefully, early next year, we'll get Hype 4.0 out the door.
Rene: Wrapping up, if people are interested in finding out more about you, more about Tumult, more about Hype, where can they go?
Jonathan: They can go to the Tumult website, which is just tumult.com. You can do tumult.com/hype to learn more about that product. There is a gallery that has tons of examples. Hype's one of these black canvas types of tools, where you can use it for a lot of different purposes.
People will make infographics, children's books, advertisements with it. It's really useful for all of that. Actually, one of my favorite features is you can also export as an animation gif. That was one thing we did improve in the last release.
Not only can you export to HTML5, and have things be interactive, but if you just need an animated gif, or if -- I don't want the pitchforks to come out, animated [soft G] gif -- you can do that, too, and put that in many places that goes.
Rene: I believe the G is silent. It's an animated if.
Jonathan: [laughs] I've heard of people combining the G and the J as well.
Rene: It's a kif. It's actually a K. I don't know, too many options.
Jonathan: You can do video formats, is the basics of it as well. Animation is really fun. I think when people play around with a product and animate, it's like you're bringing something alive. I always think it's fun to play with.
Rene: Absolutely, totally. The last time we talked, I mentioned this, but one of my early jobs was Flash animation. The technology, it was like ActiveX, where it solved a hole that existed in web technologies. Now, that hole, it no longer exists, so it no longer has a place.
I think the animation, I'm glad that products like Hype allow that rich, detailed animation to exist on the web in a cleaner, more secure, higher performance form.
Jonathan: I think the thing, too, is animation's such a visual medium that while Hype uses HTML5 technologies on the back end, you can do so much by being able to see, and have so much more sophisticated animations.
The engine we use also just has really powerful features, like being able to make custom timing functions of arbitrary degrees. One of my favorite ones is also, you can have drag interactions where you can craft a timeline, and then bind someone's swiping to that timeline as well.
[laughter]
Jonathan: There's this high degree of interactivity that doesn't really require any code. You can always extend it with code, but I feel like when you have a visual medium, for me personally, as someone who does do code, I often just prefer going to the visual tool.
It's so fun seeing what users are up to with Hype. They're such creative people.
Rene: It's like molding clay, as opposed to drawing spline, vector, or polygon paths, which is just a lot of fun. If people want to follow you on Twitter, where can they find you?
Jonathan: My Twitter handle is JMFD.
Rene: I will not ask you, sir, what the MF stands for.
Jonathan: My middle initials. What can I say?
Rene: Thank you so much for talking to me. It's always a pleasure.
Jonathan: Happy to be here.
Rene: You can find me @reneritche on Twitter, on Instagram, on all the social things. You can email me at rene@imore.com. I'd love to know what you thought of the show, what you think about the topic, what you think about the root vulnerability, and what Apple can do to address things like this going forward.
Just to let you know, if you haven't already, you can subscribe to the show. All the links are below. I want to Jim Metzendorf for editing and producing the show. I want to thank you for listening. That's it. We're out.
[music]
