If you’re like most people that stay relatively in touch with the latest that’s happening around their smartphones and computers, you’ve likely shared a great deal of excitement around the development of artificial intelligence, and the incredible things it can do for humanity. If you’re a more thoughtful or weary individual, you may have just rolled your eyes at that statement, preparing yourself for yet another exaggerated sea of paragraphs exclaiming how its adoption is a necessity, and how you’re on the verge of getting left behind. There’s no doubt that the controversy surrounding these new developments is unprecedented, and every individual has a wildly different experience with them. Like many breakthroughs in technology, there’s substantial emphasis placed on the benefits it will bring to its users, but there’s also lots happening behind the curtain that seldom reaches the public’s eye. I’m more interested in that second part. I’ve always approached new developments with an investigative mindset, that remains true even at the speed progress is moving at today. Ever since we’ve been supercharging the integration of AI into every platform imaginable, there’s something on my mind that’s been going largely unnoticed, and I wanted to take a moment to share it with you all.
There’s been no shortage of things to talk about in tech lately, and to the surprise of no one, many have been about AI. Apple just held their annual WWDC event, making strides to finally improve their artificial intelligence systems across iOS, while Google’s been hard at work at continuing the development of the AI models that power all their digital services. Just a few months prior, it was pretty much set in stone that if you wanted a cutting-edge mobile device with the latest advancements in tech, Android was the way to go. The rate at which Apple was rolling out features was downright appalling, the few features that were released were nothing short of an embarrassment, and the quality of their software was sinking off a ship. However, starting this June, there’s newfound hope. The public has now seen a preview of what the team at Apple has been working on over the past year, and their plans to make artificial intelligence deeply integrated into iOS.
Those two words quickly send a jolt of suspicion down my spine in a way that most others probably wouldn’t have even thought twice about. If we take a step back and examine the behavior of Large Language Models (LLMs) outside of their narrow applications in smartphones, what do we notice? Personally, it’s their unreliability. Apple is a company with a cemented track record for refusing to implement features until they can be sure they’ll work with a near-perfect reliability rate, something AI is historically terrible at. When Google initially launched their AI overview feature, they were immediately met with fierce backlash amongst the community for the horrendous responses it was serving them with. From instructing users to add glue to pizza to eating one rock per day, it became a game for some people to find the worst batches of problems Google’s new rollout had. This was extremely similar to what Apple had experienced when they released their AI summary feature. It’s the cost of being forced to pump out a feature for the sake of doing it faster than someone else, which is a recipe for disaster if you actually care about the quality of your software.
However, even more important than the reliability of the output is its integration into your workflow. Today, most consumers who interact with artificial intelligence are doing so through dedicated apps and websites. These platforms are capable of processing contextual information that you provide them with, as well as storing key details of that data for future reference. They can also extract context from a query or piece of data without you explicitly asking them to. These capabilities are in part what make these tools so useful, but they also abstract away a level of control we would typically have, and that’s the data the they collect about us. The goal of integrating AI into mobile and desktop operating systems is to take the tasks you do by hand and automate as many of them as possible. In order for this to work, the AI system (or “agent”) needs access to all of the same information you would normally have when performing the task yourself. One of the biggest issues arises when you combine this data access with the unreliability and lack of understanding of how an AI model locates and interprets information about you.
Since that might come off as slightly indigestible, let me give you an example. If you’re using Google Gemini, ChatGPT, Claude, or any other model via its app on your phone, and you upload a copy of your resume and ask it to make modifications for you, it will process it and attempt to make those changes. However, something else it will also do is absorb every piece of data in that resume and save it to the chatbot’s “memory”. Your address, employment history, interests/skills, etc. As you continue interacting with it, asking it different questions and answering its responses, its repertoire of information about you will continue to grow. When you later ask it something completely unrelated to your resume or job, it will still reference that information, only now under a different context you weren’t expecting it to. Think about having a conversation with someone you meet for the first time. If they remembered absolutely everything you said, even the tiniest of details, what would they be able to infer about you that you might not think about? Not that they’d do anything sinister with it, but we’re talking about a computer here, owned by a massive for-profit corporation, not a person.
Unlike when you directly hand information to a chatbot, the most dangerous aspect of context extraction occurs when you don’t have direct control over what it’s ingesting. In order for AI models to become autonomously useful and reach the level of aid companies are pitching, they need to be able to sift through all of your information across all of the apps and services you use, all the time. Your text messages, emails, calendar events, voicemails, photos, documents, etc. This way, if you ask it a personal question, it will be able to access your phone the same way you normally would, and perform the task on your behalf. The issue with this approach, however, is again the unreliability and lack of control. If a real person had access to virtually everything from all of your apps and accounts without you knowing, that would be an absolutely incomprehensible nightmare. But when an AI model does… it’s fine? Why? If you honestly told me you were oblivious to the constant stream of mistakes, hallucinations, and irrational decisions AI models make, I’d swear you were living under a rock. There’s a real problem at stake here.
I’ve been breaking down technology concepts long enough to understand the typical way people approach complex, abstract ideas like information security or data privacy. It will of course vary between individuals, but a sensible generalization is that when people aren’t actively seeing something happen on their device, it will remain almost entirely unnoticed and unconsidered. Conceptualizing something that only ever exists behind the scenes requires you to either understand how it operates, or see it happen in real time. If the latter isn’t possible, grasping it will prove extremely difficult. Many people even dismiss it as a non-issue, or tell themselves it’ll never happen to them as a way to get it off their mind, which is a dangerous precedent to set. This is the current situation with integrating agentic functionality into our devices at record speed, and it lays the groundwork for my intent behind this article.
Context-aware intelligence on smartphones before AI
Many people may have forgotten, but before the integration of LLMs into our software, we were already able to string together information about a user from across different apps, and combine it to deliver a personalized and predictive user experience. Apple had the foundation for this all the way back in 2009 with the release of iOS 3. The way that Apple’s system apps were designed exposed the data within them to the operating system directly. This meant that core information like your emails, texts, contacts, calendar events, photos, and voicemails were all indexed and accessible from a centralized location. This exact functionality still exists in modern iOS versions; just pull down on your home screen and try searching for something, like the contents of a recent text message or email. Maybe even a phone number you have saved, or the name of a calendar event. If you didn’t know you could do this, it’s probably because there are never any interaction hints nudging you to do so — a rather important yet neglected design element. This works because every system app has a dedicated “path” for iOS to interface with and retrieve information from. This type of experience was trickier to deliver on Android due to different manufacturers shipping different apps in different environments. Google could design a cloud-based implementation for their own services, but they had absolutely no way of integrating it globally across the Android project since every manufacturer wanted to deliver a slightly different user experience.
Throughout Siri’s development on iOS, it gradually became the backbone for many intelligence-based features. Starting in iOS 7 in 2013, your phone would remember at what locations and times you use specific apps to determine what was given priority to preload content in the background. In iOS 9 in 2015, similar information would be used to power Handoff suggestions. These and similar features were often branded as “Siri Suggestions”, even if you didn’t associate them with Siri in your mind. Throughout future software revisions, functionality continued to be added until “Siri” could do things like recognize events in text messages or emails and add them to your calendar, and later interface with third-party apps through Siri Shortcuts, something I’ll get to shortly. Other manufacturers have since adopted nearly identical features within their respective ecosystems, again, without requiring the use of any LLMs.
Why integrate AI?
So, if we already had perfectly fine solutions before, why did we bother integrating LLMs into our smartphones? Hype may be part of it, but it’s mainly to improve upon these existing systems even further. The features I just described had to be individually programmed for every use case. In order for the messages app to detect a date and show that as a suggestion in the calendar app, a developer had to specifically program it to do so. This substantially narrowed the scope of available intelligence features. When interacting with digital assistants like Siri or Google Assistant, they also had to be pre-programmed to identify specific keywords to gauge your intent. It’s why earlier versions of them would so frequently resort to “I don’t understand”, or “Here’s what I found on the web”. If you used words or phrases it wasn’t familiar with, it couldn’t convert your spoken language to a request it understood.
LLMs had the ability to solve this human-computer disconnect. The way in which you spoke to the assistant didn’t matter at all, because it could understand your query almost the same way a real person could. Developers didn’t have to determine and implement a specific handler for every possible query someone could ask the assistant, because the assistant could find a way to handle new tasks on the fly. It unlocked a new door for capabilities in digital assistants — admittedly at the cost of a lot more processing power and cloud compute requirements, which people frequently disregard — but it was undoubtedly a massive improvement over previous methodologies. So, why the worry?
The privacy concerns
I could blurb on about all the new types of requests these types of assistants could handle, but that isn’t why I’m here. Apple’s third attempt at modernizing their aging voice assistant carries more weight now than ever, mainly due to their smartphones still being in the hands of more people than competitors in the U.S., but also due to their exorbitant control over their app ecosystem. This time, it’s playing in their benefit.
We’ve already covered that Apple’s own system apps are tightly integrated with iOS, and have extra permissions that other apps don’t. This allows their new Siri AI platform to access your data within Apple apps, like text message history, calendar events, contacts, photos, reminders, and more. However, they’re also opening up access to app developers. Since iOS doesn’t know how to interface with apps that aren’t designed by Apple, it’s up to developers to tell it. That’s what Siri Shortcuts and App Intents is for. Although companies like OpenAI have already experimented with automating tasks in third-party services, it’s never as reliable as having a direct integration with that app or or service provider, which is what Apple is aiming to achieve at the device-level.
If you’ve used the Shortcuts app on iOS for anything other than generating custom app icons, you may be keenly aware of its integration with third-party apps already. Apps that support it are admittedly still low, likely because most people aren’t aware of its functionality and supporting it takes away from development time. The best way to experience this is to try it for yourself. If you search for and open the Shortcuts app (if you haven’t deleted it), you’ll be able to create a new shortcut and pull up the bottom panel to reveal all your apps that already integrate with it. If you select an app, you can see what actions you can run. Some apps like Instagram are great examples of this. If Instagram appears in your list, select it and scroll to the end. You’ll find recent contact info displayed there, indicating that Instagram is already using a “path” (API) to share what happens inside of it to iOS. Siri AI can then use it to perform actions related to Instagram upon your request, without you manually programming a shortcut. Think about it for a moment. Two different apps, Shortcuts by Apple, and Instagram by Meta, being able to share data between each other and understand your behavior. What possibilities could this unlock, both good and bad? In what different ways could this data be interpreted at scale that would allow you to generate a complex map of someone’s activity? What about other apps that begin integrating this?
There’s a term in cybersecurity that us tech folks like to reference, and it’s called, “not putting your eggs in one basket”. Essentially what it means, is that when it comes to keeping your information secure, the more of it you keep in one location, the higher risk you take of getting it compromised. If someone gained access to your Instagram account, they would only have access to your email, birth date, posts, messages, and other minimal information you keep there. If someone was able to extract the database on your phone that contains the core information indexed by Siri and Spotlight, they would be mostly restricted to system apps. However, with the advent of Apple’s new integrations, this surface expands significantly. Even if your phone doesn’t get compromised, the damage done by a mistake in the increasingly unreliable Siri AI is multiplied due to all the new information it has access to across your apps, and the actions it can perform within them. We’ve already seen the worst of what can happen in these situations when someone’s entire email inbox got deleted without their permission when using OpenClaw. They’re not reliable tools.
Many useful applications of AI aren’t even dependent on you deliberately interacting with it either. On Google Pixel devices, your lock screen can update in real time when traffic in an area increases to notify you if you should begin leaving earlier, by intelligently combining your Google Calendar information with crowdsourced user data from Google Maps. On iOS, The Journal app combines a plethora of data points to determine what you might want to write about before you even open the app, such as your heart rate, music you were listening to, where you were, and when. These kinds of features require constant background analysis by the operating system that the user isn’t fully in control nor aware of.
The more control you hand to a computer and the more access you provide it with, the higher the risk becomes that something goes wrong. It’s easy to overlook this amidst the hype and usefulness of these new tools, but it’s something we all need to be aware of. Device manufacturers can’t rely on us to understand their products this intricately in order to stay safe, but the manufacturers that take more cautious steps risk losing customers to the ones that don’t. It’s also surprisingly difficult to understand why a company is behaving the way they are. Most are playing it full speed ahead in the race to build the “next-gen AI-powered platform” for whatever service they’re providing. They’re selling a technology and a product before they even know how it will work. Sometimes that works, but many times — it doesn’t. There’s a lot of consideration that happens behind the scenes, something Apple’s VP of Software Engineering, Craig Federighi, talked about on an interview with Joanna Stern from The Wall Street Journal. I’ll leave timestamps at the end of this article if it’s something you want to watch.
Making sense of it all
Once the rollout of Siri AI is complete later this year, you’ll have a new friend in your pocket. An autonomous agent that has a deep understanding of who you are, what you do, and how you use your phone. An agent with complex and unpredictable behavior that makes its own decisions, with access to unprecedented amounts of centralized data about you. Can it be helpful? Sure. Does it open the floodgates for massive security and privacy issues? Time will tell. Apple can’t fix the problems that plague the foundation of LLMs, but they can design guardrails that keep us in control. Rather than betting they’ll make the right choices, I’d rather approach the new technology with the same amount of skepticism I’d approach any other AI development. While others tear down the caution tape and plunge head first into the feature influx, I’ll be here researching it and surfacing its flaws.
This was a challenging topic to try and decipher, especially considering my aspiration of making the technical side of tech more accessible. Despite the monumental leaps in artificial intelligence-based tools over the last few years, they’re still very new, and we’re learning more and more about them with each passing minute. The less consumers understand a piece of technology, the higher chance it has of being abused by the people who control it and facing minimal pushback. Many people are exhilarated, many are petrified, and many lie somewhere in between. During a re-imagined era of personal computing, it’s critical that we don’t overlook the ramifications this technology imposes, and that we continue to approach them with a cautious and investigative mindset.
. . .
Thanks for reading until the end! If you found this article thought-provoking, or know someone who might, give the gift of knowledge by sharing it with them.
If you want to be the first to know when something else goes live, you can join my newsletter below.
Bye for now 👋
Additional Resources:
Apple’s Software Chief Craig Federighi Discusses the Future of iPhone AI – Wall Street Journal (2024 Interview)
Key Timestamps:
(3:30-4:47) What’s new with Apple Intelligence?
(4:48-8:00) What is Private Cloud Compute?
(8:00-10:16) Is my data being used to train the AI model?
(10:17-12:15) Safeguards and Moderation
(12:16-14:15) Notification and Content Summary Improvements
(14:15-16:08) Siri Improvements
(16:08-20:49) Will Siri finally live up to its promise? (Why can’t it just understand me?)
(20:49-23:02) Why are Apple’s AI photo tools so behind?
(23:03-24:45) Why the delay?
