Apple Intelligence: System-wide AI system presented

As expected, Apple launched its own AI offensive at WWDC 2024 and presented Apple Intelligence, an AI system that operates system-wide and brings powerful generative models directly to the iPhone, iPad and Mac.

Apple Intelligence presented

“Apple Intelligence sets a new standard for AI privacy and understands personal context to enable intelligence that is helpful and relevant, ” says the Cupertino-based company.

Apple Intelligence is deeply integrated into iOS 18, iPadOS 18 and macOS Sequoia. It uses the full power of Apple chips to understand and generate speech and images, perform actions in apps and use personal context to simplify everyday tasks and make them faster.

The whole thing runs via private cloud compute and is intended to enable a new standard for data protection in AI, flexibly adapt the computing capacity for on-device processing and fall back on larger, server-based models that run on their own servers with Apple chips.

The new system-wide writing tools allow text to be reworded, proofread and summarized in all apps such as Mail, Notes, Pages and apps from other providers.

With Rewrite, Apple Intelligence gives you different versions of the text to choose from in order to adapt the tone to the target group and the task. This can be used to perfect a cover letter or create a creative text. Proofread checks grammar, word choice and sentence structure and makes editing suggestions including explanations, while Summerize allows you to summarize texts as paragraphs, lists of the most important points, tables or lists.

Priority Messages place important emails at the top of the email app; Apple cites invitations to dinner on the same day or a boarding pass as examples. Users can also see summaries of the entire inbox instead of a preview of the first few lines, without having to open a message. Smart Reply makes suggestions for a quick reply and recognizes questions in an email to ensure everything is answered.

Audio transcriptions for notes and calls

Thanks to Apple Intelligence, you can now record, transcribe and summarize audio in the Notes app. This also works in the Phone app: when a recording is started during a call, participants automatically receive a notification and when the call ends, Apple Intelligence creates a summary so that important points are remembered.

Image Playground

Image Playground aims to be a new way to create images and communicate in new ways. The app makes it possible to generate funny images in just a few seconds and choose between three styles: Animation, Illustration and Sketch. These can be integrated directly into apps such as messages and the images are generated directly on the app.

It is possible to enter a description for a picture, select people from the personal photo library who should be in the picture and choose the style of your choice.

In Notes, users can use the new Image Wand in the Apple Pencil tool palette to access Image Playground and make notes more visually appealing. Rough sketches can be turned into appealing images, while blank areas can be automatically filled to match the context. Image Playground is also available in apps such as Keynote, Freeform and Pages, and in third-party apps with the new Image Playground API.

New features in Photos and Siri

Searching for photos and videos is even easier with Apple Intelligence. Certain photos can be searched for using natural language. For example, you can search for images that show a specific person in a specific outfit.

Searching in videos is even more powerful with the ability to find specific moments in video clips so that users can jump directly to the relevant scene. In addition, the new Clean Up tool can identify distracting objects in the background and remove them, just like Google’s Magic Eraser.

Thanks to the power of Apple Intelligence, Siri is more deeply integrated into system functions. With more comprehensive speech understanding capabilities, Siri is more natural, relevant and personalized, and can simplify and speed up everyday tasks.

Siri understands you, even if you make a mistake, and keeps the context of several questions in a row. In addition, you can send Siri requests via text input and switch between text and voice to communicate with Siri in the way you want at that moment. Siri also has a completely new design with a bright light around the edge of the display when Siri is active.

Siri recognizes what is shown on the display and can understand the user’s content and perform actions with them in more and more apps over time.

ChatGPT integration

Apple is also integrating the popular AI chatbot ChatGPT into iOS 18, iPadOS 18 and macOS Sequoia. This means that the possibilities and abilities to understand images and documents can be used directly in the system without having to switch between apps.

Siri can use ChatGPT’s knowledge when it is helpful. Users are asked for permission before questions are sent to ChatGPT, along with documents or photos. Siri then presents the answer directly. ChatGPT will also be available in Apple’s system-wide Writing Tools to help create text, while Compose will help create images.

Apple Intelligence is free for users and will be released in a beta version as part of iOS 18, iPadOS 18 and macOS Sequoia this fall, initially in English. Apple Intelligence will be available on iPhone 15 Pro, iPhone 15 Pro Max, iPad and Mac with M1 chip and newer.