All the news on Microsoft’s latest Copilot and Windows AI features

Microsoft is testing a feature for Copilot Voice that will let Windows 11 users start a conversation with the app by saying “Hey Copilot!” Windows Insiders who have the updated app can try it out once they opt in and activate the feature.

Similar commands have been available for years with Siri, Google, Alexa, and even Microsoft’s now-departed Cortana assistant that Copilot has replaced. But it is a new addition for Copilot, if a dedicated key or combination of keys wasn’t making it easy enough to get into the app, and arrives following the launch of a more conversational Copilot Voice experience.

Copilot Vision, Microsoft’s AI assistant feature that can interpret what’s on your screen and help you use apps, is now available for free use within the Edge browser, Mustafa Suleyman, CEO of Microsoft AI, announced on Bluesky today. Vision is a “talk-based experience,” as Microsoft calls it, meaning you use it by speaking into the air, then waiting for Copilot to respond.

Suleyman says if you opt into the feature, Copilot Vision can “literally see what you see on screen.” Suleyman suggests having Copilot Vision guide you through a recipe while you cook or having it “decode” job descriptions “and jump right into customized interview prep or cover letter brainstorming.” (Although it might not be the best idea to use AI for your resume.) According to a Microsoft support page, “Copilot Vision may highlight portions of the screen to help you find relevant information,” but it doesn’t actually click links or do anything on your behalf.

Microsoft has announced Microsoft Dragon Copilot, an AI system for healthcare that can, among other things, listen to and create notes based on clinical visits. The system combines voice-dictating and ambient listening tech created by AI voice company Nuance, which Microsoft bought in 2021.

According to Microsoft’s announcement, the new system can help its users streamline their documentation through features like “multilanguage ambient note creation” and natural language dictation. Its AI assistant offers “general-purpose medical information searches from trusted content sources,” as well as the ability to automate tasks like “conversational orders, note and clinical evidence summaries, referral letters, and after visit summaries.”

Microsoft is testing AI-powered Windows search in a new dev channel build for Windows 11 Insider testers. Announced in October, it uses semantic indexing to let users search for local files using more casual language. Like other Microsoft AI features, you’ll need a Copilot Plus PC to use it.

The feature applies whether you’re using search boxes in Settings, File Explorer, or the taskbar. And you don’t need to be connected to the internet for it to work, thanks to the NPU chips on Copilot Plus computers. For now, AI search is limited to Windows settings and files with image and text formats that include JPEG, PNG, PDF, TXT, and XLS.

Asus has announced the Asus NUC 14 Pro AI, the first Copilot Plus-capable AI mini PC that crams an Intel Core Ultra 9 processor into a form factor resembling a black M4 Mac Mini. First introduced at IFA in September, Asus is providing a little more detail about the mini PC’s specs than it did before, but still isn’t saying it will become available or how much it will cost.

The NUC 14 Pro AI will come in five CPU configurations, from the Core Ultra 5 226V processor with 16GB of integrated RAM to a Core Ultra 9 288V processor with 32GB of RAM. The company says it has up to 67 TOPS of GPU performance and 48 NPU TOPS, and that its M.2 2280 PCIe Gen 4 x 4 slot supports 256GB to 2TB NVMe SSDs.

Microsoft is launching a redesigned version of Copilot today, intent on becoming an AI assistant or companion. To celebrate this, Microsoft’s new AI CEO, Mustafa Suleyman, has penned a 700-plus-word memo on what he describes as a “technological paradigm shift” toward AI models that can understand what humans see and hear.

Suleyman joined Microsoft earlier this year as the CEO of its new Microsoft AI division, amid the software giant’s hiring of a number of key Inflection AI staff. In June, Suleyman sparked controversy after brazenly claiming that anything published on the web is “freeware” that can be copied, recreated, and reproduced by AI models. Now, he’s optimistic that AI — under Microsoft’s stewardship — will create a “calmer, more helpful and supportive era of technology, quite unlike anything we’ve seen before.”

Microsoft is using AI models to greatly improve Windows search on its new Copilot Plus PCs, including the addition of a new Click to Do feature that’s very similar to Google’s Circle to Search. These search improvements will make it easier to find and interact with images, emails, documents, and even videos and are just a few of the AI-based features coming to Copilot Plus PCs starting in November.

The improved Windows search will first show up in File Explorer on Copilot Plus PCs next month, allowing you to search for pictures using words, even if the search word isn’t found in the photo or file name.

Microsoft is starting to release its Windows 11 2024 update today, also known as version 24H2. This update includes a number of small but useful additions to Windows 11 that improve the Start menu, File Explorer, Settings, and much more.

The Start menu has a big change for those using Microsoft’s Phone Link software. The Windows 11 2024 update adds a side panel to the Start menu that provides information on your phone’s battery status and notifications as well as quick access to messages, calls, and photos. The floating panel is a neat addition for Phone Link users.

Microsoft is bringing some new AI-powered Paint and Photos features to Copilot Plus PCs that could make creatives less reliant on more powerful image editing software. Generative Fill and Generative Erase — which appear to be heavily inspired by similar AI tools in Adobe Photoshop — are being introduced to Paint, allowing users to precisely add or remove objects in their images.

Both tools utilize a size-adjustable brush to “paint” over specific areas of an image to edit. Generative Erase will remove unwanted figures, objects like background clutter, and other distractions, similar to the Magic Eraser feature on Google’s Pixel phones. Generative Fill allows Paint users to add new AI-generated assets to an image using a text description and select precisely where they should be placed — much like the Photoshop tool that shares the same name.

Microsoft is unveiling a big overhaul of its Copilot experience today, adding voice and vision capabilities to transform it into a more personalized AI assistant. As I exclusively revealed in my Notepad newsletter last week, Copilot’s new capabilities include a virtual news presenter mode to read you the headlines, the ability for Copilot to see what you’re looking at, and a voice feature that lets you talk to Copilot in a natural way, much like OpenAI’s Advanced Voice Mode.

Copilot is being redesigned across mobile, web, and the dedicated Windows app into a user experience that’s more card-based and looks very similar to the work Inflection AI has done with its Pi personalized AI assistant. Microsoft hired a bunch of folks from Inflection AI earlier this year, including Google DeepMind cofounder Mustafa Suleyman, who is now CEO of Microsoft AI. This is Suleyman’s first big change to Copilot since taking over the consumer side of the AI assistant.

Microsoft is working on an overhaul of its Copilot mobile app that includes a new feature that will transform the AI assistant into a virtual news presenter. Multiple sources familiar with Microsoft’s plans tell me that the software giant has been testing a completely redesigned Copilot app in recent weeks that looks unlike any of Microsoft’s other apps.

The Copilot redesign surfaces topics you can choose from based on your own interests or your history of asking Copilot questions. The AI assistant might offer to generate a story for you one day, ask if you to do a workout the next, or simply surface the latest sports scores without you having to ask for them.