How Microsoft is thinking about the future of Copilot and AI hardware

It’s been a big week for Copilot and Microsoft’s AI efforts. Microsoft unveiled a redesigned Copilot that’s aimed at consumers, complete with voice capabilities and the ability to understand what you’re looking at on a computer. These new Copilot Voice and Vision features feel like a key evolution in Microsoft’s effort to make Copilot a more personal companion and hint at what’s to come for AI experiences.

I got to speak to Windows and Surface chief Pavan Davuluri and consumer chief marketing officer Yusuf Mehdi to better understand the future of AI for Copilot and Windows. Microsoft is thinking about totally reimagined apps on Windows thanks to AI and the possibility of dedicated Copilot hardware in the future.

The redesigned Copilot is unlike anything I’ve seen Microsoft release in recent years. After hiring key Inflection AI staff earlier this year, including Microsoft AI CEO Mustafa Suleyman, the new team has moved to quickly and clearly exert its influence on Microsoft’s consumer efforts. The marketing videos are slicker, more friendly, and have less of a corporate feel. The new Copilot experience itself also looks more like Inflection AI’s Pi chatbot, with new visual elements and prompts that are designed to get you interacting with Copilot more. It’s all very different from how Copilot started out inside of the Bing search engine.

“We’re moving from tool to companion,” says Mehdi in an interview with me last week. “I think about it as Copilot++… it’s an extension of what we’ve done. We started with Bing Chat, trying to revolutionize search in a world of AI. We got that underway and then we made the move to Copilot, to think about AI as a companion to help you across everything.”

Microsoft is now adding voice capabilities to Copilot so you can hold a conversation with the AI assistant. At an event in New York City earlier this week, Mustafa Suleyman demonstrated Copilot Voice to reporters and creators to show how you could use it to recall previous conversations. After using it for a few days, I’ve been surprised at its ability to hold a conversation, but I’ve been unable to get it to refer to previous conversations like a human companion would.

What’s interesting about the voice capabilities is that they really prompt you to keep going, to keep talking to Copilot about anything and everything. It’s not at the level of the human-like AI depicted in Spike Jonze’s Her, but you can certainly see why futurists predicted a decade ago that Her would be a reality by 2029. Copilot Voice even matches your behavior, so if you sound mundane or sad, it will match that tone, but if you’re excited, it’ll pick up and respond with high energy. “I think that emotional interaction is going to be a really important part of how we build the next phase of Copilot,” says Suleyman.

Voice also helps Microsoft bring back some of the personality that’s been missing in Copilot ever since the company had to pull back from the Sydney persona it had been pursuing for years. Sydney made headlines after it declared its love for New York Times columnist Kevin Roose; now, more than a year later, Microsoft says it has updated its models to bring some of that Sydney personality back with Copilot Voice.

“The tech is moving so quickly, and voice is now at a place where it’s good,” says Mehdi. “When we launched with Bing Chat, there was a whole bunch of personality with Sydney, which struck a chord with people but wasn’t quite in the zone. We’ve now been able to get that thing tuned so it really does work.”

Moving Copilot toward more of a companion that shows up everywhere also means that this AI assistant needs to understand what you see onscreen to really be smarter. Microsoft took a first step toward that with Recall, the controversial feature for Copilot Plus PCs that takes snapshots of mostly everything you see or do inside Windows. Researchers picked Recall apart, highlighting its security and privacy issues, forcing Microsoft to delay it and secure it properly.

Now, Microsoft is moving ahead with Copilot Vision, a feature that lets you hold a conversation with Copilot and allows it to see what you see on a webpage. You can use it to ask questions about the photos you’re looking at or to compare products while you’re shopping. Microsoft is also adding a Click to Do feature to Windows that’s very similar to Google’s Circle to Search. Windows search is also getting AI features to make it easier to find the images or files you need.

Copilot Vision and Click to Do feel like early work toward a future where you can summon Copilot anywhere in Windows to help you inside apps or even in Xbox PC games. Because you have to take an explicit action to enable Copilot Vision or Click to Do, I think there will be less of a backlash than what we saw with Recall automatically taking screenshots in the background. Either way, Microsoft clearly wants to warm people up to the idea of AI seeing what you’re doing and assisting you in ways far beyond a chat prompt. I wouldn’t be surprised if this is a big focus for Windows 12 or whatever comes next.

“The reason things like Recall are interesting to us, and the improved file search, is that I think we’re in a world where the superpower of Windows is user context,” says Pavan Davuluri in an interview with The Verge. “If these AI capabilities have an understanding of context and they can understand your context across things, they’re just more powerful for you for being seamless during day-to-day tasks.”

Microsoft is building capabilities like Recall or Click to Do directly into Windows because it wants them to be “pervasive, we want them to be real time and fluid,” according to Davuluri. It means that parts of Windows are now getting overhauled in interesting ways that are actually useful for daily tasks.

“Every app is starting to get reimagined. Photos is reimagined, Paint is reimagined, search is reimagined,” says Mehdi. “We’re just scratching the surface, and that’s what’s really exciting… I think you’re just getting a look at what’s possible when you look at things like Recall and Click to Do and you look at how the apps are being reimagined.”

This is the kind of stuff that gets me excited about AI, seeing where it can improve the way we use apps and Windows daily. In many ways, the current Copilot chatbot experience is still a solution looking for a problem. It’s still generating incorrect answers to questions or getting very confused at even the most basic questions. I’m more interested to see where Microsoft can take the Windows or Office AI integration to improve the apps and tools I use daily. “We want to take these simple things and have them be delightful,” says Davuluri.

While Microsoft has challenges ahead to convince Windows users that AI isn’t a security or privacy nightmare, it also faces a unique test of how it builds Copilot and Windows AI experiences. This week’s Copilot refresh is only for consumers, creating an even clearer split between Copilot and the paid Microsoft 365 Copilot that businesses use. “We were on the path of one Copilot that does everything. There’s still this idea of an integrated whole, but there’s unique Copilot experiences for unique audiences and needs,” says Mehdi.

The teams that build Microsoft 365 Copilot and the consumer Copilot are different, and I’ve written previously about how nobody really owns the totality of Copilot inside of Microsoft. By splitting these up, Microsoft is trying to “get the best of both worlds,” according to Mehdi. “Both teams can really innovate in what makes sense, and we can borrow ideas. A nice thing about having distinct teams is that each team can focus on its customer audience.”

That’s allowed the new Microsoft AI team to tackle the consumer side quickly and not have to worry about corporate needs. “We don’t have to worry that there’s a bunch of extra code for enterprises so that we have to slow the consumer app down. The apps can move at their own speed. I think it’s been a really nice sweet spot,” says Mehdi.

Microsoft will also need to use this separated Copilot consumer effort to make the AI assistant relevant beyond Windows. Apple and Google are both building their own AI features directly into mobile operating systems, pushing people away from dedicated apps.

One answer could be dedicated AI hardware. We’ve seen plenty of this new hardware emerging from startups, and even longtime Apple designer Jony Ive is now working with OpenAI on an AI hardware project after leaving Apple five years ago. Microsoft has tried and failed in the past to make its Cortana assistant work well on speakers and thermostats. “The big difference now for us vs. the past is the agent experience itself is complete enough, meaningful enough, that it gives us permission to look at other form factors and devices,” says Davuluri. “I think it will free up the ability to innovate in hardware and come out with purpose-built hardware. I think the fundamental thing is, can these things actually complete a set of scenarios, do a set of things that are magical for you that makes it worth having an additional device or another device?”

The dream of ambient computing has existed for decades, and it finally feels like Microsoft, Amazon, Google, and others now have all the necessary tools to make it a reality. “For as long as computers have been around, we’ve had to learn their language,” says Suleyman. “The burden has been on us to adapt to computers. Today, computers have learned our language, and that completely changes the interactive paradigm between us and them. We’re now in a moment where these new conversational interfaces are changing what it means to interact with a computer.”

Thanks for subscribing and reading to the very end. What would you like to see in Windows 12? You can reach me via email at notepad@theverge.com.

If you’ve heard about any of Microsoft’s other secret projects, you can also speak to me confidentially on the Signal messaging app, where I’m tomwarren.01. I’m also tomwarren on Telegram, if you’d prefer to chat there.