Microsoft Copilot can now read your screen, think deeply, and speak aloud to you
A week after announcing a wave of updates for its enterprise suite of Copilot AI-powered products, Microsoft is launching new Copilot capabilities on Windows for all users, including a tool that can understand and respond to questions about what’s on your screen.
Refreshed Copilot apps for iOS, Android, Windows and the web are rolling out today, and all feature a Copilot with a more “warm” and “distinct” style, as Microsoft describes it. Microsoft is also bringing the chatbot to WhatsApp, letting users chat with Copilot via DM, similar to the experience you get with other bots on Meta’s messaging platform.
Copilot Vision has a view of what you’re viewing on your PC — more specifically, a lens into the sites you’re visiting with Microsoft Edge. Gated behind Copilot Labs, a new, Copilot Pro-exclusive opt-in program for experimental Copilot capabilities, Copilot Vision can analyze text and images on webpages and answer queries (e.g., “What’s the recipe for the food in this picture?”) about them.
Vision, which can be pulled up by typing “@copilot” in Edge’s address bar, isn’t exactly a technical marvel. Google offers similar search technology on Android, and recently brought bits and pieces of that tech to Chrome as well.
