Microsoft’s Copilot Vision Analyzes Your Screen for Enhanced AI Assistance

Today, Microsoft introduced an innovative feature called Copilot Vision as part of its new Copilot Labs. This initiative allows Microsoft to pilot new AI functionalities with a limited group of users to gather valuable feedback and enhance the features. Copilot Vision bears resemblance to the video capabilities demoed by OpenAI earlier this year, offering users real-time assistance tailored to the content displayed on their screens.

Currently, Copilot Vision is exclusively available within Microsoft Edge. Once activated, users visiting supported websites can engage with Copilot by asking questions related to the page’s content in real-time. The feature not only assists with answering queries but can also suggest subsequent actions to help with ongoing tasks. The ability to speak your queries naturally, rather than typing, makes it feel as if an expert is sitting beside you, offering help as you navigate.

Microsoft has designed Copilot Vision with a strong emphasis on privacy. This feature operates on an entirely opt-in basis, meaning it is only enabled when users specifically opt for it. Importantly, Microsoft assures users that no data—including audio, images, text, or conversations from Copilot Vision—is stored or utilized for model training. After your Vision session concludes, all associated data is permanently erased.

Moreover, Copilot Vision is not applicable on websites that feature paywalls or sensitive content. During this preliminary stage, Microsoft has restricted Copilot Vision to a select list of pre-approved websites, with plans to broaden its availability in the future, always prioritizing user safety and responsibility.

It’s crucial to clarify that Copilot Vision does not act as an agent; it cannot perform actions on your behalf. Instead, it functions as a helpful companion, answering questions throughout your browsing journey.

Copilot Vision is now being rolled out to a select group of Copilot Pro users in the United States. This feature has the potential to transform how we engage with the web, creating a more intuitive and supportive browsing experience. As Microsoft further develops and expands this revolutionary feature, we can anticipate an even more effective Copilot experience in the near future.

Source