Microsoft Introduces Windows 11 Taskbar Feature to Share App Windows with Copilot

Microsoft has seamlessly integrated Copilot into the taskbar of Windows 11, a feature that is now rolling out to users. By simply hovering over apps in the taskbar, you can share the window of that application with Copilot. This allows for interactive support, such as receiving suggestions on how to respond to emails in Outlook. While it may seem trivial, the functionality behind Copilot Vision is significant.

I recently discovered the “Share with Copilot” option on my Intel Lunar Lake Copilot+ PC. However, this feature is being made available to all users, regardless of whether their PCs are classified as “AI” systems. This inclusivity means that users with standard AMD or Intel setups will also see the “Share with Copilot” option on their taskbar, further enhancing accessibility.

Understanding the “Share with Copilot” Feature

During our examination, we noted that the “Share with Copilot” feature appears when you hover over any open application on the taskbar, including those like Cloudflare WARP, a VPN service. Notably, even applications with DRM-protected content, such as Netflix, display this feature, although functionalities may not operate as expected for copyright reasons.

Upon sharing an application window, Copilot employs cloud-based AI to analyze the displayed content and provide actionable insights. For instance, when I opened an email in Outlook and shared the window, Copilot could read the content but was unable to navigate to other emails or perform any clicks on my behalf.

Copilot offering hands-on assistance on Windows 11

Copilot is limited by visibility; it can only see what you see and therefore can provide guidance based on that. When I inquired about how to reply to an email, Copilot directed me to locate a nonexistent “send” button, demonstrating a limitation known as “hallucination.”However, when prompted to explain the process, Copilot’s response improved. It activated a virtual cursor, indicating precisely where I needed to click.

Fortunately, Copilot took corrective measures by highlighting the correct ‘reply’ option on the screen instead of the suggested ‘send’ button. When I then asked how to delete the email, it correctly pinpointed the appropriate action.

Microsoft refers to this function as “guided assistance.” According to their official documentation, Copilot Vision aims to help users effectively, though it is noted that this feature will not function with DRM-protected or sensitive content.

Mechanics Behind the Integration of “Share with Copilot”

As reported by Windows Latest, Microsoft utilizes a specific Windows API designed for communication applications to integrate the “Share with Copilot” feature into Windows 11’s Shell, which includes the taskbar and sharing user interface. This API enables the display of “Start sharing”and “Stop sharing”commands while accurately passing the selected WindowId to the chosen app.

This functionality is primarily intended for applications like Microsoft Teams, but Microsoft is broadening its application to incorporate Copilot as well. Importantly, this API is not available for general developer use; it is labeled as a Limited Access Feature, typically requiring prior approval from Microsoft for access and implementation.

If organizations like OpenAI wish to replicate this functionality, they would need to seek approval from Microsoft to use the API. This exclusivity is the reason why we currently see the “Share with Copilot” feature implemented solely for Copilot, with Microsoft having the ability to provide access to select partners.

Source&Images