Microsoft 365 Copilot’s Researcher Agent Gains Ability to Take Action on ‘Computer Use’

Microsoft 365 Copilot Enhances Researcher Agent with New Capabilities

In June, Microsoft introduced an innovative tool dubbed the Researcher agent as part of its Microsoft 365 Copilot suite. This advanced functionality enables multi-step research utilizing OpenAI’s sophisticated research model, enhanced by the orchestration and in-depth search features available within Microsoft 365 Copilot.

Recently, on October 2023, Microsoft made an exciting announcement regarding an update to the Researcher agent. The newly introduced “Computer Use”capability empowers the agent to perform actions on behalf of users by leveraging a secure Windows 365 virtual machine (VM) to access password-protected online content. This means that the Researcher agent can now retrieve subscription-based resources and authenticated reports, such as those from Gartner or Forrester, provided the user has the required credentials.

https://www.youtube.com/watch?v=N3vLF9mnd8w

Researcher with Computer Use (https://www.youtube.com/watch?v=N3vLF9mnd8w)

Getting Started with Computer Use Capability

To activate this new Computer Use feature, users simply need to select the corresponding option in the Researcher prompt box. Once this feature is activated, the Researcher agent gains access to various tools including a visual browser, text browser, terminal, and Microsoft Graph. It is important to note that access to enterprise data is disabled by default when this capability is turned on.

Functionality of the New Feature

Here’s a breakdown of how the Computer Use feature operates:

Researcher Orchestration: This component connects the AI with the necessary tools, managing workflows and providing updates on the AI’s thought processes back to the user.
Temporary Virtual Computer: The AI will create a secured virtual computer within the Microsoft Cloud when its actions require interaction, such as form submissions.
Sandboxed Execution Environment: This virtual computer operates in a safe, isolated state, ensuring that no personal information is stored or shared, and runs only during the session.
Multi-Tool Access: Within the sandbox, the AI can employ a web browser for surfing, a terminal for executing code, and a text browser for scanning and searching web pages efficiently.
Network Safety Protocols: Each action conducted online is screened using safety classifiers to prevent potentially hazardous or irrelevant activities.
Visual Feedback: Users can monitor the AI’s activities in real-time via screenshots and outputs displayed in a desktop interface.
User Control: Should there be a need for login or manual entry, users can securely take control of the sandbox through screen sharing.

The Researcher with Computer Use capability is currently being rolled out to customers within the Microsoft 365 Copilot Frontier program. This marks a significant stride in the evolution of AI-driven research tools.

Source & Images