OpenAI finally launches screen and live video observation for paying ChatGPT users
OpenAI today announced that it was finally giving ChatGPT the ability to observe screens and live video to provide comments and feedback, a day after Google LLC launched similar functionality in Gemini 2.0.
The functionality, which essentially gives ChatGPT eyes or sight, was first teased in May at the launch of the multimodal GPT-4o model, with fans waiting for the functionality since that time as OpenAI has rolled out other features.
With the new functionality, unveiled in a live stream today, ChatGPT can recognize objects via a camera, be it on a smartphone or webcam and react to what is on a user's computer screen, as well as anything open on a user's computer or phone.
In a demo from OpenAI, researchers turn on the screen-sharing feature to allow ChatGPT to provide real-time feedback during a graphic design project. ChatGPT identified on-screen elements such as colors, layouts, and text, offering actionable suggestions to enhance the design.
The live video feature works similarly. When a user gives access to the camera on their device, ChatGPT processes the video feed in real-time to identify and analyze objects, movements and contexts. The information is then used to provide relevant insights or instructions, such as identifying a piece of hardware in a tech setup or giving step-by-step guidance during a repair.
Additionally, OpenAI introduced a Santa voice feature, which allows users to interact with ChatGPT in a festive tone, complete with holiday-themed responses. The seasonal addition has been designed to add a layer of engagement and fun, particularly for families and children.
During the demonstration, ChatGPT's Santa voice delivered personalized holiday stories and answered questions about Christmas traditions.
While the features sound great, anything that uses live video or has live access to screens is going to raise security concerns.
To address the concerns, OpenAI noted that the new functionality uses guardrails to protect user data. For live video and screen-sharing, all data is processed locally whenever possible to ensure that sensitive information remains private. Additionally, OpenAI added that any data shared with the cloud is encrypted and not stored beyond the session's duration.
The new features are being rolled out to ChatGPT Plus and Enterprise users starting from today, with wider availability planned for early next year. However, free users will not be getting access to the live video and screen-sharing capabilities due to OpenAI wanting to focus on monetizing its premium offerings.