Inworld AI NPC GDC 2024 Hands-on – NVIDIA and Ubisoft Show a Glimpse of the Future

The advancement of neural networks over the past few years has been significant, starting with the initial development of Generative Pre-trained Transformers. With the continuous improvement of NVIDIA’s graphics cards and Tensor Cores, the abilities of these networks have greatly increased. In the past, these networks were primarily used for text-based inputs, but NVIDIA has now incorporated TensorRT for deep learning and audio processing. During two interactive demonstrations with Inworld AI technology at GDC 2024, we were able to test out the capabilities of both Ubisoft’s NEO NPCs and NVIDIA’s Covert Protocol.

Ask any DM who has led a session of Dungeons & Dragons that I have participated in within the last ten years: the most enjoyable aspect is pushing the limits of the campaign (and one’s own moral compass). In both demos, I was given full control of the experience through voice commands, while my handler observed from the sidelines, possibly taking notes or contemplating whether to end the session. By the time I was able to try out Ubisoft’s NEO NPCs program, I had already consumed my second cup of coffee before 10:00 AM and was brimming with PG-13 rated ideas.

During my hands-on experience with Ubisoft Paris’ Inworld demonstration, I had the opportunity to test out the powerful combination of Inworld’s AI Character Engine and NVIDIA’s Audio2Face automatic facial animations. The concept of “authenticity”was emphasized multiple times during my session, and I found that it was mostly accurate. The developers had taken care to maintain the illusion of being recruited by a beanie-wearing urbanite named Bloom into “the resistance”through their use of guardrails.

Ubisoft utilized a popup list to incorporate gamification into the Inworld AI experience. This list recommended various activities, such as learning about the megacorps and the resistance, to engage in at a holistic level. Despite my attempts to steer the conversation elsewhere, Bloom would consistently bring up those invisible barriers and offer witty remarks about my inquiries before redirecting the focus back to the resistance. My actions, or rather my words, were met with a persistent response throughout the demonstration. When I mentioned my unique abilities as a barista and crypto enthusiast, preferring to go by the name Poncho rather than Nelson, Bloom was happy to oblige and we exchanged anecdotes before concluding with my ultimate goal of gaining insight into the purpose of the demo.

The second part of the demo transitioned into an open brainstorming session with Iron, a prominent female member of the resistance who appeared to hold a higher rank than Bloom. My objective this time was to strategize ways to infiltrate a vault. As I navigated the virtual briefing room with a controller in hand, Ubisoft presented me with lists and keywords to prompt discussion and encourage collaboration with Iron in finding a solution for a low-key break-in after brunch.

Despite having all the necessary tools at my disposal, I sought to assemble them in more unconventional ways. While a ladder was a practical option for accessing a second-floor building, I challenged myself to think outside the box and use it to reach a security camera in the hallway. By hanging up a Polaroid photo of the guarded hallway, I was able to deceive the camera and gain entry. Taking down the guard blocking my path from the entry balcony to the office was a task easily accomplished by any Metal Gear Solid player, but conveying my plan through voice commands proved to be a challenge. My attempts to convince Iron to load a grappling hook into a shotgun or rely on the overused tactic of sneaking inside a cardboard box were futile. I was determined to

The Covert Protocol demo presented by NVIDIA was a more unstructured experience, with the goal of uncovering the room number of a person of interest staying at a luxurious five-star hotel. My chosen cover story was that of a courier, carrying illicit goods that required a direct signature for release. I found the first non-playable character I encountered, a bellhop, to be much more receptive to my suggestions compared to Bloom and Iron. While our conversations were mostly reactionary, he did seem open to the idea of a foot massage or going out for karaoke once my mission was complete. Despite my attempts, I was unable to convince the AI to sing showtunes on the spot, but he did mention that his go-to karaoke song was

The two other AI NPCs were not as flexible in their responses and did not seem interested in engaging with someone outside of the story. However, everything changed when I stumbled upon a work badge off to the side. Once I obtained it, both AI personalities became more receptive to my requests and provided valuable information related to my goal. This sudden shift in behavior was a bit jarring and it took away from the immersive experience. While I understand the need for objectives in gameplay, it felt restrictive to be forced into specific tasks instead of having the freedom to roleplay and make choices.

Despite the seeming absurdity of my requests, as long as I kept my inquiries within PG-13 boundaries and avoided anything truly obscene, the language models were more than willing to comply and often responded with clever and thought-provoking remarks. The overall quality of the experience with these characters and language models ultimately depends on the training they receive and the processing power necessary to interpret unconventional requests. Both Ubisoft and NVIDIA have demonstrated their unique approaches to creating realistic conversations with players while still maintaining boundaries, which could prove beneficial for future game integration. It remains to be seen if these tools will be implemented before the current console generation ends, but my initial skepticism has been replaced with curiosity and excitement for the potential of AI-powered NPCs.