Generative AI Transforms Virtual Characters

Exciting advancements in generative AI technology are reshaping the world of gaming, from conception to production to gameplay. Game developers are now exploring the impact of these cutting-edge technologies on the creation of 2D and 3D content. One particular area of excitement lies in the ability to create dynamic gaming experiences in real-time, pushing the boundaries of what was previously possible.

The development of non-player characters (NPCs) evolved as games became more sophisticated. The sheer number of pre-recorded lines, interactive options, and realistic facial animations has increased. However, interactions with NPCs often feel scripted and transactional, with limited dialogue options. Now, generative AI is revolutionizing NPCs by enhancing their conversational skills, creating evolving personalities, and enabling dynamic responses tailored to each player.

At the recent COMPUTEX 2023 event, NVIDIA unveiled the future of NPCs with the groundbreaking NVIDIA Avatar Cloud Engine (ACE) for Games. This revolutionary custom AI model foundry service empowers game developers, middleware providers, and tool creators to infuse intelligence into NPCs through AI-powered natural language interactions.

The ACE for Games platform offers a range of optimized AI foundation models to generate NPCs, including:

• NVIDIA NeMo: This foundation language model provides game developers with the tools to further customize models for their characters. The models can be seamlessly integrated end-to-end or in combination, allowing specific character backstories and personalities to fit perfectly into the game world.
• NVIDIA Riva: Offering automatic speech recognition (ASR) and text-to-speech (TTS) capabilities, Riva enables real-time speech conversations with the NeMo model. Note that you can experience the wonders of speech synthesis firsthand by exploring QuData's free text-to-speech services, allowing you to effortlessly convert text into natural-sounding speech.
• NVIDIA Omniverse Audio2Face: This remarkable feature instantly generates expressive facial animations for game characters using just an audio source. With Omniverse connectors for Unreal Engine 5, developers can effortlessly add lifelike facial animations to their MetaHuman characters.

To breathe life into NPCs, the NeMo model alignment techniques come into play. By utilizing behavior cloning, developers can instruct the base language model to perform specific role-playing tasks. To align NPC behavior further, reinforcement learning from human feedback (RLHF) can be employed to receive real-time feedback from designers during the development process.

Once the NPC is fully aligned, NeMo Guardrails can be applied. This toolkit adds programmable rules to ensure NPCs behave accurately, appropriately, and securely within the game. NeMo Guardrails natively supports LangChain, a toolkit for developing applications powered by large language models (LLMs).

To showcase the power of ACE for Games, NVIDIA collaborated with Convai, a startup specializing in creating and deploying AI characters in games and virtual worlds. By integrating ACE modules seamlessly into their offering, Convai leveraged NVIDIA Riva for speech-to-text and text-to-speech capabilities, NeMo for conversational language modeling, and Audio2Face for AI-driven facial animation. Together, they brought the immersive NPC Jin to life within Unreal Engine 5 and MetaHuman.

Excitingly, game developers, among which is also Absolutist, are already embracing NVIDIA's generative AI technologies. Stay tuned for thrilling updates and indulge in captivating gameplay enhancements that are sure to elevate your gaming experience.