AI-Unity Project

Overview

This project is an ongoing effort to understand how the advancement of LLMs might be applied to interactive media projects. With the rise of consumer use of generative AI models like Chat-GPT or Microsoft Co-Pilot (or the many many others out there), I found it interesting that there hasn’t been a similar surge in their use for games and XR projects. My preliminary findings suggested that several tools are offered for doing so, mostly centered around the game engine Unity. I took to exploring potential integrations using 3rd party tools, and ultimately decided upon the free and open source LM Studio to send and receive prompt data between the two softwares.

I initally prototyped using Godot, a game development engine I have more familiarity with, to assess the project’s viability. Screenshots of this initial attempt are included in the following section.

Following the successful deployment of the project in Godot, I recreated the project again in Unity. Since then, I’ve completed several upgrades ranging from user experience to prompt engineering and to simply improving performance. Now, as I near the final version of this initial prototype, the project will begin to split into a multitude of sub-projects.