Navigating Unity as an ML Practitioner: A Personal Journey

The following article will focus on my first experiences as an ML practitioner in Unity, a popular game engine. First, we’ll start by introducing game engines and Unity and how they work. Then, we’ll review my learning process and the resources I used to start my journey in Unity (as a machine learning guy). In the third section, we’ll take a closer look at my first “game” in Unity, providing a concrete example of the challenges and triumphs I encountered along the way. And finally, in the fourth section, we’ll discuss my plans for further learning and development.

Whether you’re a beginner or an experienced developer, I hope this article provides valuable insights and inspiration.

Understanding Game Engines and Unity

A game engine is a software framework designed to develop video games. It typically includes relevant libraries and support programs and offers game development tools and features.

The goal of a game engine is to give developers the ability to code their games, including rendering elements of the game, animating and implementing logic for elements like in-game AI, creating UI and cinematics, and building games to be run on consoles or PCs. Game engines are also used in the movie industry.

If you want to learn more about the topic, many useful resources are available, such as this video from Linus on the subject.

In the industry, there are different game engines; there are some that I have in mind (but it’s just a subset):

Unity From Unity technologies (of course)
Unreal Engine from Epic
Godot
Amazon Lumberyard from Amazon
Snowdrop, Anvil at Ubisoft
…

Game engines can be broadly classified into two categories: proprietary engines, like the ones used by Ubisoft, and open engines, like Unity. Both open and proprietary game engines have their benefits and drawbacks, which can influence a developer’s decision about which engine to use for a particular project.

Here are some pros and cons to consider:

Using an open engine (like Unity and unreal), at some point, you will have to give them some of your sale money if you are publishing your game and starting to make money. For example, consider the two popular games, PUBG (developed by PUBG Corp using the Unreal engine) and Fortnite (developed by Epic using the Unreal engine). These two games are competitors, but every sale of PUBG also generated revenue for Epic because PUBG was built using the Unreal engine.
Developing a proprietary game engine can be challenging, as it requires ongoing maintenance and documentation. However, it can also offer benefits such as avoiding revenue sharing with the engine provider and greater flexibility to customize the engine to meet specific project needs.

To conclude the first section on game engines, let’s talk about Unity. As mentioned earlier, Unity was developed by Unity Technologies and was first released in 2005. It is now widely used in various industries, including robotics, AI, and the film industry. One example of its involvement in the film industry is Unity’s investment in Weka, the company behind many VFX in movies such as the Marvel movies and the Lord of the Rings movies.

For those more interested in video games, Unity is also used in acclaimed games like Cuphead or Ghost of a tale.

From the developer side (and my perspective), Unity is centred around its editor where Scene (that can be seen as level) can be built and composed from different objects with script in C# attached to it to embed the logic.

As an ML practitioner, why did I choose to use Unity? Firstly, I have always been interested in the gaming industry, even before working at Ubisoft so Unity was under my radar. Additionally, I had previously explored Unity during sharing sessions or hackathons at my previous job at EDF (big up to Jean-François HULLO), but I never really took the time to dive deep into it. However, I purchased books, assets, and training materials during discounts on platforms like Humble Bundle and Udemy, “just in case.”

Recently, I wanted to try something new for my side projects and prototype with Unity as a side project. It may also give me ideas for my job at Ubisoft. So let’s see my learning processes/resources.

Learning Unity

So to start to learn Unity, I took three different axes in the following order:

Online course
Youtube videos
ChatGPT

The online course was The Ultimate Guide to Game Development with Unity (Official), created by Jonathan Weinberger from the GameDevHQ in collaboration with Unity. The course was published in 2019 and is still up to date (just the UI that can be different but nothing impossible).

The course is divided into three sections:

Building a space shooter
Making cinematics
Building a 2D platformer.

For my training, I mainly focused on the first section with the space shooter to get the basics of Unity (but I had a quick look at the other sections, which looked good). The goal of the space shooter was to implement the control, enemy spawning and power-ups (code the power-ups), and there was a split; first of all, build a prototype with simple objects in Unity, and after that, add animation /sprites etc. (making more like a game than a prototype).

I enjoyed this split, and the prototype part is enough for my experiment with Unity (I am not aiming to become a game developer). Still, there is a good explanation of Unity concepts (I will go further in the next section). It took me around 10-15 hours to experiment with this course, and I highly recommend it to a beginner (because there are some exercises to complete on each video of the course).

I still have other courses to dig into in the future that looks good also (1,2,3), but I decided to go the youtube way to support me in my first experiment.

On Youtube, there is excellent channels, but for me, the best one was Brackeys (the channel is closed now), where the team explained different Unity concepts applied to develop a specific kind of game. An excellent example to illustrate that is the one around coding player movement in an FPS game.

The channel closed a long time ago but the concept presented are still align with th reality of Unity. It uses essential Unity functions, whereas some good channels, like Code Monkey, rely on custom packages with helpers functions. Youtube is a good source of information, but I suggest following online courses that are more structured.

And finally ChatGPT, it was the hot thing in the past weeks, and it seems to be a revolution, so I wanted to get my hand on it to see if it can be helpful in my learning ( I think I will do a more general article on LLM, but I need to level up on this topic).

There are a few good articles in the Verge on the topic to show some applications of it:

Generally, it’s broad; you can do a lot of stuff, from coding to getting recipes or training programs (only the sky is the limit, or human imagination and training sets). If we go back to my learning of Unity, there are some of my takeaways on ChatGPT:

It can generate some code in C# after providing a prompt; my experience was that the structure was good, but when executed in Unity, the logic did not work.
The model can provide suggestions to improve the code that you are providing, and it’s pretty good at it
The application can also give you pieces of advice to set up the settings of your project and more details on functions of Unity (like more intelligent search in the documentation)
The cool thing is the fact that you can start different chats, and a model like that will keep the context of it (so you can come back later and continue your discussion), really good when you work on different subjects
There is the ability to evaluate also the quality of the answer with a simple thumbs up, thumb down

Overall, it’s pretty impressive, but it’s far from being the perfect AI that will replace all coders in the world. However, as a developer and writer, a model like ChatGPT can be a good assistant on a day-to-day basis (I’m using it right now to review this article and get some feedback loop between Grammarly and chatGPT).

With all these resources, it’s time to get my hands dirty and work on a game.

Creating a Game in Unity: Challenges and Lessons Learned

So for my first game, I wanted to have something simple that reminded me of my first video game moments, and I had these two games in mind: Doom, the first FPS I played and Pac-Man, an excellent old arcade game with simple logic. I also wanted to include a resource-collecting element similar to games like Minecraft or Valheim.

With that in mind, I came up with the following design for the game:

The fundamental concepts of this experience are

The player can move along the x and z axis like in doom, and it can shoot an energy bullet.
Energy bullets can also be shoot by an enemy; if an enemy bullet is touching the player, it’s losing energy but if a player’s bullet is touching an enemy, he disappears.
Every movement or energy bullet shot cost energy (but not moving is also costing energy), but it can collect energy pods (in green on the schema) to increase its energy level (these energy pods are popping randomly)
There are enemies spawning on the level; if the player is in their detection area, they are rushing on it and shoot at it (so it generates energy level decrease to the player)
If the player kills an enemy with a bullet, it can collect an energy pod. Still, if the player comes into contact with the enemy (or vice versa), the enemy disappears, and the player loses some energy.
If the player reaches an energy level of 200, it’s entering in “overdrive mode,” which allows the player to move faster and come into contact with enemies without losing energy. But if the energy level exceeds 300, overdrive mode is over, and the player’s energy level goes back to 100 as a penalty.
The more time the player survives, the better score he will get.

With this concept, I started to build a prototype; I am not going to enter too many details of the game (I can share the code if you want), but there are still a few findings (as a data scientist starting to code with Unity) that I want to share with you :

There is a concept in C# of public and private variables that offer the ability to make a variable available of an entity available to another one. Very useful for the counters, for example.
My project was structured with the following arborescence between the script en entities.

I don’t know if it’s the most efficient, but I like the idea of decomposing the logic of energy level in PlayerCore, from the movement etc.

I keep the logic of a spawn manager to handle the energy pods and enemies with the concept of Coroutine with IEnumerator.

A coroutine is a function in Unity that can pause and resume later. It’s helpful in doing things that take a long time or happen over multiple frames, like waiting for something to happen or repeating an action. It’s a very particular concept that I initially had some issues getting into, but it’s pretty simple.

Emoji can not be handled in the Text component 😭
Game behaviour in the Unity editor is not the reality.

During my development, I was testing the game in the editor, and it was running great, but when I was building the game, the execution locally was super laggy during movement. I have a decent setup at home, so it came from the game. Online or on chatGPT, the first advice was about modifying the player settings or the time attribute, which could be better. I dug a bit on the subject, and two optimizations came out, like capping the game to 60 FPS (done on the spawn manager, cf screenshot) and building a smoothing function in the PlayerMovement script (thanks chatGPT on this one).

With all of that, it’s to show off there is a record of a session of the PC version of the game.

If you are excited to play with it, there is the ability to build a WebGL version of the game and deploy it in Unity connect (you need to install the correct package). The link for the game is HERE , Enjoy :)

Future Plans

It has been a while since I last followed an online course, and the material for this Unity course was excellent. Building my first game was challenging, but I learned more about game development practices (but it was just a toe in the game dev water) and coding in general.

With all of that, what are the next steps for me in this topic:

Bring some AI into Unity with the ml-agents package to build agents with reinforcement learning on top of Unity.
Explore ML applications around level generation, game adjustment, etc.
Explore new mechanics for this experience (add a day-night cycle, cooperation mode with an AI, etc.)
Studying more the data side of a game, like implementing tracking to build ML service on top of it
Explore other game experiences; I recently got hooked by Marvel Snap or Vampire survivors, so why not build copycats to make ML in it?

Hopes that you enjoyed the reading.