In recent years, the term "reinforcement learning" has become popular.
Just like machine learning as a whole, reinforcement learning, a field of machine learning, is attracting attention because
That's understandable.

First of all, what is reinforcement learning?

And what makes it better than other machine learning solutions? This article examines reinforcement learning from a "real-world problem" perspective and provides examples of its applications in the games and entertainment industry.

Model real-world problems

If you think about when humans actually do some work in the real world, you will find several keywords. In any work, the agent, that is, the person doing the work and the agent who achieves the goal, must always grasp the situation in which they are placed and find information that is useful for the work or that will hinder the work. It is necessary to carry out a series of workflows in order to complete the task while being aware of external factors that may occur.

For example, in the job of food delivery, the delivery person (agent) must follow the correct route (understand the state) and advance along the road (take action) in order to reach the customer's home (achieve the goal). execution). You get the biggest reward when you arrive at the customer's doorstep and deliver the food, but if you take a wrong turn or get stuck in traffic along the way and can't complete the job, you pay the price. .

Let's look at a simple example to help you understand how to translate the keywords explained above into a game. In the game "Mario Bros.", the main character Mario must achieve a certain goal by performing a predefined series of actions. Mario must reach the flag, collecting coins and items along the way and avoiding enemies and pitfalls (penalties).

The formal mathematical definition of reinforcement learning is as follows.

At point 1.to, the agent does not know what action to perform, so it may perform random actions or adopt other strategies if it has prior knowledge.

2. At time step tₜ, the agent executes action (Aₜ).

3. At the next time step (tₜ ₊ ₁), the agent recognizes the new state (Sₜ ₊ ₁) and also considers the reward (Rₜ ₊ ₁) it received from the environment as a result of its action.

4. If the reward is low, the agent adjusts to choose another action.

5. This process repeats until the agent completes running the episode.

Application of reinforcement learning in the gaming industry

From the examples above, it's easy to see how this definition of reinforcement learning fits perfectly into many real-world problems. Games are also often based on real-world scenarios, making them perfect applications for reinforcement learning. The previous section introduced a simple game application, but now let's take a look at how reinforcement learning is used in the gaming industry.

Game testing automation

Imagine an FPS like the screenshot above, rather than a simple game like the Mario example. There are nearly infinite patterns of actions an agent can perform. You can walk, run, shoot, change weapon types, reload, manipulate objects and items, and the list goes on.

In traditional game testing, beta testers spend dozens of hours playing the game to make sure there are not just any coding bugs, but also bugs in the specifications (such as a ledge being too high for players to jump on). It was normal to check that the game was working properly. By automating gameplay, testers only need to set the time for the AI to complete the game, and if the AI is unable to complete the game, the tester can log why it was unable to complete the game. All you have to do is check.

Automate game tuning

Another common use of reinforcement learning is tuning the behavior of NPCs and enemies. For example, in fighting games, developers typically want to create a variety of characters with different skills and abilities, but equally powerful overall. By using reinforcement learning, it is possible to automate battles between AI characters and collect statistics on their wins and losses. If stats are heavily biased toward a particular character, developers can adjust that character's abilities or damage dealt. All of these tests can be automated, significantly reducing the tester's burden.

Source: "Virtua Fighter"

If technology like this had been available a long time ago, many more games would have been released each year.

Industry and market conditions

Unfortunately, introducing reinforcement learning into real-world game development workflows is fraught with numerous challenges, and creating a common platform and best practices is difficult. At this point, there is no major company offering reinforcement learning services to the gaming industry, although most gaming companies have started developing some kind of machine learning solution in-house. The biggest obstacles to widespread adoption of common reinforcement learning platforms and pipelines in the games industry must be overcome:

• Challenges in introducing reinforcement learning
• Reinforcement learning behavior is unpredictable

As with all areas of machine learning, it has become increasingly complex for humans to fully understand why AI agents behave the way they do. For game designers who want to provide a consistent gaming experience, this poses a barrier to implementing AI in game characters.

Reinforcement learning architecture design works in conjunction with game design

The actions agents can perform vary by game, genre, and company.
And of course, the environment also plays a big role in reward design. Given these constraints, designing reinforcement learning architectures also requires access to the game design and source code. That can cause a lot of problems, but the biggest problem is that for most gaming companies, their code and assets are their most valuable intellectual property. Many game companies are very reluctant to share code and try to do all development in-house.

Reinforcement learning requires a large amount of computation for inference.

Games that use reinforcement learning can only be run on high-performance game consoles or PCs. Even if the purpose is different, it is desirable for the device to have a GPU of some capacity. However, many regular gamers play games on smartphones that have low specs or consume too much battery power to perform reinforcement learning inference, making it difficult to widely implement reinforcement learning. you can't. Therefore, game companies will also need to invest in parallel programming and various optimizations.

What game companies can do to reduce the unpredictability of reinforcement learning

Game developers can introduce reinforcement learning in less important parts of their games. Testing the most common, low-level enemy behaviors or testing only limited areas of the game are good candidates for limited reinforcement learning implementations.

What companies other than game companies are doing to accelerate the advancement of reinforcement learning

As gaming companies have been reluctant to share code and knowledge, many have begun offering tools and platforms for companies in other fields to build their own reinforcement learning systems. OpenAI and DeepMind Lab are leading companies that provide reinforcement learning tools and platforms that enable engineers to develop and test customized reinforcement learning strategies. On the other hand, game engine manufacturers such as Unity and Unreal also offer reinforcement learning libraries that can be easily integrated into their respective software. These efforts have made it much easier to get into reinforcement learning, especially for game developers.

summary

In this article, we explained the basics of reinforcement learning and how problems similar to most problems encountered in the real world occur in reinforcement learning. Next, we looked at some applications of reinforcement learning in the game industry and the challenges faced when implementing reinforcement learning into game development workflows. Finally, we touched on some ways to alleviate the challenges and discussed the state of adoption of reinforcement learning in the gaming industry.