Introduction

In this series of posts, I’ll discuss some ideas about how to unit test your game code if your game uses Unreal Engine.

In my recent experience in games industry, I noticed that there’s a lot of misunderstandings about what unit test is and how to write unit tests. Mostly I believe, comes from difficulties from the C/C++ toolset, libraries and a little bit of culture as well. The idea of this blog series is to give a different perspective on the topic.

What unit test means?

Like a lot of things in software development, it depends. I’m personally very comfortable with the definition of the agilists, people like Kent Beck, Robert C. Martin (uncle Bob), etc. But instead of trying to find a definition, I’ll try to explain my theory about this epic misunderstanding.

In a bit more prescriptive methodologies, commonly associated with waterfall (mid 90’s) like the Unified Process (The Unified Development Software Process), there’s the implementation phase, where some of the activities are the coding and the unit test. In this context unit test meaning the tests that the developer does to make sure his bit works. The “Test” phase contains an integration test, which means the system as a whole is tested from the user perspective (black box testing).

Later on (late 90’s), Kent Beck published “Extreme Programming Explained“. Most of the ideas we use today like continous integration (which evolved to continuous deployment and continuous delivery) and I’d say even Scrum (since what is described as the planning game in XP is not very different to Scrum, although there are some differences), came from this book. This is probably the first publication that defines the term Test-Driven Development (TDD). The idea here is to write write an automated test, specifying which is the behaviour he expects to do and then write the code that implements that behaviour, then incrementally improving this code not only with new requirements, but with refactoring. The red-green-refactory approach. Also, since it is first publication TDD imply small increments or baby steps.

When test automation started trending, a lot companies and people with waterfall mindset started automating tests, but not doing TDD. These tests, although valuable, usually contains a lot of dependencies, like an entire application that tests one part of the code. This was called by this group of people: unit test. I’ll call “waterfall unit tests” in this section, just to try to remove the ambiguity between both approaches which are completely different. The people that embraced the TDD approach also called their tests unit test. I’ll call these agile unit tests, just for simplicty. And here is the ambiguity.

As the TDD concept evolved, a lot was taking into consideration about how the test-first approach influences the design and makes code more cohesive. There are studies like Realizing quality improvement through test driven development: Results and experiences of four industrial teams that discuss how TDD improves the complexity and cohesion of the source code.

Some of the long term issues you will find with waterfall unit tests are:

• The tests are fragile. Because they rely on the behaviour of a lot of classes, if any of those classes changes, it is likely that the tests will break. Fragility will eventually cause maintainability problems as well.
• The tests takes a long time to run. Instead of milliseconds, the tests takes seconds to run. When you start having thousands of tests, it is not uncommon for them to take hours to run.
• The tests are flaky. Because they rely on files, infrastructure, configuration, etc, they tend to fail a lot for environmental problems.

With agile unit tests you won’t have many of the issues mentioned before, because:

• A unit test should aim to test one single behaviour of a single class, removing any dependencies.
• Because they don’t have dependencies, they will run fast. If you don’t hit files, I/O, network, threads, they should be faster.
• Because they don’t have dependencies, they will be easier to maintain. Technically you change one class, you should have to change one test class and no other tests that seems to not have any connection with other tests.

But on the downside, will introduce you some new problems:

• Because you validate unit behaviour, if the interaction between the units changes, they will break. That’s why some integration tests are recommended.
• The unit test will ensure that you code work as expected. If your expectation is wrong, you will have bugs.
• Your code will have a lot of new interfaces.

Unreal Test Framework and Waterfall x Agile Unit Tests

Unreal has its own test framework, which starts the engine and enable you to spawn (how they call creating an instance in games) actors in the world. Actors usually rely on some of the code generation magic done by Unreal Engine (see Understanding Unreal Build Toolsfor details).

Although it is quite handy to use the test framework which gives you a lot of things for free, it will encourage you to bring your dependencies with your code instead of isolating your code from your dependencies, since Unreal Engine is a dependency itself.

Also, Unreal Test Framework is built on top of the engine, what means your linking times will be really long. To achieve a good workflow with unit tests, is really important to have quick feedback from new tests you write. If takes 5-10 minutes everytime you change and run your tests, it is very likely that TDD will be a burden, not a tool for you.

This is the reason why my suggested approach will be using Google Tests instead of Unreal Test Framework. If you what you want to achieve is integration tests (or waterfall tests) building and running the whole engine, stop reading now, otherwise you will just waste your time.

From this point on, I won’t mention “waterfall unit test” and every time I mention “unit test”, I mean “agile unit test”.

Why Google Test

Google Test is a very popular test framework in C++. It is one of the few that supports a good mocking framework (Google Mock) that can do the job. Unfortunately, Google Mock is not easily integrated with other test frameworks. The same approach can be done with other test frameworks.

But the key thing here is avoiding bringing Unreal Engine dependencies with the tests and ensuring a fast TDD cycle. This will makes sense as we discuss compilation, linking and dependency inversion in the later posts.

Conclusion

I hope this post helps you to understand the motivations of this series and also clarify which unit tests I’ll mean in this series. For people practicing TDD this may seem to be quite unnecessary, but because I’ve experienced this confusion so many times, specially in the games industry, that’s why I decided to clarify up front.