Playtesting like Valve

Games are the most complex software ever built, and product designers could learn a few things from how they are built

Games are the most complex software ever built, and product designers could learn a few things from how they are built. Specifically the ones from Valve. There’s a video that highlights how Valve approaches playtesting that’s worth watching:

The video distills a few lessons, like the value of testing early and often, selecting the ideal testers, and ensuring that designers run the tests themselves so they serve as a source of inspiration. I don’t want to spoil the video by providing a listicle, so please watch it. It’s worth every minute.‍

There are a few lessons I ended up incorporating into my work from this video:

Testing early and often is the only way to better work

Valve tests as early as the first week of a project. Each level of ‘Half-Life 2’ had over 100 playtesters. In every playtest, designers found clues and ideas to improve the game.

Unfortunately, testing in software design ends up being disconnected from the act of designing, done at the end, like a checkmark that every project needs.

Testing can act as a feedback loop to inspire the work and lead to better problem-solving, and designers should lead the testing process.

A great example from the video of this in action: In the VR ‘Half-Life: Alyx,’ designers noticed that players kept covering their mouths to avoid noise in a level where a giant monster tries to chase the player. They then embedded that idea into the game, leading to one of its most memorable parts.

Testing will adjust your course but not tell you where to go

Testing can only tell the current state of the thing you’re building, not where it should go. That’s up to the team developing it, and it requires a commitment to a vision and a target audience.

During the development of the game ‘Portal,’ hardcore FPS players wanted more action during the game’s final level. But the idea ended up being a mistake:

“The vast majority of playtesters who had gotten used to the slower-paced, cerebral nature of Portal were just frustrated, confused, and dissatisfied"

– Erik Wolpaw, Valve

Teams often hear that users want X, Y, or Z. Listening to them is critical. But take their suggestions mindlessly, and you might end up with a Frankenstein that pleases no one.

Users can provide clues to a great product, but it’s up to the team to use that feedback to inspire their work, not dictate it.

Do not release if you’re not happy with it

Valve does not release their games until they’re satisfied with what they see during playtesting. You could argue that it’s a rare position – after all, they generate over $10 billion from Steam alone each year. If you read about the development of their first game, Half-Life, and the process is the same.

In tech, we can improve something over time through updates. Testing can be done in the open more efficiently than ever.

An unintended consequence of that practice is a lessening of quality standards. “Just ship it; we’ll improve later.” Unless a product team has a high bar of quality in place already, that moment rarely comes. And that bar starts to erode the moment you make compromises.

It’s a hard thing to balance. But holding a standard is as important as shipping. If done well, releasing often can be a fantastic way to improve a product. Do it without a vision and a quality standard, and it all becomes busywork with few results.