A curated list of Reassembly bugs that were fixed after release

Every game developer strives to release a perfect game. Many bugs are hard to predict, especially for indies with only a few computers and no experience. I wanted to document some of the things I missed in the hope that other developers will not miss them. These are especially relevant for native (i.e. C++) games which are a little closer to the platform than e.g. Unity games. Almost all of these bugs were fixed during our Early Access period and before the “official” release.

The vast majority of Reassembly work/bugs during the Early Access period related to gameplay elements. The bugs listed here are selected for possibly relevancy to other games and are mostly in the platform layer. A complete list of changes is available on the Steam Announcement Page.

Localization/unicode related

• Steam usernames are UTF8 and frequently contain ƃᴉzɐɯ∀ st☢ff. Reassembly uses Google’s Droid Sans Fallback and Open Sans Emoji to provide additional characters for rendering usernames, and SDL_ttf or Cocoa/NSFont to actually render the text, depending on the platform. In an ironic twist of fate, the first version of this article was lost because wordpress choked on a U+E10D (rocket) character.
• The game may be installed in e.g. C:/вещи/Steam/SteamApps/…. so any assets must be loaded using unicode paths. OSX and Linux use UTF8 for everything and are easy to support but Windows builds must convert to wchar_t* UCS2 encoding via WideCharToMultiByte/MultiByteToWideChar or similar.
• Non-English versions of Windows return native-language error messages in GetLastError/FormatMessage. This means that any logging mechanisms must support unicode. Printf/sprintf on windows support a %ls format specifier for wide character strings but will crash on non-ascii (thanks to Russian alpha tester Krypt for helping me track down this bug).
• Games that hardcode WASD will not be playable on many Non-English keyboards. Keybinding systems must support unicode keyboard input.

GPU/Graphics related

• Vsync on/off/adaptive options are probably a good idea. Adaptive Vsync (aka tear control) is a good solution but is not supported on all drivers. Many players have old/slow GPUs that won’t achieve 60fps and consequently would have a bad experience under Vsync. Some players have 144hz monitors and would like to take advantage of them.
• Should the game launch in a window, in “true” fullscreen mode, or in a borderless fullscreen window (fake fullscreen)? True fullscreen mode has performance and especially smoothness advantages, while fake fullscreen allows easier alt-tabbing. Reassembly simplifies video monitor modeset issues by always using the same fullscreen mode as the desktop, and defaults to true fullscreen.
• Some people have their desktop set to 16 bit color depth. This may cause OpenGL context creation to fail if you request a 24 bit framebuffer.
• Significant numbers of Steam users have GPUs that only support OpenGL 2.1. Things like framebuffer objects, floating point framebuffer formats, and various functions may not be available or may only be available through older EXT variants (check GL_EXTENSIONS).
• Some people have broken or seriously out of date graphics drivers. This is mostly a support problem but…
• Various GPUs do a poor job of implementing certain OpenGL features. For example, antialiased (GL_LINE_SMOOTH) lines look terrible on many ATI GPUs. Adding video options to the game allows players to work around unforeseen problems.
• Many people still have 4:3 monitors. If the game resizes UI elements according to the current aspect ratio, make sure to test at 4:3 (in addition to 16:9, 16:10).
• Many people will disregard minimum system requirements and try to run the game on very old/slow computers. Detecting the OpenGL version and at least popping up a sympathetic dialog is much better than crashing in these cases. Adding options to reduce CPU/GPU usage at the const of graphics quality will also allow many people play your game that otherwise could not.

Hardware/System related

• Test the game on hard drive systems and SSD systems, particularly if any kind of asynchronous/streaming system is involved. Hard disks can be VERY slow – you may need loading progress bars in places were SSD systems would not.
• Different gamepads have different polling intervals, button mappings, deadzone requirements, etc. Test at least PS4, PS3, Xbox 360, and Xbox One controllers. With an event based API like SDL_GameController, make sure to poll for ALL events every frame instead of just processing one, or events can get very backed up. Consider analog stick deadzone and smoothing options. Test mouse and keyboard functionality in UIs with a gamepad plugged in, in case stray gamepad events interfere. Make sure to handle the case where a gamepad is detected but initialization fails, or where the gamepad is unplugged/plugged in during gameplay.
• 5 button mice are common. Make sure you don’t e.g store mouse button state in a bool[3] array with unrelated variables immediately afterwards.
• Key bindings should probably still work even with caps-lock on.
• Many Steam users still use Windows XP. Supporting XP/Vista/7/8 with one build involves selecting the “_xp” toolkit in Visual Studio and using GetModuleHandleEx/GetProcAddress for any APIs that were introduced after XP, with fallbacks. Make sure 3rd party DLLs also support XP.
• Test any scroll wheel functionality on mice AND touchpads. Apple touchpads in particular can generate scroll events at a very different rate than traditional mice.

Steam/Steamworks related

• There is a 1GB/1000 file limit on Steam Cloud saves. Exceeding this limit results in lost save game data.
• Many Steamworks API calls (achievements/stats APIs, cloud save APIs, etc. including SteamAPI_RunCallbacks()) seem to grab a global mutex and consequently can block for long periods (multiple frames) of time if another thread is for example writing a large file using the API. My solution was to only ever call steam APIs in a non-critical-path thread.
• Many API calls (ISteamRemoteStorage::GetFileNameAndSize, ISteamRemoteStorage::FileExists) can be very much slower than OS equivalents. If you call these frequently or depend on their performance, it may make sense to maintain a cache of results.

Crash handling

Reassembly implements a crash handling and reporting system similar to Mozilla/Google Breakpad that has been invaluable in improving quality. Via the Win32 API call SetUnhandledExceptionFilter, it is possible to catch NULL pointer dereferences and other errors that would otherwise crash the program. The program can then collect a stack trace and log file and upload it to a server before popping up an apologetic message and actually crashing. The Reassembly code for this is on github (including a similar implementations via signals for OSX and Linux). We don’t ship .pdb files but include the .dll load addresses to enable associating symbols with the memory addresses in the stack trace.

A CGI script that accepts file uploads can run on any web hosting solution and contain less than 100 lines of code. Compressed HTTP uploads are available through libcurl or dozens of other easy to use libraries.

By collecting (frighteningly) large numbers of crash reports, we were able to triage and fix the most frequently occurring crashes first. It also quickly became obvious that only a small percentage of players report their crashes via email, steam discussion boards, forums, or other explicit methods, and that crashes so reported are not always representative.

This method was very effective for fixing crashing of all kinds, and particularly for race conditions and other bugs that can be hard to reproduce without specific hardware.

It’s very important to anonymize collected log files, including removing information like the current system username which may be present in paths.

Professional QA

Indie Voyage, our Kickstarter/publishing partner, hired a professional QA tester (Chris Watkins) in the month leading up to release. He uncovered a large number of UI, tutorial, and general user experience problems and greatly increased the game quality. In retrospect I would have allocated more of our Kickstarter budged to this area and started sooner.

One of the dangers of Early Access and similar community-driven systems is that players quickly learn the game and focus on advanced end-game features. Players that are put off by the initial experience do not contribute to the community. Focusing development on the end-game is valuable but can leave new players confused.

Final thoughts

Despite my best efforts Reassembly is still not 100% bug free. I’m very proud of how close we have come, and hope the accumulated knowledge is useful to someone.

I did search for a service that would automatically run the game across hundreds of machine and OS configurations looking for problems but was unable to find one. We had a system (actually, at least three different systems) like this while I was at Nvidia for testing the driver and various internal tools and it was great.

Hardware dependent graphics problems are a persistent issue. The cross product of GPU/driver/OS is very large. Actual crashes are great in the sense that they produce stack traces and can usually be worked around directly by e.g. checking GL_EXTENSIONS more carefully or avoiding undefined behavior. Rendering glitches – parts of the scene mysteriously not visible, unexplained z-fighting, etc. – are more insidious. I have not found a good way to debug these besides guessing or buying the GPU in question and hoping to be able to reproduce.