RenderDoc

If you're working with OpenGL you NEED to learn renderdoc. Download it and try it out.

Challenges with debugging with RenderDoc

During early rapid development, you'll encounter bugs which RenderDoc may help you out. However this requires you to manually launch your application using RenderDoc. Beginners tend to have issues specifing the correct working directory and executable location. On top of this RenderDoc can't help if the bug your hunting down isn't easy to reproduce.

Wouldn't it be helpful to ALWAYS have RenderDoc attached to your application and your debugger at the same time? This can be done! Plus we can get some pretty cool extra features too such as customizing RenderDoc keybinds and triggering captures within code.

Loading the RenderDoc shared library

Attaching RenderDoc can be pretty simple; Just load the RenderDoc shared library before you create your rendering context.

C and C++ don't have a standard way to do this so we must either use third party libraries or manually use OS apis. For this tutorial I will demonstrate the process using both Windows and Unix operating system APIs.

We're going to define a few files to make implmentation easy across platforms. The .inl files will be conditionally included into the .cpp depending on the platform.

• rdoc.hpp
• rdoc.cpp
• rdoc_win.inl
• rdoc_linux.inl

The header and source files are pretty simple.

rdoc.hpp

void setup(bool load); // Used to attach and detect renderdoc

// Uses the renderdoc apis
bool is_attached();
bool is_target_control_connected();
bool is_frame_capturing();
void trigger_capture();
void launch_replay_ui();

rdoc.cpp

#include "rdoc.hpp"
#include <renderdoc_app.h> // found in your renderdoc installation directory

namespace {
	RENDERDOC_API_1_0_0* gApi = nullptr;
}

#ifdef APP_WINDOWS // True for windows
#	include "rdoc_win.inl"
#endif

#ifdef APP_LINUX // True for linux
#	include "rdoc_linux.inl"
#endif

void setup(bool load) {
    attach_shared_lib(load); // Load shared library
    if (!is_attached()) return;

    // If attached, configure
    gApi->MaskOverlayBits(eRENDERDOC_Overlay_None, eRENDERDOC_Overlay_None);
    gApi->SetCaptureOptionU32(eRENDERDOC_Option_DebugOutputMute, 0);
}

bool is_attached() {
    return gApi;
}

bool is_target_control_connected() {
    if (!gApi) return false;
    return gApi->IsTargetControlConnected();
}

bool is_frame_capturing() {
    if (!gApi) return false;
    return gApi->IsFrameCapturing();
}

void trigger_capture() {
    if (gApi) gApi->TriggerCapture();
}

void launch_replay_ui() {
    if (gApi) gApi->LaunchReplayUI(true, nullptr);
}

Inside the following inl files use os specific apis, since these are conditionally included, the headers and apis dont need preprocessor guards.

rdoc_win.inl

#include <format>

#include <Windows.h>
#include <shlobj_core.h>

namespace {
	void attach_shared_lib(bool load) {
        // Is the shared library already loaded (ie: Lauched via renderdoc)
		HMODULE library = GetModuleHandleA("renderdoc.dll");
		
        // If not and we want to attach the shared library
        // Attempt to do so
		if (load && library == nullptr) {
            // Fetch program files directory
			CHAR pf[MAX_PATH];
			SHGetSpecialFolderPathA(nullptr, pf, CSIDL_PROGRAM_FILES, false);

            // Attempt to load the shared library
			library = LoadLibraryA(std::format("{}/RenderDoc/renderdoc.dll", pf).c_str());
		}

        // If the library is still not attached then renderdoc is unavailable
		if (library == nullptr) return;

        // Fetch the renderdoc api function and load them
        // See: https://renderdoc.org/docs/in_application_api.html
		pRENDERDOC_GetAPI getApi = (pRENDERDOC_GetAPI)GetProcAddress(library, "RENDERDOC_GetAPI");
		if (getApi == nullptr) return;
		getApi(eRENDERDOC_API_Version_1_0_0, (void**)&gApi);
	}
}

rdoc_linux.inl

#include <dlfcn.h>

namespace {
    // Control flow is nearly identical to using the windows api except dlopen already knows about library paths
	void attach_shared_lib(bool load) {
		void* library = dlopen("librenderdoc.so", RTLD_NOW | RTLD_NOLOAD);
		if (load && library == nullptr) library = dlopen("librenderdoc.so", RTLD_NOW);
		if (library == nullptr) return;

		pRENDERDOC_GetAPI getApi = (pRENDERDOC_GetAPI)dlsym(library, "RENDERDOC_GetAPI");
		if (getApi == nullptr) return;
		getApi(eRENDERDOC_API_Version_1_0_0, (void**)&gApi);
	}
}

Conclusion

Having the ability to load and customize how RenderDoc behaves directly in our application provides a great flexability. We can load renderdoc based on anything. (As long as that decision is made before your context is created.) This means for debug builds we can always attach RenderDoc. We can implement our own frame captures. Save caputure files. Loading RenderDoc like this also avoids the confusing topic of working directories as your current tools still control this.

Theres much more that you can do, but with basics covered I'll leave that as a topic for your own exporation.

Reference implementation

VulpEngine has an implmentation of this.

https://github.com/anthofoxo/vulpengine/tree/master/src/rdoc