Every WebPageTest instance is essentially a relay server so there’s nothing you need to do specifically to enable the functionality. Just set the server up so you can run tests directly on it and then you can set up the client instance that relays tests to it.
The main issue comes if you want to lock the relay server down (i.e. if it is running outside of a firewall). In that case there are a few options:
1 - Enable API keys (settings/keys.ini) so anything using the API requires an API key (relayed tests go through the API)
2 - Make the server API-only by adding headless=1 to settings/settings.ini
3 - Set up firewall rules so that only the agents and the client can connect to it
The client WPT server submits a relay test to the relay WPT instance which runs the test like normal but stores the results in a separate location. When you check the client to see if the test is done, it checks with the server and if the test is complete the full test is downloaded off of the server (and deleted) and only kept on the client instance.