Private instance custom server settings

We’ve been running a private instance for a couple of years now and its occurred to me that whenever we run into any server/controller related problems, we’ve had nowhere to turn to for known errors and fixes specific to WebPageTest.
With that in mind, I’d like post our current setup along with certain tweaks for problems we’ve run into, and I’m hoping others can also post.

We run about 8k tests a day on about 50-60 Agent VMs.

WinServer 2012 R2 100GB RAM 3TD SSD
Apache3 + Php 5.6

Error: Apache Restarts with 3221226356 exit code
Seems to be a windows apache specific issue with the default thread size being too small. Started seeing this with tests that have a huge amount of user timings due to an error from web devs.
Set mpm_winnt_module->ThreadStackSize to 16MB to mitigate

Error: “The test was not started but the test job could not be found. Forcing it to end.” message in the test logs. Still not 100% sure about the cause, but I’m pretty sure its related to high disk i/o when trying to read test info from files on disk. As a mitigation, I’ve added a retry to FindJobPosition in and it has reduced the occurrence to one or twice a month.

Some tweaks:
Use azure storage file shares for archiving. You can mount these shares as a network drive, and point to it in settings.ini. I’ve also heard that you can use the builtin AWS support to use these azure shares, but I haven’t tried it yet.

Use a script to dynamically generate locations.ini file based on db entries. Not quite satisfied with how we do this, I would prefer to create some sort of UI which can also manage Agent settings to make it easier to create/delete/move/configure locations. Interested to see how others do this, especially for large systems where manually editing configs would get tedious.

Create small .php pages to get some settings like contents of connectivity.ini, mobile_devices.ini, and all data in testinfo. Some of this info isn’t available via REST calls, and its cleaner to query for settings to use them when calling runtest.