I have been using Webpagetest.org for a while now. My projects also needed a way to measure rendering performance in addition to network and content delivery optimizations that webpagetest does.
I created a tool for that and thought that folks on this forum may be interested in it. The tool is called browser-perf (browser-perf - npm) and runs tests similar to Chromium perf tests (telemetry).
A hosted instance of WebPageTest is great, and taking the cue from it, I also created a hosted instance of the browser-perf. The site is available at http://perfmonkey.com and has a way to try out the tests on web pages. You can collect metrics like mean frame time, time taken for paints, etc.
You can try the tests and see the results on [b]Perfmonkey.com is for sale | HugeDomains
I am working on creating a system where people can signup and view these performance results over time in a graph, to identify the commits that made the pages janky. Would greatly appretiate feedback on this effort.