Feature Request: Urlblast auto registration

WebPageTest Private Instances
#1

Not sure if this is the correct place for this, but…

How difficult would it be to have an auto registering urlblast agent?

The idea would be that you simply point your urlblast application to a host running the web page test server and it would auto register that agent and add it to the list of available locations. You could either have the user input a label and location in urlblast, or use the geo location of the requesting agent when adding to the location list.

Also, if the remote agent doesn’t ping the server periodically that location would be removed from the list.

We are setting up a server that will likely have many remote agents. This would be most helpful from the perspective of managing the server configuration.

Thanks,

Tony

#2

It should be doable without too much work. I obviously wouldn’t want to do that for the public instance but it does make some sense for private agents (actually, auto-hiding agents that haven’t checked in in a while is on my list to do even for the public instance as we sometimes have boxes go down where there is only a single test machine). On the public instance it might also make sense to allow for auto-registration but with a secret key exchange of some kind.

I wouldn’t want to use the geo-location info though. For it to be useful the agent will also need to advertise it’s connectivity which should probably be configured, not detected (though detecting the browser version would be possible).

#3

Additionally it would be handy if the agent would pass in some basic info upon registering.

Like…

CPU:
RAM:
OS:
Browser:
…

#4

I’m thinking of modifying the code to support wild cards in specifying where the locations are configured.

This would be used in conjunction with the auto registering agents. The idea would be that when an agent registers instead of modifying the locations.ini it would create a new file like locations_keller.ini.

If the remote agent doesn’t check in periodically this file gets deleted, which would result in it no longer showing up on the “test” page.

Can you think of any issues I might run into?

Thanks.

#5

There are a couple of issue that come to mind (when it comes to implementing anyway):

  • When a location checks for work, the code checks to make sure the location is valid. This is where I’d probably recommend doing the registration
  • When the test submission page is loaded it pulls the list of locations (other than adding the logic to pull in the other locations this can pretty much stay as is)
  • When work is submitted the location is checked to make sure it is valid and to decide where to store the job file. This is where things get a bit tricky because technically you could have a location disappear between when the test page is loaded and when the request is submitted

The other thing that needs to be considered is finding a good place to add the code that removes idle locations. There is support for a cron-job based request that checks all of the locations and sends a mail alert if a location hasn’t checked in in a while - that might be a good place to put it. You could also do it on every visit to the test page but that seems like an awful lot of checking (maybe would work well for a private install that doesn’t get a lot of traffic).

#6

Ok. I have auto registration working. It needs some more tweaking, but so far it works quite nicely.

The server settings no longer have to be updated for each remote agent. That process is handled automagically now. :slight_smile:

I discovered that you can have several params in the location= param in urlblast.ini.

My example…

urlblast.ini

…
location=Keller_TX&connectivity=FIOS&down=25000000&up=15000000&latency=5&browser=IE+8
…

I added some settings to settings.ini

// Used to enable agent auto registration
enableAutoRegister=1
// The key expected from the agent
autoRegisterKey=akey
// The base location for the jobs
autoRegisterLocalDir=/var/www/webpagetest/work/jobs/

So, in getwork, if autoregister is on, I grab all the params and create a file like…

locations_Keller_TX.ini

[Keller_TX]
browser=IE 8
down=25000000
up=15000000
connectivity=FIOS
latency=5
label=Keller_TX
localDir=/var/www/webpagetest/work/jobs/Keller_TX

I moved the retrieval of locations to a function in common.inc

function GetLocations()

This function will find all files that match locations_*.ini and process them. It builds and returns the expected array.

Also, on the “New Test” tab/page, I check to see if the agent has checked in in the last 5 minutes. If not, the ini file gets deleted and it won’t show up. As soon as the agent hits getwork again the file is recreated and he shows up.

If two agents come in with the same location, they will collide. With a little more work I could add support for multiple agents from the same location, but I’m not sure yet if that’s needed. I’ll just have my users confirm that they didn’t use an already used location name.

I hope to get this code back into the main repo. Do you prefer that I send a patch file for your review when I’m done?

btw. Is latency just an estimate, or is it calculated by some means?

Thanks,

Tony[hr]
If you’d like to test it, do the following…

In you urlblast.ini

Url Files Url=http://71.170.186.242/work/

Set the location param to something like.
*** Replace the items like Keller_TX, FIOS, etc. with your info.

Location=Keller_TX&connectivity=FIOS&down=25000000&up=15000000&latency=5&browser=IE+8
Location Key=br549

Fire it up and go to http://71.170.186.242 and try it.

#7

Yes, please send a patch and I can get it merged into the mainline. I’ll probably want to enhance it to detect Dummynet and support dynamic bandwidth as well (a recent addition).

The latency is the measured RTT for the first hop and configured manually (tracert from the test machine out and checking the latency to the ISP).

I can do the multiple agent/same location work when I merge if you’d like. I actually have 8 test systems running in Dulles and they basically get work in a first-come first-served basis. I’ll also see if I can eliminate some of the absolute paths that are in the configs and generate them dynamically (that’s one of the nagging things I wanted to clean up anyway).

Thanks for working on this!

#8

One thing just came to mind…

You might want to update the “last checked time” every time results are posted back as well. If someone does 10 test runs first & repeat view of a slow page it could take as long as 20 minutes to go through them and the location would hide itself while working on the test.

#9

That’s a good point. I was thinking of making the timeout a settings.ini configurable.[hr]

Yeah, I guess as long as the multiple agents are all from the same area it’s fine for them to share the work.

#10

This is just what i was looking for! :slight_smile:

Has this feature been implemented yet, or can someone send over a patch to me also?

I’m just in the process of setting up agents, and these changes would help now and in the future! :slight_smile:

Is there a tool or script that auto cleans jobs older than a month for example from a private wpt instance?

Thanks

Tom

#11

Nope, hasn’t been implemented (at least in the mainline code). Deleting old jobs (or tests) should be a pretty simple shell script that just deletes files older than X from the appropriate directory tree.

#12

Ok no probs, i’ll ping you an email if you could send across the patch back? :slight_smile:

I’ve already started on a little script that will clean up the results and remove them from the log files too, but i’m going to try and have this configurable - similar to show slow.

When running WPT Monitor i noticed it was quite slow, and it turns out the request for ‘jquery.validate.js’ no longer exists within the jquery dev channel, so i’ve modified this locally so both js files load from the same cdn for now (reduce extra dns lookups):

<script type="text/javascript" src="http://ajax.aspnetcdn.com/ajax/jQuery/jquery-1.7.2.min.js"></script>
<script type="text/javascript" src="http://ajax.aspnetcdn.com/ajax/jquery.validate/1.9/jquery.validate.min.js"></script>

Not sure if you prefer using publicly hosted js libraries via a cdn, or (my preference depending the type of site) keep them within the app so network issues or missing files don’t cause further problems?

Would be great if i can have access to the svn repo on google code, as id be more than happy to help :slight_smile:

Shall send you my google details in that email also!

Thanks

Tom

#13

Hi there,

That would be an awesome feat.
What should be the trigger that include this into main stream ?
If there’s any way to give a hand on this one…

Regards,

–
Regis A. Despres

p.s. : wonderful peace of software. thanks a lot for all this.