Mac/Linux Support for Agent

WebPageTest Private Instances
1

Hello,

I was trying to install private instance on my mac,
and realize that Windows was the only machine to setup the test agent.
Is this going to support Mac/Linux machine for future?

thanks!

2

The only thing that supports Mac/Linux right now is the NodeJS agent for controlling a tethered Android phone. Technically it also supports running Desktop Chrome but it doesn’t really get any testing.

We don’t have any near-term plans to support either Mac or Linux, most of the active development is on Mobile right now (though I’ve been toying with creating a cross-platfrom Chrome extension that could run as an agent which would support Chrome on all platforms, including Chrome Books but won’t be able to do video capture).

3

Hi @pmeenan,

Just came across this thread. Were you able to complete the chrome extension that would allow cross-platform testing?

Thanks

4

Hi Pat,

Is there any rough documentation anywhere on how to go about setting up nodejs to work on desktop chrome? Im trying to get the agent running here but its trying to run adb commands which are causing a fail.

Any guidance on this would be appreciated if its possible, It would be great to get an agent running on a Linux platform.

Cheers

Sam

5

Docs for nodejs are of course in /agent/js/README, looks to be working for me so far! Awesome software :slight_smile:

6

FWIW, Linux falls into the rough, uncharted territory where the code worked at one point, may still work but isn’t something I use regularly so it may rust.

7

Hi, Is there a agent version that support linux desktop chrome?I would appreciate it if anyone help me , thank you in advance!

8

I see there is a new Test location “Dulles, VA - Linux (Experimental)”, with Chrome.

Is it some experimental Linux agent?

9

Yep. It’s Chrome-only but it’s in good shape (and as of Friday runs all of the mobile agent testing as well): GitHub - WPO-Foundation/wptagent: Cross-platform WebPageTest agent

10

I’m not able to make it work. I mean, the python code works and sends requests, but testing does not proceed.

My locations.ini:


[locations]
1=Office
default=Office

[Office]
1=WPT_Chrome
label="Test - Original Location Office"

[WPT_Chrome]
browser=Chrome
label="Original Location - Office"

I run python code as follows:

$ python wptagent/wptagent.py --server http://localhost/work/ --location Office  --xvfb -vvvvv --shaper none
fatal: Not a git repository (or any of the parent directories): .git
Python 2.7.12
12:17:01.701 - Waiting for Idle...
Running agent, hit Ctrl+C to exit
12:17:01.702 - Checking for work: http://localhost/work/getwork.php?f=json&shards=1&location=Office&pc=webpagetest&freedisk=244.905
12:17:01.706 - Starting new HTTP connection (1): localhost
12:17:01.711 - http://localhost:80 "GET /work/getwork.php?f=json&shards=1&location=Office&pc=webpagetest&freedisk=244.905 HTTP/1.1" 200 0
12:17:06.718 - Checking for work: http://localhost/work/getwork.php?f=json&shards=1&location=Office&pc=webpagetest&freedisk=244.905
12:17:06.720 - Resetting dropped connection: localhost
12:17:06.725 - http://localhost:80 "GET /work/getwork.php?f=json&shards=1&location=Office&pc=webpagetest&freedisk=244.905 HTTP/1.1" 200 0
12:17:11.727 - Checking for work: http://localhost/work/getwork.php?f=json&shards=1&location=Office&pc=webpagetest&freedisk=244.905
12:17:11.729 - Resetting dropped connection: localhost
(...)

This one doesn’t return any data:

$ curl -v "http://localhost/work/getwork.php?f=json&shards=1&location=Office&pc=webpagetest&freedisk=244.905"
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /work/getwork.php?f=json&shards=1&location=Office&pc=webpagetest&freedisk=244.905 HTTP/1.1
> Host: localhost
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 30 Mar 2017 12:22:10 GMT
< Server: Apache/2.4.18 (Ubuntu)
< Access-Control-Allow-Origin: *
< Set-Cookie: o=2cc9445c61c0a6eea3fcbed67a6cd7e3c7762a5b; expires=Fri, 30-Mar-2018 12:22:10 GMT; Max-Age=31536000; path=/
< Cache-Control: no-cache, must-revalidate
< Expires: Sat, 26 Jul 1997 05:00:00 GMT
< Content-Length: 0
< Content-Type: text/plain;charset=UTF-8
<
* Connection #0 to host localhost left intact

A call to /install/ shows that no agents are connected:

Test Locations

Office : Test - Original Location Office
WPT_Chrome : Original Location - Office - No Agents Connected

Not sure how to debug this.

11

The agent should be configured to point to WPT_Chrome, not Office. Office is just a container that groups together (possibly multiple) actual location ID’s.

12

Yeah! Finally something happens :slight_smile:

I see it fails to connect to localhost:9222, defined here:

internal/webpagetest.py:                task['port'] = 9222

Here is the output from the agent:

15:24:46.784 - Preparing browser
15:24:46.784 - Terminating all instances of google-chrome-stable
google-chrome-stable: no process found
15:24:46.794 - /usr/bin/google-chrome-stable --disable-background-networking --no-default-browser-check --no-first-run --process-per-tab --new-window --disable-infobars --disable-translate --disable-notifications --disable-desktop-notifications --allow-running-insecure-content --disable-component-update --disable-background-downloads --disable-add-to-shelf --disable-client-side-phishing-detection --disable-datasaver-prompt --disable-default-apps --disable-domain-reliability --safebrowsing-disable-auto-update --disable-background-timer-throttling --disable-sync --host-resolver-rules="MAP cache.pack.google.com 127.0.0.1","MAP clients1.google.com 127.0.0.1" --window-position="0,0" --window-size="1024,768" --remote-debugging-port=9222 --user-data-dir="/usr/src/wptagent/work/webpagetest/browser.170330_KY_A.1" --disable-gpu --host-resolver-rules="MAP cache.pack.google.com 127.0.0.1","MAP clients1.google.com 127.0.0.1" --window-position="0,0" --window-size="1024,768" --remote-debugging-port=9222 --user-data-dir="/usr/src/wptagent/work/webpagetest/browser.170330_KY_A.1" --disable-gpu
15:24:46.798 - Starting new HTTP connection (1): localhost
15:24:46.798 - Connect to dev tools Error: HTTPConnectionPool(host='localhost', port=9222): Max retries exceeded with url: /json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7978896810>: Failed to establish a new connection: [Errno 111] Connection refused',))
15:24:47.301 - Starting new HTTP connection (1): localhost
15:24:47.302 - Connect to dev tools Error: HTTPConnectionPool(host='localhost', port=9222): Max retries exceeded with url: /json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f79788383d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
15:24:47.804 - Starting new HTTP connection (1): localhost
15:24:47.805 - Connect to dev tools Error: HTTPConnectionPool(host='localhost', port=9222): Max retries exceeded with url: /json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7978838310>: Failed to establish a new connection: [Errno 111] Connection refused',))
15:24:48.308 - Starting new HTTP connection (1): localhost
15:24:48.309 - Connect to dev tools Error: HTTPConnectionPool(host='localhost', port=9222): Max retries exceeded with url: /json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7978838350>: Failed to establish a new connection: [Errno 111] Connection refused',))

(google-chrome-stable:23940): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
/usr/bin/dbus-launch terminated abnormally without any error message

(google-chrome-stable:23940): LIBDBUSMENU-GLIB-WARNING **: Unable to get session bus: Failed to execute child process "dbus-launch" (No such file or directory)
15:24:48.811 - Starting new HTTP connection (1): localhost
15:24:48.812 - http://localhost:9222 "GET /json HTTP/1.1" 200 374
15:24:48.813 - Dev Tools tabs: [{"description":"","title":"New Tab","url":"chrome:\/\/newtab\/","webSocketDebuggerUrl":"ws:\/\/localhost:9222\/devtools\/page\/2135b47a-7b19-4e07-a49a-b7842a7db8d8","type":"page","id":"2135b47a-7b19-4e07-a49a-b7842a7db8d8","devtoolsFrontendUrl":"\/devtools\/inspector.html?ws=localhost:9222\/devtools\/page\/2135b47a-7b19-4e07-a49a-b7842a7db8d8"}]

Did I forget to run anything which should be listening on port 9222?

13

Try pointing it directly to the chrome binary in browsers.ini:

[Chrome]
exe=/opt/google/chrome/chrome

I’ll update the install docs to point that out (been considering auto-detecting the common locations and not requiring browses.ini by default anyway)

14

Actually (even though the binary should be changed) it looks like it connected after all at the end of the log where it listed “Dev tools tabs”. The failures before that are normal as it retries until the browser comes up.

Did it not do anything after the dev tools tabs line?

15

I had:

$ cat browsers.ini
[Chrome]
exe=/usr/bin/google-chrome-stable

ls -l /usr/bin/google-chrome-stable
lrwxrwxrwx 1 root root 32 Mar 28 22:27 /usr/bin/google-chrome-stable -> /opt/google/chrome/google-chrome

Changing that to this one didn’t really help:

[Chrome]
exe=/opt/google/chrome/chrome

Here is a full log, after I’ve added “example.com/1” for a test:

$ python wptagent/wptagent.py --server http://localhost/work/ --location WPT_Chrome  --xvfb -vvvvv --shaper none
fatal: Not a git repository (or any of the parent directories): .git
Python 2.7.12
15:35:24.127 - Waiting for Idle...
Running agent, hit Ctrl+C to exit
15:35:24.128 - Checking for work: http://localhost/work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.374
15:35:24.133 - Starting new HTTP connection (1): localhost
15:35:24.137 - http://localhost:80 "GET /work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.374 HTTP/1.1" 200 0
15:35:29.145 - Checking for work: http://localhost/work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.369
15:35:29.147 - Resetting dropped connection: localhost
15:35:29.151 - http://localhost:80 "GET /work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.369 HTTP/1.1" 200 0
15:35:34.158 - Checking for work: http://localhost/work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.369
15:35:34.159 - Resetting dropped connection: localhost
15:35:34.162 - http://localhost:80 "GET /work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.369 HTTP/1.1" 200 0
15:35:39.164 - Checking for work: http://localhost/work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.369
15:35:39.169 - http://localhost:80 "GET /work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.369 HTTP/1.1" 200 0
15:35:44.176 - Checking for work: http://localhost/work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.369
15:35:44.178 - Resetting dropped connection: localhost
15:35:44.183 - http://localhost:80 "GET /work/getwork.php?f=json&shards=1&location=WPT_Chrome&pc=webpagetest&freedisk=247.369 HTTP/1.1" 200 191
15:35:44.195 - Job: {"processResults":1,"runs":1,"bwIn":5000,"plr":0,"latency":28,"url":"http:\/\/example.com\/1","Capture Video":1,"IPAddr":"10.192.1.11","bwOut":1000,"orientation":"default","Test ID":"170330_YX_B","browser":"Chrome"}
15:35:44.195 - [{'record': True, 'command': 'navigate', 'target': u'http://example.com/1'}]
15:35:44.200 - Preparing browser
15:35:44.200 - Terminating all instances of chrome
chrome: no process found
15:35:44.218 - Clearing profile /usr/src/wptagent/work/webpagetest/browser.170330_YX_B.1
15:35:44.219 - /opt/google/chrome/chrome --disable-background-networking --no-default-browser-check --no-first-run --process-per-tab --new-window --disable-infobars --disable-translate --disable-notifications --disable-desktop-notifications --allow-running-insecure-content --disable-component-update --disable-background-downloads --disable-add-to-shelf --disable-client-side-phishing-detection --disable-datasaver-prompt --disable-default-apps --disable-domain-reliability --safebrowsing-disable-auto-update --disable-background-timer-throttling --disable-sync --host-resolver-rules="MAP cache.pack.google.com 127.0.0.1","MAP clients1.google.com 127.0.0.1" --window-position="0,0" --window-size="1024,768" --remote-debugging-port=9222 --user-data-dir="/usr/src/wptagent/work/webpagetest/browser.170330_YX_B.1" --disable-gpu
15:35:44.228 - Starting new HTTP connection (1): localhost
15:35:44.228 - Connect to dev tools Error: HTTPConnectionPool(host='localhost', port=9222): Max retries exceeded with url: /json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fcdcc92ca50>: Failed to establish a new connection: [Errno 111] Connection refused',))

(chrome:24352): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
/usr/bin/dbus-launch terminated abnormally without any error message

(chrome:24352): LIBDBUSMENU-GLIB-WARNING **: Unable to get session bus: Failed to execute child process "dbus-launch" (No such file or directory)
15:35:44.730 - Starting new HTTP connection (1): localhost
15:35:44.731 - http://localhost:9222 "GET /json HTTP/1.1" 200 2225
15:35:44.732 - Dev Tools tabs: [{"description":"","title":"CryptoTokenExtension","url":"chrome-extension:\/\/kmendfapggjehodndflmmgagdbamhnfd\/_generated_background_page.html","webSocketDebuggerUrl":"ws:\/\/localhost:9222\/devtools\/page\/7722bfa3-06ab-4ca5-8a51-99344da294d8","type":"background_page","id":"7722bfa3-06ab-4ca5-8a51-99344da294d8","devtoolsFrontendUrl":"\/devtools\/inspector.html?ws=localhost:9222\/devtools\/page\/7722bfa3-06ab-4ca5-8a51-99344da294d8"},{"description":"","title":"Google Network Speech","url":"chrome-extension:\/\/neajdppkdcdipfabeoofebfddakdcjhd\/_generated_background_page.html","webSocketDebuggerUrl":"ws:\/\/localhost:9222\/devtools\/page\/19fe8849-aa19-4ff6-9c44-f49ae4038071","type":"background_page","id":"19fe8849-aa19-4ff6-9c44-f49ae4038071","devtoolsFrontendUrl":"\/devtools\/inspector.html?ws=localhost:9222\/devtools\/page\/19fe8849-aa19-4ff6-9c44-f49ae4038071"},{"description":"","title":"Feedback","url":"chrome-extension:\/\/gfdkimpbcpahaombhbimeihdjnejgicl\/_generated_background_page.html","webSocketDebuggerUrl":"ws:\/\/localhost:9222\/devtools\/page\/65924e93-cc3c-4b34-8bee-cb8ad2cf5532","type":"background_page","id":"65924e93-cc3c-4b34-8bee-cb8ad2cf5532","devtoolsFrontendUrl":"\/devtools\/inspector.html?ws=localhost:9222\/devtools\/page\/65924e93-cc3c-4b34-8bee-cb8ad2cf5532"},{"description":"","title":"Google Hangouts","url":"chrome-extension:\/\/nkeimhogjdpnpccoofpliimaahmaaome\/background.html","webSocketDebuggerUrl":"ws:\/\/localhost:9222\/devtools\/page\/02e33509-2c86-4aaf-8c6a-87cb12209928","type":"background_page","id":"02e33509-2c86-4aaf-8c6a-87cb12209928","devtoolsFrontendUrl":"\/devtools\/inspector.html?ws=localhost:9222\/devtools\/page\/02e33509-2c86-4aaf-8c6a-87cb12209928"},{"description":"","title":"Take Chrome everywhere","url":"chrome:\/\/welcome\/?variant=everywhere","webSocketDebuggerUrl":"ws:\/\/localhost:9222\/devtools\/page\/d9de8d2b-9dfc-46b0-84ad-446fedd80b27","type":"page","id":"d9de8d2b-9dfc-46b0-84ad-446fedd80b27","devtoolsFrontendUrl":"\/devtools\/inspector.html?ws=localhost:9222\/devtools\/page\/d9de8d2b-9dfc-46b0-84ad-446fedd80b27"}]
15:35:44.733 - Devtools connected
15:35:44.733 - DevTools websocket connected
15:35:44.733 - Sending: {"params":{},"id":1,"method":"Network.clearBrowserCache"}
15:35:44.735 - {"id":1,"result":{}}
15:35:44.735 - Sending: {"params":{},"id":2,"method":"Network.clearBrowserCookies"}
15:35:44.736 - {"id":2,"result":{}}
15:35:44.736 - Sending: {"params":{"returnByValue":true,"expression":"navigator.userAgent"},"id":3,"method":"Runtime.evaluate"}
15:35:44.739 - {"id":3,"result":{"result":{"type":"string","value":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"}}}
15:35:44.740 - Sending: {"params":{"url":"about:blank"},"id":4,"method":"Page.navigate"}
15:35:44.755 - {"id":4,"result":{"frameId":"24649.1"}}
15:35:44.755 - Waiting for Idle...
^CWill exit after test completes.  Hit Ctrl+C again to exit immediately
[24352:24352:0330/153557.248838:ERROR:chrome_browser_main_extra_parts_x11.cc(62)] X IO error received (X server probably went away)
15:35:57.253 - DevTools websocket disconnected
^C15:35:58.573 - Cleaning up

[hr]
OK, had some better results with:

Xvfb :10 -ac -screen 0 1280x1024x24

Then:

export DISPLAY=:10
python wptagent/wptagent.py --server http://localhost/work/ --location WPT_Chrome  --xvfb -vvvvv --shaper none

Still not getting any results (“First View: Test Error: Error configuring traffic-shaping”), but at least it did run Chrome.

Should wptagent.py launch Xvfb automatically, when --xvfb flag is used? Or does Xvfb need to be run before wptagent.py is started?
[hr]
I think I was confused by “connection refused to localhost:9222”.

It is running as expected, no need to run xvfb manually.

The only thing I had to set was setting “Connection” to “Native connection (No Traffic Shaping)”, otherwise, was getting “First View: Test Error: Error configuring traffic-shaping”.

To sum up - it works! That’s excellent!

16

To fix traffic shaping you should just need to add the user running the testing to sudoers with NOPASSWD so it can configure tc (I wouldn’t recommend running the script itself as sudo).