OSError: [Errno 12] Cannot allocate memory

WebPageTest Private Instances
1

Our test agents have started to fail after successfully running for a few hours. The error is: OSError: [Errno 12] Cannot allocate memory

We use docker images on c4.large instances (4GB RAM) but have also seen this failure using on-prem instances with 8GB of ram. Rolling back to an image from early December did not fix the issue, so I suspect this could be due to a browser change.

To get agents working again, we can restart the container. Adding swap also seems to fix the issue, but we have not run tests long enough to see if this is a permanent fix.

Anyone else see this issue and have a permanent fix?

We run via docker with these options:
docker run -d --shm-size=1g -e “SERVER_URL=http:///work/” -e “LOCATION=${region}_Linux” --cap-add=“NET_ADMIN” --restart unless-stopped --name /webpagetest/agent:latest

2

Any news here? Our 2 agents exhaust 8GB of memory within less than 8 hours.

3

We are working around the issue by doing an hourly container restart. If you do a “ps” on the host, you will see a lot of defunct chrome processes.

From our user-data script:
“# Restart once/hour for zombie procs\n”,
“sudo echo '#!/bin/bash\n”,
“sleep $(( ${RANDOM:0:1}*60 ))\n”,
“docker restart wpt’ > /etc/cron.hourly/wpt-restart\n”,
“sudo chmod 755 /etc/cron.hourly/wpt-restart\n”

4

Thank you, jedmichnowicz. So do we restart them with some frequency. Weird, why just two of us are here. I believe it’s a common issue.
The only reason for no answers, I guess, is that wpt docker is not maintained here.

5

You’re not alone! I’m also experiencing this problem.

6

Same problem here. I just upgraded a few of my agents to the latest docker image (published on mar-20, 2020) and started seeing this error. Agents are running version 20.01 … anybody found a solution for this yet?

7

We do as @jedmichnowicz suggested, restart them periodically. In our case (8G of memory for two agent containers) every 8 hours is good enough.

8

yea, that’s a good idea and i was already restarting twice a day. I just rolled back my agents to v18.10 and those are working fine. When i build a fresh new container with a v20.01 agent, i get these memory errors immediately, on the first test i run. So it’s not happening over time. There is definitely a difference between the 18.10 and 20.01 images (I haven’t tried the 19.04 image yet) … i don’t have the exact log lines anymore but i recall seeing the memory error occur in python’s subprocess module … i’ll post here if i see it again

9

Hello,

I have also this memory problem. I first thought it was because of the defunct process too.
Passing the --init flag to docker run fixes the defunct chrome processes problem but not the memory problem.

The defunct chrome processes are not well managed by the container. What you can do to fix that correctly is to init the container with GitHub - krallin/tini: A tiny but valid `init` for containers as the init (PID 1) of the container instead of the /wptagent/entrypoint.sh command. tini is designed to manage defunct processes
But it doesn’t fix the meory problem :confused:

10

we are also seeing "can not allocate memory issue "
wpt agentt 20.01
max memory : 4gb, cpu =2
total consumued mem is 95%
consumed memory : python 65 %, aws 6.7 , firefox 5.9 ,