Get Even More Visitors To Your Blog, Upgrade To A Business Listing >>

SOLVED: Splash (+scrapy) inside docker - timeout

Maxime De Bruyn:

I have the following issue with splash running inside a docker container and controlled by scrapy (scrapyrt).


2018-02-07 22:10:26.619754 [events] {"maxrss": 336348, "client_ip": "172.24.0.6", "method": "POST", "timestamp": 1518041426, "load": [6.32, 4.38, 2.03], "status_code": 504, "user-agent": "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36", "args": {"headers": {"Cookie": "PHPSESSID=heojt74big4t6iqvqd2a0eg5l1; BOAsoft606lang[lang]=nl; BOAsoft606lang[langnum]=0", "User-Agent": "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36", "Referer": "https://www.baltagroup.com/nl/Contacteer_ons", "Accept-Language": "en", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"}, "timeout": 3600, "uid": 139765009450936, "url": "https://www.baltagroup.com/nl/Contacteer_ons#mm-5", "images": 0}, "rendertime": 5.554704904556274, "active": 0, "fds": 26, "path": "/render.html", "error": {"description": "Timeout exceeded rendering page", "error": 504, "info": {"timeout": 3600.0}, "type": "GlobalTimeoutError"}, "qsize": 0, "_id": 139765009450936}

It seems the error is Timeout exceeded rendering page code 504, but the rendertime is equal to 5.554 and the timeout is set to 3600.0. I don't get this error message.

This error starts appearing after a few minutes. The (multiple) splash containers run inside a docker-compose application. The --timeout parameter is set to 3600 when launching splash.



Posted in S.E.F
via StackOverflow & StackExchange Atomic Web Robots
This Question have been answered
HERE


This post first appeared on Stack Solved, please read the originial post: here

Share the post

SOLVED: Splash (+scrapy) inside docker - timeout

×

Subscribe to Stack Solved

Get updates delivered right to your inbox!

Thank you for your subscription

×