Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent spiders among different servers #45

Closed
OlgaCh opened this issue Apr 30, 2014 · 3 comments
Closed

Inconsistent spiders among different servers #45

OlgaCh opened this issue Apr 30, 2014 · 3 comments

Comments

@OlgaCh
Copy link

OlgaCh commented Apr 30, 2014

We have 7 servers with running scrapyd.
Repo clones to the first server and the deployed to another 6 servers.


Already on 'master'
From github.com:....
e51238d...82cb8 master -> origin/master
Updating e51238d..82cb8

Packing version 27-01-2014-3012-ged82cb8
Deploying to project "1" in http://server1:6800/addversion.json
Server response (200):
{"status": "ok", "project": "1", "version": "27-01-2014-3012-ged82cb8", "spiders": 1893}

Packing version 27-01-2014-3012-ged82cb8
Deploying to project "1" in http://server2:6800/addversion.json
Server response (200):
{"status": "ok", "project": "1", "version": 27-01-2014-3012-ged82cb8"", "spiders": 1893}

Packing version 27-01-2014-3012-ged82cb8
Deploying to project "1" in http://server3:6800/addversion.json
Server response (200):
{"status": "ok", "project": "1", "version": "27-01-2014-3012-ged82cb8", "spiders": 1893}


But when I start the spiders with scrapyd at different servers I receive the different results. Some old code still remains at scrapyd.
Despite the versions at spiders are the same:
user@server1:~/de-scrapy$ curl http://localhost:6800/listversions.json?project
...

{"status": "ok", "versions": ["27-01-2014-3012-ged82cb8-master",...

user@server6:~$ curl http://localhost:6800/listversions.json?project=1
{"status": "ok", "versions": ["27-01-2014-3012-ged82cb8-master",...

I tried to restart scrapyd service but this doesn't help.
Only one working solution was to delete project at each server and create it from scratch. But this doesn't guarantee that problem can't occur again.

Please help with fixing this.
Thanks!

@pablohoffman
Copy link
Member

do you think this could be related to #34?

@OlgaCh
Copy link
Author

OlgaCh commented Apr 30, 2014

I'm not sure because I checked the versions at servers and they were the same as on server1 (where spiders works well).
Is scrapyd can choose different versions for different spiders?

@pablohoffman
Copy link
Member

no, scrapyd always run the latest version, but that may not work sometimes because of #34

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants