502 Bad Gateway error tied to param visibility

[rayrrr]

We use Luigi as an unpinned requirement that rebuilds whenever we deploy, in a cloud-hosted environment.

We deployed some changes today that did not involve our Python code; after this deploy, our service went offline with 502 Bad Gateway responses from the scheduler.

Our team dug in and found the following error:

AttributeError: 'Task' object has no attribute 'param_visibilities'
 File "/usr/local/lib/python3.6/site-packages/luigi/scheduler.py", line 825, in add_task

It looks like the Scheduler class needs to be updated to work with the latest changes to how the Task class handles param visibility included in release 2.7.7.

[dlstadther]

@Gr1f0n6x If this is related to #2278, could you advise?

[nryanov]

@rayrrr hm, it's strange =\ Probably, i missed something
I will look at this.
If it is possible, could you provide some code to reproduce it?

[rayrrr]

@Gr1f0n6x can't provide our company code, sorry, but it looks as though it'll happen with any Task run against a scheduler. I can tell you we are using a central scheduler, and we are able to run 2.7.6 without issue, but anything later than that results in the bug occurring.

[nryanov]

@rayrrr ok, then just one question: do you save tasks state anywhere after you disable luigi's central scheduler in pickled file? Probably, it's the source of this problem as old tasks which were saved before update has no this attribute, but after update scheduler try to get it

[nryanov]

Yes, as i thought. The problem was in saving old versioned tasks.

    result = getattr(self._scheduler, method)(**arguments)
  File "/Users/grifon/anaconda/lib/python3.6/site-packages/luigi/scheduler.py", line 825, in add_task
    if not task.param_visibilities:
AttributeError: 'Task' object has no attribute 'param_visibilities'

[rayrrr]

@Gr1f0n6x we do save state in a pickle file as per the built-in state_path attribute of the scheduler but that's it. We currently run our jobs as "nightly batch jobs," so there were no old tasks waiting to be executed when this happened to us AFAIK.

The bug has something to do with https://github.com/spotify/luigi/blob/master/luigi/scheduler.py#L825 which is a conditional statement dealing with an attribute called task.param_visibilities, which doesn't exist in the case of the bug. It appears that this portion of the code is intending to set this attribute if it doesn't already exist...but it also seems this attribute is added on to the instance of the class at some point after it is created...so if something didn't happen as expected in that whole flow, this bug could result, because

>>> class Task(object):
...     builtin = 'hello'
... 
>>> task = Task()
>>> task.builtin
'hello'
>>> task.param_visibilities
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Task' object has no attribute 'param_visibilities'

so if you want to check for the existence of an attribute on a class instance:

>>> hasattr(task, 'param_visibilities')
False

Might it be better to use if hasattr(task, 'param_visibilities') on line 825 in that conditional statement instead of if task.param_visibilities?

[nryanov]

@rayrrr yes, exactly. I tried to replace this line if not task.param_visibilities by
if not getattr(task, 'param_visibilities', None) and, at least, it doesn't fail with this error anymore

[nryanov]

@rayrrr i've created PR.
I believe that it will fix this issue.