Environment issue with newer versions of SGE

[hlasimpk]

System information

• PyJob version: 0.3.1
• Python version: ccp4-python
• Environment (if applicable):

Problem description

The PATH variable is no longer passed through the -V option in SGE. To access additional PATH elements the following needs to be set:

export PATH=$PATH:$SGE_O_PATH 

This needs to be added in the environment of the job. This is backwards compatible with older versions.

Need to work out how best to implement this - I encountered it running SIMBAD on the new cluster and can probably add something to the scripts to deal with it, but I think it might be something useful to add to pyjob.

[fsimkovic]

@hlasimpk thanks for reporting this. I currently do not have access to SGE, so cannot implement/test this myself easily. What are the signs of it "not working" anymore? Do you see an error message or is the environment simply not forwarded? Has there been a deprecation announcement or other documentation that suggests the export solution you propose?

[hlasimpk]

No error message - just the environment not being forwarded. It manifests itself in SIMBAD with ccp4-python not found in the log files.

I've not fully looked into the problem but Jens/Ronan proposed the solution mentioned above after encountering and fixing it in AMPLE.

[fsimkovic]

I saw the change Jens/Ronan made, something I don't think is the long-term/correct solution. The SGE_O_PATH variable has been available for over a decade. If it was the way to resolve the problem, this would have been the standard way.

Have you tried explicitly providing the environment variables you require?

[hlasimpk]

I haven't tried that. I'll have a go now, but in the short term, is this something available in pyjob already?

[fsimkovic]

It is currently not possible to forward -v to the job run script though this should be easy to test by modifying the SunGridEngineTask._create_runscript function.

[hlasimpk]

Okay super. I'll give you an update on this later today - can't currently access the ccp4 gateway machine so can't test it.

[hlasimpk]

The gateway machine is back up. I had a play with the -v option and it didn't seem to be doing what we wanted. I've realised that we can side step the issue however if we just source CCP4 in the input scripts. This seems to be the easiest solution.