Can I PR to add user_name to the job info?

[robgics]

I can access user_id of course, but for our data purposes we need user_name, and I'd rather not do an extra lookup on the system. I did a test where I added it to the pyslurm.pyslurm.job class and that seems to work.
Can I PR this, or is there another way I don't know of to access the username for a given userid?

[giovtorres]

I'm curious how you implemented this. I had tried this years ago and on large clusters, the local user lookups added a fair amount of latency on nodes with a large number of users. Then i took it out and left just the user ids, which had expected performance.

I think slurm caches this lookup internally somewhere.

[robgics]

If it's been tried before then I think I can safely say I didn't do it right.
The comments for slurm_job_info in slurm.h said that user_name wasn't always set, but would be correct if it was. So I just relied on that and put this in pyslurm.pyx:

Job_dict['user_name'] = slurm.stringOrNone(self._record.user_name, '')

At least in our case (RHEL8, slurm 23.2, python3) it is being set consistently.

[giovtorres]

This was a few years ago and could be better now. If slurm exposes it, then maybe it's okay. I would take your test and try to run it on a test machine with like 2000 fake users, and then try to get all jobs (which should include the user names) and see what the performance is.

[robgics]

I think that's something I can probably get my team lead to let me do. But can you give me some idea of how best to measure that performance? What measure/numbers would be meaningful?
Thanks.

[giovtorres]

you can test inside a vm, or you can use a slurm container to test. go ahead and do a PR. i'll try to find my old implementation

[robgics]

No, I mean...what am I measuring? I have a vagrant setup I'm using for slurm development that I can probably repurpose with as many nodes as I need....but what am I measuring to see if there is a performance impact? CPU load on the slurm controller node? slurm scheduler runtime or queue depth? I'm just want to know what I should be measuring so that I can reasonably assert that the performance impact is negligible (or not).

[tazend]

Hi @robgics

just wanted to note that the pyslurm.job class has been marked deprecated.
There are new classes available as a replacement: pyslurm.Job (which already has the user_name attribute, and a collection pyslurm.Jobs, which I'd recommend to use for the future.

When you get all Jobs with pyslurm.Job.load(), there is an optional parameter preload_passwd_info that will fetch passwd/group info of all users upfront, and cache that in each pyslurm.Job instance, so when you finally access the user_name attribute, it will just take the user name from that cache. By default preload_passwd_info is set to False, and the default behaviour is to attempt converting the UID to a string each time the user_name attribute is accessed.

But now that you mention it, it might make sense to add whether user_name in job_info_t is set first, and only to the conversion as a fallback.
Adding to that though: The slurm devs also seem to always explicitly convert the UID/GID to a string for every job in scontrol - see here (following that path there doesn't seem to be any cache, as they end up calling getpwuid_r). They also don't check the user_name provided in job_info_t - but scontrol always felt pretty fast for me on my cluster with many Jobs.

However if you'd still want to continue using pyslurm.job for now, then sure, as @giovtorres said feel free to open the PR. My suggestion would be:

1. Parse user_name from job_info_t (as you already did in the code snippet)
2. If user_name is empty, attempt to lookup the username by UID (there is a uid_to_name helper function which does that for example)

and see how that works out.

[robgics]

I did NOT know that...thank you for telling me, I'll investigate that. We're just getting into this, so it would make no sense to implement a deprecated interface from the get-go.

And ya, the reason I began this search is because I'm replacing some older code that called scontrol and then parsed the output, and I noted that scontrol job info always seemed to include the username. So I figured it had to be accessible through slurm somehow.

Thanks for the suggestions, I'll look into them.

[robgics]

Ok...there doesn't seem to be any point in submitting a PR to change a deprecated job class. And I have confirmed that pyslurm.Jobs jobs do indeed contain user_name. So that is resolved for me.

Thanks for the help.