Error when handling "unknown" jobs on LSF system

[nyanss]

Dear maintainers, here is an error when running DPGEN:

Our machine uses LSF system to manage jobs. Sometimes the job state would become "UNKNOWN"* due to network's problems between computer nodes, and it would automatically return to normal ("RUN" or "PEND"). According to the error information, it seems that DPGEN ends running when meeting unknown jobs.
If so, I would like to know whether there is any method to skip or ignore such situation, or could you please add some functions to process it? If not, I hope you can tell me why and how to deal with it. Thank you!

DPGEN version: 0.10.7.dev0+g8dea29e.d20220801

*or maybe "UNKNWN" when querying job state. I hardly see it, and the LSF documentation does not mention it.

[njzjz]

dpdispatcher/dpdispatcher/lsf.py

Lines 135 to 146 in a3773f9

	if status_word in ["PEND", "WAIT", "PSUSP"]:
	return JobStatus.waiting
	elif status_word in ["RUN", "USUSP"]:
	return JobStatus.running
	elif status_word in ["DONE", "EXIT"]:
	if self.check_finish_tag(job):
	dlog.info(f"job: {job.job_hash} {job.job_id} finished")
	return JobStatus.finished
	else:
	return JobStatus.terminated
	else:
	return JobStatus.unknown

This is the current list of keywords to be processed. Contribution is welcome if you found new words.

[nyanss]

I found a similar issue at other project.

Snakemake-Profiles/lsf#28

As it said, "UNKNWN" job is caused by kinds of problems on different cluster systems. It's difficult to generalize all situations. I will firstly fix it on our own machine, and then consider how to deal with it more gracefully. You can alse take that issue as reference to improve it. Thanks again.