Refactor HPCSocket for Modern Job Management and Better Reliability

[stevenweaver]

1. Socket Connection Management

• Replace callback-based connection handling with Promise/async-await pattern
• Improve connection error handling and reconnection logic
• Add timeout handling for stale connections

2. Job Status Management

• Implement atomic status updates to prevent race conditions
• Add robust event handling for job lifecycle events
• Improve job progress tracking and reporting

3. Error Handling

• Implement comprehensive error classification
• Add proper error recovery mechanisms
• Improve error reporting and logging
• Add retry logic for recoverable errors

4. Code Organization

• Define clear interfaces between components, independent of hyphy so that other tools can be easily implemented.
• Improve code reusability

5. Configuration

• Add validation for configuration options
• Support dynamic cluster configuration

6. Testing & Monitoring

• Add unit test coverage
• Add integration test framework
• Improve logging and monitoring capabilities
• Add health check mechanisms

Technical Requirements:

• Use ES6+ features where appropriate
• Maintain backwards compatibility with existing job types
• Document new interfaces and methods

Success Criteria:

• Improved reliability of job submission and monitoring
• Reduced frequency of stale connections
• Better error recovery
• Cleaner, more maintainable codebase
• Comprehensive test coverage