Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JENKINS-62939: Use latest for getting console output if Nitro #707

Merged
merged 2 commits into from
Feb 27, 2022
Merged

JENKINS-62939: Use latest for getting console output if Nitro #707

merged 2 commits into from
Feb 27, 2022

Conversation

patbos
Copy link
Contributor

@patbos patbos commented Feb 14, 2022

Added check for if hypervisor is Nitro and if so add latest to getConsoleOutput to decrease the time for when
console output can be parsed and ssh host key can be validated.

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests - that demonstrates feature works or fixes the issue

@patbos patbos marked this pull request as ready for review February 14, 2022 19:33
@res0nance res0nance added the enhancement Feature additions or enhancements label Feb 17, 2022
@res0nance res0nance merged commit aa37bd0 into jenkinsci:master Feb 27, 2022
@patbos patbos deleted the nitro_latest_test branch February 27, 2022 14:50
@krdlab
Copy link

krdlab commented Mar 1, 2022

Hello.
I am using version 1.67 which contains this PR.

When GetConsoleOutputRequest.setLatest is set to true, the result of getConsoleOutput may be returned even if the host key is not yet visible.

As the result, if "Host Key Verification Strategy" is "check-new-hard", the instance will be launched and terminated repeatedly.

I guess it is necessary to modify to repeat getConsoleOutput several times at intervals.

@res0nance
Copy link
Contributor

Hello. I am using version 1.67 which contains this PR.

When GetConsoleOutputRequest.setLatest is set to true, the result of getConsoleOutput may be returned even if the host key is not yet visible.

As the result, if "Host Key Verification Strategy" is "check-new-hard", the instance will be launched and terminated repeatedly.

I guess it is necessary to modify to repeat getConsoleOutput several times at intervals.

Do you have the logs of what is happening?

@krdlab
Copy link

krdlab commented Mar 2, 2022

Thanks for the reply.

The following log was repeated. (This job was working fine before the EC2 plugin was upgraded)

Log
2022-02-28 03:12:22.861+0000 [id=34]	INFO	c.a.j.e.NoDelayProvisionStrategy#apply: label [slave]: currentDemand 1 availableCapacity 0 (availableExecutors 0 connectingExecutors 0 plannedCapacitySnapshot 0 additionalPlannedCapacity 0)
2022-02-28 03:12:22.861+0000 [id=34]	INFO	hudson.plugins.ec2.EC2Cloud#provision: SlaveTemplate{description='', labels='jenkins-slave-node slave'}. Attempting to provision agent needed by excess workload of 1 units
2022-02-28 03:12:23.273+0000 [id=34]	INFO	hudson.plugins.ec2.SlaveTemplate#getImage: Getting image for request {ExecutableUsers: [],Filters: [],ImageIds: [ami-aaaaaaaaaaaaaaaaa],Owners: [],}
2022-02-28 03:12:23.340+0000 [id=34]	INFO	hudson.plugins.ec2.SlaveTemplate#provisionSpot: Launching ami-aaaaaaaaaaaaaaaaa for template 
2022-02-28 03:12:23.737+0000 [id=34]	INFO	hudson.plugins.ec2.SlaveTemplate#setupRootDevice: AMI had /dev/sda1
2022-02-28 03:12:23.737+0000 [id=34]	INFO	hudson.plugins.ec2.SlaveTemplate#setupRootDevice: {DeleteOnTermination: true,SnapshotId: snap-sssssssssssssssss,VolumeSize: 15,VolumeType: gp2,Encrypted: true}
2022-02-28 03:12:23.737+0000 [id=34]	INFO	hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{description='', labels='jenkins-slave-node slave'}. EBS default encryption value set to: Based on AMI (null)
2022-02-28 03:12:24.926+0000 [id=34]	INFO	hudson.plugins.ec2.SlaveTemplate#provisionSpot: Spot instance id in provision: sir-ssssssss
2022-02-28 03:12:24.927+0000 [id=34]	INFO	hudson.plugins.ec2.EC2Cloud#provision: SlaveTemplate{description='', labels='jenkins-slave-node slave'}. Attempting provision finished, excess workload: 0
2022-02-28 03:12:24.927+0000 [id=34]	INFO	hudson.plugins.ec2.EC2Cloud#provision: We have now 1 computers, waiting for 1 more
2022-02-28 03:12:24.927+0000 [id=34]	INFO	h.s.NodeProvisioner$StandardStrategyImpl#apply: Started provisioning EC2 (jenkins-slave-on-ec2) -  from ec2-jenkins-slave-on-ec2 with 1 executors. Remaining excess workload: 0
2022-02-28 03:12:35.894+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud$2#call: SlaveTemplate{description='', labels='jenkins-slave-node slave'} Node EC2 (jenkins-slave-on-ec2) -  (sir-ssssssss) moved to RUNNING state in 8 seconds and is ready to be connected by Jenkins
2022-02-28 03:12:42.861+0000 [id=40]	INFO	h.p.ec2.EC2RetentionStrategy#start: Start requested for EC2 (jenkins-slave-on-ec2) -  (sir-ssssssss)
2022-02-28 03:12:42.862+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Launching instance: i-iiiiiiiiiiiiiiiii
2022-02-28 03:12:42.862+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: bootstrap()
2022-02-28 03:12:42.862+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Getting keypair...
2022-02-28 03:12:42.862+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Using private key jenkins-slave (SHA-1 fingerprint ***********************************************************)
2022-02-28 03:12:42.862+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Authenticating as ubuntu
2022-02-28 03:12:42.867+0000 [id=40]	INFO	hudson.slaves.NodeProvisioner#update: EC2 (jenkins-slave-on-ec2) -  provisioning successfully completed. We have now 2 computer(s)
2022-02-28 03:12:42.977+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Connecting to 192.168.41.25 on port 22, with timeout 10000.
2022-02-28 03:12:52.979+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
2022-02-28 03:12:52.979+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
2022-02-28 03:12:58.030+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Connecting to 192.168.41.25 on port 22, with timeout 10000.
2022-02-28 03:13:08.031+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
2022-02-28 03:13:08.031+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
2022-02-28 03:13:13.120+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Connecting to 192.168.41.25 on port 22, with timeout 10000.
2022-02-28 03:13:13.123+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 192.168.41.25:22
2022-02-28 03:13:13.123+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
2022-02-28 03:13:18.179+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Connecting to 192.168.41.25 on port 22, with timeout 10000.
2022-02-28 03:13:18.477+0000 [id=245]	INFO	hudson.plugins.ec2.EC2Cloud#log: The instance EC2 (jenkins-slave-on-ec2) -  (sir-ssssssss) has a blank console. Maybe the console is yet not available. If enough time has passed, consider changing the key verification strategy or the AMI used by one printing out the host key in the instance console
2022-02-28 03:13:18.486+0000 [id=245]	INFO	hudson.plugins.ec2.EC2Cloud#log: The instance console is blank. Cannot check the key. The connection to EC2 (jenkins-slave-on-ec2) -  (sir-ssssssss) is not allowed
2022-02-28 03:13:18.488+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 192.168.41.25:22
2022-02-28 03:13:18.488+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
2022-02-28 03:13:23.544+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Connecting to 192.168.41.25 on port 22, with timeout 10000.
2022-02-28 03:13:23.826+0000 [id=246]	INFO	hudson.plugins.ec2.EC2Cloud#log: The instance EC2 (jenkins-slave-on-ec2) -  (sir-ssssssss) didn't print the host key. Expected a line starting with: "ssh-ed25519"
2022-02-28 03:13:23.827+0000 [id=246]	INFO	hudson.plugins.ec2.EC2Cloud#log: The SSH key (ssh-ed25519 ***********************************************) presented by the instance has not been found on the instance console. Cannot check the key. The connection to EC2 (jenkins-slave-on-ec2) -  (sir-ssssssss) is not allowed
2022-02-28 03:13:23.832+0000 [id=240]	INFO	hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 192.168.41.25:22
2022-02-28 03:13:23.976+0000 [id=248]	INFO	hudson.plugins.ec2.EC2SpotSlave#lambda$terminate$0: Cancelled Spot request: sir-ssssssss
2022-02-28 03:13:24.217+0000 [id=248]	INFO	hudson.plugins.ec2.EC2SpotSlave#lambda$terminate$0: Terminated EC2 instance (terminated): i-iiiiiiiiiiiiiiiii

When I tested the result of getConsoleOutput with the following code and c5.large instance, I found that the first non-null string returned by the method did not contain the host key, and then the next result returned after waiting a few seconds contained the host key.

Test code
import com.amazonaws.services.ec2.AmazonEC2;
import com.amazonaws.services.ec2.AmazonEC2ClientBuilder;
import com.amazonaws.services.ec2.model.GetConsoleOutputRequest;
import com.amazonaws.services.ec2.model.GetConsoleOutputResult;

public class App {
    public static void main(String[] args) {
        final String instanceId = args[0];

        final String output = new App().getConsoleOutput(instanceId);

        System.out.println(output);
        System.out.println();
    }

    private String getConsoleOutput(final String instanceId) {
        final AmazonEC2 ec2 = AmazonEC2ClientBuilder.defaultClient();

        final GetConsoleOutputRequest request = new GetConsoleOutputRequest(instanceId);
        request.setLatest(true);

        final GetConsoleOutputResult result = ec2.getConsoleOutput(request);
        return result.getDecodedOutput();
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Feature additions or enhancements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants