Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JENKINS-41745] Non-Remoting-based CLI #2795

Merged
merged 64 commits into from
Apr 8, 2017
Merged
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
492dbbe
[JENKINS-41745] Make jenkins-cli.jar connect to the SSH port by default.
jglick Mar 10, 2017
2fe2487
FindBugs, and more clearly stating which transport is in use.
jglick Mar 10, 2017
8302b85
Draft of a new CLI transport that can operate over CLIAction with the…
jglick Mar 11, 2017
7da0d39
Argument processing mistake.
jglick Mar 13, 2017
1e860c8
Apply SUREFIRE-1226 workaround across all modules, including cli.
jglick Mar 13, 2017
624ba59
Allow tests to run which use CLI.<init> in process.
jglick Mar 13, 2017
4df2bf1
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Mar 13, 2017
c2a5d85
Establishing baseline behavior of JENKINS-12543: no workaround when u…
jglick Mar 13, 2017
8d622e0
Clarifying role of API tokens in -remoting.
jglick Mar 13, 2017
12ae48e
Deprecating --username/--password and login/logout in favor of new -a…
jglick Mar 13, 2017
b8ad360
Marking various APIs, and a couple of commands, deprecated when they …
jglick Mar 13, 2017
82cd1bd
Deleteing apparently unused class SequenceOutputStream.
jglick Mar 13, 2017
5f0bb2a
Some more work deprecating channel/checkChannel.
jglick Mar 13, 2017
9ffb5d8
Hoping to fix an unreproducible test hang.
jglick Mar 13, 2017
2cf0ac5
Pick up https://github.com/jenkinsci/sshd-module/pull/10 so we are us…
jglick Mar 15, 2017
9e427ce
Noting potential issue in ConsoleCommand.
jglick Mar 15, 2017
f437ccc
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Mar 15, 2017
9434d0e
UI to enable or disable CLI over Remoting.
jglick Mar 16, 2017
71db95a
Allow http://repo.jenkins-ci.org/public/org/jenkins-ci/modules/sshd/1…
jglick Mar 16, 2017
996c52e
Trying to avoid a premature test timeout.
jglick Mar 17, 2017
59d5013
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Mar 17, 2017
473744c
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Mar 17, 2017
289db88
Using Logger consistently for all messages that are not expected to b…
jglick Mar 17, 2017
f3dae19
Fixed locale handling.
jglick Mar 17, 2017
7174e3e
Allowing install-plugin to load a file from standard input so as to w…
jglick Mar 17, 2017
7baf81d
Allowing `build -p fileParam= prj` to load a file from standard input…
jglick Mar 17, 2017
e69ba0f
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Mar 24, 2017
60632c0
Added -strictHostKey option to CLI in -ssh mode.
jglick Mar 24, 2017
466d74d
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Apr 3, 2017
f168936
Test failure on Windows, perhaps specific to filesystem type of Tempo…
jglick Apr 3, 2017
551808c
Using kill -QUIT to try to diagnose client hangs observed on ci.jenki…
jglick Apr 3, 2017
f6314b7
Testing interrupt behavior.
jglick Apr 3, 2017
9812db7
Improved handling of stream closure.
jglick Apr 3, 2017
71d7562
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Apr 4, 2017
56cee95
Found another case where the Windows CI build was complaining about k…
jglick Apr 4, 2017
c74999f
Found a race condition which could explain random CI hangs of CLIActi…
jglick Apr 4, 2017
4c1910a
Silence an annoying message.
jglick Apr 4, 2017
4069e67
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Apr 4, 2017
03aa2a1
Simpler to catch ReadPendingException in just one place.
jglick Apr 4, 2017
fb311c6
Suppress a meaningless ClosedChannelException thrown by Jetty when ki…
jglick Apr 4, 2017
caf3e8e
Removing comment corresponding to something I can no longer reproduce.
jglick Apr 4, 2017
7f633de
CLITest.interrupt could fail if you had a stale localhost entry in ~/…
jglick Apr 4, 2017
f7006b4
Jetty 9.4.3.v20170317 refuses to set an empty HTTP header, leading to…
jglick Apr 4, 2017
3b92399
Tested Jetty 9.4.3.v20170317 but ReadPendingException’s did not disap…
jglick Apr 4, 2017
94d5f96
Figured out why Security232Test.commonsCollections1 was being skipped…
jglick Apr 4, 2017
dbe24a8
Using CLICommandInvoker in cases where we did not actually need to be…
jglick Apr 4, 2017
5caee58
Noting that ssh-cli-auth is obsolete.
jglick Apr 4, 2017
9fc59bf
Finally figured out how to suppress the ReadPendingException from Jetty.
jglick Apr 4, 2017
7bca0a1
Refined implementation of -logger.
jglick Apr 5, 2017
3939820
Using logger as suggested by @rsandell.
jglick Apr 5, 2017
da58159
-http mode broke when the user omitted the mandatory final `/` in the…
jglick Apr 5, 2017
724ebbc
Improved error message displayed when using -http against a pre-JENKI…
jglick Apr 5, 2017
49f0360
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Apr 6, 2017
c41c97b
@stephenc requested documentation of the deprecated FullDuplexHttpStr…
jglick Apr 6, 2017
6c63890
Demonstrating that interleaved stdio does work in -http mode.
jglick Apr 6, 2017
9150d5b
Suppress an idle timeout from Jetty which would otherwise interfere w…
jglick Apr 6, 2017
6ffc57a
Verifying that PlainCLIProtocol ignores unrecognized opcodes, so long…
jglick Apr 6, 2017
2e7fe04
Merge branch 'master' into SSH-CLI-JENKINS-41745
jglick Apr 7, 2017
f3da0e4
Minor review comments from @oleg-nenashev.
jglick Apr 7, 2017
0f87e10
@oleg-nenashev requested this be decoupled from https://github.com/je…
jglick Apr 7, 2017
98f227c
Showing general help message rather than suggesting obsolete authenti…
jglick Apr 7, 2017
45092f0
checkChannel() provides a better error message in case you are not us…
jglick Apr 7, 2017
27d508d
sshd 1.11
jglick Apr 7, 2017
7ae404c
Revert "Noting that ssh-cli-auth is obsolete."
jglick Apr 7, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions cli/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@
<artifactId>commons-codec</artifactId>
<version>1.4</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>remoting</artifactId>
Expand All @@ -50,6 +54,16 @@
<version>1.24</version>
</dependency>
<dependency>
<groupId>org.apache.sshd</groupId>
<artifactId>sshd-core</artifactId>
<optional>true</optional> <!-- do not expose to core -->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's already in the core due to the sshd-module

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it is in war.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, nvm

</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<optional>true</optional> <!-- ditto -->
</dependency>
<dependency> <!-- TODO remove and replace PrivateKeyProvider with SecurityUtils.createFileKeyPairProvider() as in SshClient -->
<groupId>org.jenkins-ci</groupId>
<artifactId>trilead-ssh2</artifactId>
<version>build214-jenkins-1</version>
Expand Down
309 changes: 282 additions & 27 deletions cli/src/main/java/hudson/cli/CLI.java

Large diffs are not rendered by default.

12 changes: 11 additions & 1 deletion cli/src/main/java/hudson/cli/CLIConnectionFactory.java
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ public CLIConnectionFactory url(String jenkins) throws MalformedURLException {

/**
* This {@link ExecutorService} is used to execute closures received from the server.
* Used only in Remoting mode.
*/
public CLIConnectionFactory executorService(ExecutorService es) {
this.exec = es;
Expand Down Expand Up @@ -59,15 +60,24 @@ public CLIConnectionFactory authorization(String value) {

/**
* Convenience method to call {@link #authorization} with the HTTP basic authentication.
* Currently unused.
*/
public CLIConnectionFactory basicAuth(String username, String password) {
return basicAuth(username+':'+password);
}

/**
* Convenience method to call {@link #authorization} with the HTTP basic authentication.
* Cf. {@code BasicHeaderApiTokenAuthenticator}.
*/
public CLIConnectionFactory basicAuth(String userInfo) {
return authorization("Basic " + new String(Base64.encodeBase64((userInfo).getBytes())));
}


/**
* @deprecated Specific to Remoting-based protocol.
*/
@Deprecated
public CLI connect() throws IOException, InterruptedException {
return new CLI(this);
}
Expand Down
2 changes: 2 additions & 0 deletions cli/src/main/java/hudson/cli/CliEntryPoint.java
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@
* Remotable interface for CLI entry point on the server side.
*
* @author Kohsuke Kawaguchi
* @deprecated Specific to Remoting-based protocol.
*/
@Deprecated
public interface CliEntryPoint {
/**
* Just like the static main method.
Expand Down
3 changes: 3 additions & 0 deletions cli/src/main/java/hudson/cli/Connection.java
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@
import java.security.interfaces.RSAPublicKey;
import java.security.spec.X509EncodedKeySpec;

/**
* Used by Jenkins core only in deprecated Remoting-based CLI.
*/
public class Connection {
public final InputStream in;
public final OutputStream out;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
package hudson.cli;

import java.io.PrintWriter;
import java.io.StreamCorruptedException;
import java.io.StringWriter;

// TODO COPIED FROM hudson.remoting

/**
* Signals a {@link StreamCorruptedException} with some additional diagnostic information.
*
* @author Kohsuke Kawaguchi
*/
class DiagnosedStreamCorruptionException extends StreamCorruptedException {
private final Exception diagnoseFailure;
private final byte[] readBack;
private final byte[] readAhead;

DiagnosedStreamCorruptionException(Exception cause, Exception diagnoseFailure, byte[] readBack, byte[] readAhead) {
initCause(cause);
this.diagnoseFailure = diagnoseFailure;
this.readBack = readBack;
this.readAhead = readAhead;
}

public Exception getDiagnoseFailure() {
return diagnoseFailure;
}

public byte[] getReadBack() {
return readBack;
}

public byte[] getReadAhead() {
return readAhead;
}

@Override
public String toString() {
StringBuilder buf = new StringBuilder();
buf.append(super.toString()).append("\n");
buf.append("Read back: ").append(HexDump.toHex(readBack)).append('\n');
buf.append("Read ahead: ").append(HexDump.toHex(readAhead));
if (diagnoseFailure!=null) {
StringWriter w = new StringWriter();
PrintWriter p = new PrintWriter(w);
diagnoseFailure.printStackTrace(p);
p.flush();

buf.append("\nDiagnosis problem:\n ");
buf.append(w.toString().trim().replace("\n","\n "));
}
return buf.toString();
}
}
191 changes: 191 additions & 0 deletions cli/src/main/java/hudson/cli/FlightRecorderInputStream.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
package hudson.cli;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Arrays;

// TODO COPIED FROM hudson.remoting

/**
* Filter input stream that records the content as it's read, so that it can be reported
* in case of a catastrophic stream corruption problem.
*
* @author Kohsuke Kawaguchi
*/
class FlightRecorderInputStream extends InputStream {

/**
* Size (in bytes) of the flight recorder ring buffer used for debugging remoting issues.
* @since 2.41
*/
static final int BUFFER_SIZE = Integer.getInteger("hudson.remoting.FlightRecorderInputStream.BUFFER_SIZE", 1024 * 1024);

private final InputStream source;
private ByteArrayRingBuffer recorder = new ByteArrayRingBuffer(BUFFER_SIZE);

FlightRecorderInputStream(InputStream source) {
this.source = source;
}

/**
* Rewinds the record buffer and forget everything that was recorded.
*/
public void clear() {
recorder = new ByteArrayRingBuffer(BUFFER_SIZE);
}

/**
* Gets the recorded content.
*/
public byte[] getRecord() {
return recorder.toByteArray();
}

/**
* Creates a {@link DiagnosedStreamCorruptionException} based on the recorded content plus read ahead.
* The caller is responsible for throwing the exception.
*/
public DiagnosedStreamCorruptionException analyzeCrash(Exception problem, String diagnosisName) {
final ByteArrayOutputStream readAhead = new ByteArrayOutputStream();
final IOException[] error = new IOException[1];

Thread diagnosisThread = new Thread(diagnosisName+" stream corruption diagnosis thread") {
public void run() {
int b;
try {
// not all InputStream will look for the thread interrupt flag, so check that explicitly to be defensive
while (!Thread.interrupted() && (b=source.read())!=-1) {
readAhead.write(b);
}
} catch (IOException e) {
error[0] = e;
}
}
};

// wait up to 1 sec to grab as much data as possible
diagnosisThread.start();
try {
diagnosisThread.join(1000);
} catch (InterruptedException ignored) {
// we are only waiting for a fixed amount of time, so we'll pretend like we were in a busy loop
Thread.currentThread().interrupt();
// fall through
}

IOException diagnosisProblem = error[0]; // capture the error, if any, before we kill the thread
if (diagnosisThread.isAlive())
diagnosisThread.interrupt(); // if it's not dead, kill

return new DiagnosedStreamCorruptionException(problem,diagnosisProblem,getRecord(),readAhead.toByteArray());

}

@Override
public int read() throws IOException {
int i = source.read();
if (i>=0)
recorder.write(i);
return i;
}

@Override
public int read(byte[] b, int off, int len) throws IOException {
len = source.read(b, off, len);
if (len>0)
recorder.write(b,off,len);
return len;
}

/**
* To record the bytes we've skipped, convert the call to read.
*/
@Override
public long skip(long n) throws IOException {
byte[] buf = new byte[(int)Math.min(n,64*1024)];
return read(buf,0,buf.length);
}

@Override
public int available() throws IOException {
return source.available();
}

@Override
public void close() throws IOException {
source.close();
}

@Override
public boolean markSupported() {
return false;
}

// http://stackoverflow.com/a/3651696/12916
private static class ByteArrayRingBuffer extends OutputStream {

byte[] data;

int capacity, pos = 0;

boolean filled = false;

public ByteArrayRingBuffer(int capacity) {
data = new byte[capacity];
this.capacity = capacity;
}

@Override
public synchronized void write(int b) {
if (pos == capacity) {
filled = true;
pos = 0;
}
data[pos++] = (byte) b;
}

public synchronized byte[] toByteArray() {
if (!filled) {
return Arrays.copyOf(data, pos);
}
byte[] ret = new byte[capacity];
System.arraycopy(data, pos, ret, 0, capacity - pos);
System.arraycopy(data, 0, ret, capacity - pos, pos);
return ret;
}

/** @author @roadrunner2 */
@Override public synchronized void write(byte[] buf, int off, int len) {
// no point in trying to copy more than capacity; this also simplifies logic below
if (len > capacity) {
off += (len - capacity);
len = capacity;
}

// copy to buffer, but no farther than the end
int num = Math.min(len, capacity - pos);
if (num > 0) {
System.arraycopy(buf, off, data, pos, num);
off += num;
len -= num;
pos += num;
}

// wrap around if necessary
if (pos == capacity) {
filled = true;
pos = 0;
}

// copy anything still left
if (len > 0) {
System.arraycopy(buf, off, data, pos, len);
pos += len;
}
}

}

}
47 changes: 47 additions & 0 deletions cli/src/main/java/hudson/cli/HexDump.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
package hudson.cli;

// TODO COPIED FROM hudson.remoting

/**
* @author Kohsuke Kawaguchi
*/
class HexDump {
private static final String CODE = "0123456789abcdef";

public static String toHex(byte[] buf) {
return toHex(buf,0,buf.length);
}
public static String toHex(byte[] buf, int start, int len) {
StringBuilder r = new StringBuilder(len*2);
boolean inText = false;
for (int i=0; i<len; i++) {
byte b = buf[start+i];
if (b >= 0x20 && b <= 0x7e) {
if (!inText) {
inText = true;
r.append('\'');
}
r.append((char) b);
} else {
if (inText) {
r.append("' ");
inText = false;
}
r.append("0x");
r.append(CODE.charAt((b>>4)&15));
r.append(CODE.charAt(b&15));
if (i < len - 1) {
if (b == 10) {
r.append('\n');
} else {
r.append(' ');
}
}
}
}
if (inText) {
r.append('\'');
}
return r.toString();
}
}
Loading