| Wiki Reorganisation | |:-------------------:| | This page has been classified for reorganisation. It has been given the category **MOVE**. | | The content of this page will be revised and moved to one or more other pages in the new wiki structure. | ## General ### This fact shows commands that refers to variables. What do they mean? To simplify the cut & paste of commands we have listed all used variables here along with their default values. Setting up your local environment once is easier than altering every line of code showed at this FAQ. Variable | Purpose | Example/default value ---|---|--- server|remote server name (gateway)|pcmdi3.llnl.gov port|port to the remote SSL connection|443 truststore_path|location of tomcat's truststore|/usr/local/tomcat/conf/esg-truststore.ts store_pass|password for the truststore|changeit The default values for the following commands are (you may copy & paste this, altering them as required): truststore_path=/usr/local/tomcat/conf/esg-truststore.ts store_pass=changeit port=443 server=pcmdi3.llnl.gov Also assure the ` keytool ` app is in the current path (it's part of the JDK). ## Java Truststore/Keystore + Tomcat ### Truststore/keystore? What's the difference? In reality none. But for the ESG security there is. The keystore holds the private key which will be used for signing outgoing SSL trafic (the server certificate). This ensures the client the traffic will come always from the contacted server and will be encrypted. In order for the client to assure the server is really the one he tried to contact, the server certificate must be trusted. In the X509 context a certificate is really a chain of certificate. A certificate is said to be trusted if there is a trusted certificate (any AFAIK) in this chain. So the client _ must _ have already a certificate from this chain available locally. Normally this happens with the CAs, whose self-signed certificates are pre- installed in browsers, distros or even by the JVM. The typical file name in ESG is ` keystore-tomcat ` . The trustore , on the contrary, holds certificates for client authentication. These are used to guarantee the server that clients are those _ trusted _ . While relying on MyProxy , the trustore must contain at least one certificate from the MyProxy certificate chain. Normally this is a common CA or the server generated SimpleCA. This way the server knows the client is to be trusted. The typical file name in ESG is ` jssecacerts ` . See: [ http://java.sun.com/j2se/1.4.2/docs/guide/security/jsse/JSSERefGuide.ht ml#Stores ](http://java.sun.com/j2se/1.4.2/docs/guide/security/jsse/JSSERefGui de.html#Stores) ### How do I check a certificate is in a tomcat truststore? A monster one-liner for checking the SSL is trusted is: test $(keytool -keystore $truststore_path -storepass $store_pass -list | grep "$(echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -noout -fingerprint -md5 | sed -n 's/^[^=]*=//p')" | wc -l) == 1 && echo "Already trusted" || echo "Not Trusted" ### How do I add a public certificate to a tomcat truststore? The result will either be ` Already trusted ` or ` Not Trusted ` in which case the certificate should be added to the node truststore. Another one-line Monster for adding the certificate to the tomcat truststore: echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'| keytool -keystore $truststore_path -storepass $store_pass -importcert -noprompt -alias $server The result will either be ` certificate added ` or an error message. In case the certificate is not in the truststore but you still get an ` alias already in use ` error message, you should just change the alias parameter to something else (e.g. ` -alias ${server}-1 ` ). The alias is only used to avoid referring to the certificate by its real name (aka. the fingerprint). After altering the truststore (or the keystore for what it matters) you MUST always restart tomcat. NOTE: If the node was installed using the esg-node script, it is simple to add a host's certificate to the local truststore. Simply use the --register flag. Ex %> esg-node --register see the [ esgf installer site ](http://esgf.org/esgf-installer-site) ### How do I create an empty keystore? simply by creating a new keystore and deleting the private key in it. #create one with a self-signed cert keytool -genkey -keyalg RSA -alias "selfsigned" -keystore empty.ks -storepass "$store_pass" -validity 360 #delete the cert keytool -delete -alias "selfsigned" -keystore empty.ks -storepass "$store_pass" ### How do I import my own .pem certificate and private key into a keystore? This is a little complicated since neither ` keytool ` nor ` openssl ` tools can achieve this. For this you'll need the extkeytool from the [ Shibboleth ](http://shibboleth.internet2.edu/) project. 1. You will have to export the key to der (pkcs8) format: openssl pkcs8 -in hostkey.pem -topk8 -nocrypt -outform DER -out hostkey.pkcs8 2. Then if you are exporting a certificate chain, concatenate every certificate from the one whose private key we've just transformed to the root self-signed CA: cat hostcert.pem CA1.pem CA2.pem ... > cert.bundle 3. Create an empty keystore as mentioned in [ How do I create an empty keystore? ](https://github.com/ESGF/esgf.github.io/wiki/Security|FAQ) 4. Use the extkeytool to import the certificate chain and key into the trustore. extkeytool -importkey -keystore empty.ks -alias tomcat -storepass $store_pass -keyfile hostkey.pkcs8 -certfile cert.bundle -provider org.bouncycastle.jce.provider.BouncyCastleProvider 5. Verify everything is in place keytool -v -list -keystore empty.ks OR If you used the esg-node installation script to setup your node you can import certificates, update certificates, create keystores, etc. The above steps are all done using the _ \--install-keypair _ option. See the [ installer page ](http://esgf.org/esgf-installer-site/) for more details on the esg-node script and it's options. _ \--install-keypair _ \- takes as input private key and public cert files and installs them. [ ] %> esg-node --install-keypair private_key.pem public_cert.pem The user is them prompted for they certificate chain all the way to the root. The user may supply a cert bundle, the individual certs and / or the URL(s) to said certs or cert chain. Also the various certificates are updated and the truststore is maintained and certs verified and displayed. The extkeytool suite is downloaded and installed and used for operations (described above). "Pretty much, use the esg-node installer to perform your key maintenance tasks, among other things. ;-) We've got you covered." -gavin ### How do I change an key-/truststore password? #if this is a keystore you'll have to change the private key pass also keytool -keypasswd -keystore keystore.ks -storepass $store_pass -alias tomcat -keypass $store_pass -new # with this command you can change the password of the store itself. keytool -storepasswd -keystore keystore.ks -storepass $store_pass -new The password of the keystore and that from the private key saved in it MUST be the same (see [ this ](https://github.com/ESGF/esgf.github.io/wiki/|bugzilla|show_bug.cgi) ) ### How do I export a certificate from the truststore? keytool -export -alias -keystore -file exportcert.pem The output format from the above command is binary.... To get text output, do the following: keytool -export -alias -keystore -rfc -file exportcert.pem If you don't know the alias of the certificate, just list everything and look for it: keytool -list -v -keystore (note: maybe your keystore or truststore file. It refers to anything file is in java keystore format. ### How do I debug tomcat SSL connection? There are two steps to perform: 1) Activate Tomcat general logging : This depends on the logging procedure selected; and the default one depends on the tomcat version. Assuming Tomcat 6.x, the default logging framework is ` java.util.logging ` (you can change it to be log4j, but this would require tampering with the library path of the tomcat installation, and therefore is not addressed here. See links for more info). The easiest way to control this is to write a logging properties file, e.g. # logging properties for apache .handlers = java.util.logging.ConsoleHandler java.util.logging.ConsoleHandler.level = FINEST java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter org.apache.catalina.level=FINEST org.apache.catalina.loader.level=INFO and tell tomcat where this is located by passing to the JVM the argument ` java.util.logging.config.file ` , e.g. java ... -Djava.util.logging.config.file="" ... Links: [ tomcat 6.0 ](http://tomcat.apache.org/tomcat-6.0-doc/logging.html) , [ (older tomcat 5.5.) ](http://tomcat.apache.org/tomcat-5.5-doc/logging.html) 2) Activate SSL debugging : in order to tell tomcat that it should also debug connections a special VM argument is also required ` javax.net.debug ` for debugging JSSE SSL. The values are one of the following: all turn on all debugging ssl turn on ssl debugging The following can be used with ssl: record enable per-record tracing handshake print each handshake message keygen print key generation data session print session activity defaultctx print default SSL initialization sslctx print SSLContext tracing sessioncache print session cache tracing keymanager print key manager tracing trustmanager print trust manager tracing pluggability print pluggability tracing handshake debugging can be widened with: data hex dump of each handshake message verbose verbose handshake message printing record debugging can be widened with: plaintext hex dump of record plaintext packet print raw SSL/TLS packets If you set it to ` help ` the above message will be displayed and the application quited. Links: [ debugging SSL Java SE 6 ](http://download.oracle.com/docs/cd/E17409_0 1/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#Debug) * For Example if you are using eclipse alter the tomcat server start script (probably in ` Run->Run Configurations...->"Tomcat vX.X Server"->Arguments ` ) and add the following VM arguments: -Djava.util.logging.config.file="" -Djavax.net.debug=ssl,handshake,verbose ... ### I might be debugging... but I get a lot of: log4j:ERROR Attempted to append to closed appender named [xxxx]. There's a problem with the logger. Use ` -Dlog4j.debug ` while starting tomcat to see what's happening. The probable cause is that there are several loggers defined with the very same name. This might even be defined across different property files which might be inside jar files!! (I had that problem with the ` eske.jar ` library) To find the files within the jars you can go to ` $CATALINA_HOME/webapps ` and execute this: find . -type f -name '*.jar' | xargs egrep '[^/]log4j\.(xml|properties)' Then you'll have to expand the jars, delete the files and then recreate the jars. There's another problem caused when an appender with the same name as that from the parent appender is added to a child category. So change this: [ 1] [ 2] [ 3] [ 4] [ 5] [ 6] [ 7] [ 8] into this: [ 1] [ 2] [ 3] [ 4] [ 5] [ 6] [ 7] ` some.class ` is already referencing ` my_log ` , so there's no need to do it twice. ` additivity="false" ` is apparently forbidding ` some.class ` _ descendants _ to write to the previous appenders, ` some.class ` itself is not being affected by this. (not clear to me, see [ this ](http://logging.apache.org/log4j/1.2/manual.html#additivity) ### What does "javax.net.ssl.SSLPeerUnverifiedException: Target is not trusted" mean? A full error being thrown from a datanode (via Thredds) may look like the below: SEVERE: Servlet.service() for servlet FileServer threw exception javax.net.ssl.SSLPeerUnverifiedException: Target is not trusted at esg.security.utils.ssl.CertUtils.retrieveCertificates(CertUtils.java:131) at esg.orp.app.AuthenticationFilter.retrieveORPCert(AuthenticationFilter.java:153) at esg.orp.app.AuthenticationFilter.attemptValidation(AuthenticationFilter.java:117) at esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:62) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662) I was able to determine that this coincides with the following lines logged in the threddsServlet.log file: WARN - esg.security.utils.ssl.CertUtils - java.security.cert.CertificateException: No name matching HOSTNAME found ERROR - esg.orp.app.AuthenticationFilter - The server at https://HOSTNAME/OpenidRelyingParty/home.htm is not trusted. This means one of several things. First check that the Datanode trusts itself, by making sure that it has an entry for itself in the esg-truststore.ts file. You can do this by running: esg-node --register localhost:443 Even if it says it's already present, it usually doesn't hurt to have the duplicate in your own truststore, but others may argue against it for size restrictions. Running the command to completion is also beneficial because it makes sure that your system has a sync'd esg-truststore.ts and jssecacerts file. Syncing these helps to be certain that if there's an error in your configuration on which truststore to use, even if it gets confused and uses the system wide file, it will match the esg required truststore. If the issue persists, the next thing to check is that the certificate on your datanode has a valid CN parameter. No matter how you generated the certificate, it is REQUIRED that the CN parameter exactly matches your datanode's fully qualified domain name. For example, if your hostname is foo.bar.com, the certificate must report that the CN=foo.bar.com. To verify this, run the following command to output information about your host certificate: $JAVA_HOME/bin/keytool -list -alias MY-HOST-ALIAS -v -keystore esg-truststore.ts NOTE: MY-HOST-ALIAS is often your hostname, but it may differ. If you're unsure how to determine which alias you should be looking at, you should run the "esg-node --register" command listed above and it will tell you what alias name the certificate was added as. If the CN does NOT match your fully qualified hostname, you will see the error above, even if the certificate is present in the truststore. If this is the case, you should regenerate your node's certificate with the proper CN. ### How do I regenerate my node's certificate? (Manually) This is a complicated operation, and there are various pieces of information on how to do this scattered around this FAQ and others. Below is a comprehensive set of steps to do this manually from scratch. There may exist other tools to help, but this is presented as a standalone procedure. #### Step 1: Locate Java's keytool program Whatever you do, do NOT confuse Java's keytool program with the GNU keytool program. They have similar syntax and will cause many headaches if you mix and match commands. If you need to, substitute every keytool command shown below with "$JAVA_HOME/bin/keytool". Also, make sure that your system has OpenSSL installed. #### Step 2: Generate a new CSR (Certificate Signing Request) using OpenSSL openssl req -out NEW-CERT.csr -new -newkey rsa:2048 -nodes -keyout NEW-KEY.key You will be prompted for information about your machine, but the most important part is that when you are prompted for the Common Name, which looks like this, Common Name (eg, YOUR name) []: you MUST enter your machine's fully qualified hostname. This step also generates a new private key file. While you don't use it right now, "keep it secret, keep it safe". NOTE : If you know which CA you plan to send this certificate to for signing, be sure to check with them on what their signing policy is. There may be specific values required that you should enter above (other than the Common Name we pointed out). #### Step 3: Ship the CSR to your CA This can either be done via e-mail, scp, etc. Each CA has their own process, so contact your CA. The CA basically takes your certificate request and outputs a signed certificate that you will use. If you are running your own CA, you can use the Globus Simple CA commands to sign a CSR as follows: $GLOBUS_LOCATION/bin/grid-ca-sign -in NEW-CERT.csr -out NEW-CERT-signed.pem The output file that needs to be shipped back is "NEW-CERT-signed.pem". A few notes about signing certificates with your CA: First is that each CA has a specific signing policy that enforces which subjects can be used in the certificate being signed. Make sure that the certificate generated in Step 1 is compliant with the target CA. Also, you will be prompted for the CA password when you issue this command to sign a certificate. If you're running Globus Simple CA and do not remember the CA password, it is available in your MyProxy configuration file _ /etc/myproxy-server.config _ . #### Step 4: Get the signed Certificate back from the CA Whether you get it from your CA via email, scp, or simply copy it from another directory, somehow locate the signed certificate. At this point, you have a signed certificate and a private key (generated from Step 1 above). #### Step 5: Convert the PEM formatted Certificate/Key to DER format Once you get it back from the CA, it will be in PEM format (a particular kind of Certificate format). We need to convert the signed certificate to DER format so that it can be used with Java. You can convert both the signed certificate and the key in two separate steps as follows: # First convert the signed certificate (note the extension difference of the output file) openssl x509 -outform der -in NEW-CERT-signed.pem -out NEW-CERT-signed.der # Then convert the private key openssl pkcs8 -topk8 -nocrypt -in NEW-KEY.key -inform PEM -out NEW-KEY.der -outform DER #### Step 6: Use ImportKey tool to create a new keystore The next step is to create a new keystore and load it up with this new certificate/key pair. If you do not have ImportKey already on your system, you can download it from here: [ http://www.agentbob.info/agentbob/79-AB.html ](http://www.agentbob.info/agentbob/79-AB.html) Download and compile the code as follows: $ wget http://www.agentbob.info/agentbob/80/version/default/part/AttachmentData/data/ImportKey.java $ javac ImportKey.java Now run the tool with the following command, using your DER cert and key as input: java ImportKey NEW-KEY.der NEW-CERT-signed.der This creates a new keystore located at ${HOME}/keystore. ImportKey and the default alias is "importkey" and the default keystore password is "importkey". #### Step 7: Rename and reset the keystore password(s) For ESG, we'll move this file to a more well known location and change the password on it. There are actually two passwords to change: 1) The password of the keystore itself, and 2) The password of the key inside the keystore. Right now, they're both "importkey" and we'll change them both below, but note that they must also match, regardless of what new password we use. We also want to rename the default alias it was created under, as some tools require it to be another name. All of those steps are shown below with comments: # Change directory to where you'd like the keystore to reside (using /usr/local/tomcat/conf as an example) cd /usr/local/tomcat/conf # Move the keystore here mv ~/keystore.ImportKey esg-keystore # Change the keystore alias (remember, the default password is "importkey") # ESG requires this alias to be named "tomcat", so that's what we change it to here keytool -changealias -alias importkey -destalias tomcat -keystore esg-keystore # Now let's change the password of the key stored inside the keystore keytool -keypasswd -alias tomcat -keypass importkey -new NEWPASSWORD -keystore esg-keystore # Now let's change the password of the keystore itself, which is separate from the password # of the encrypted key stored inside, but MUST match that value for ESG tools to work keytool -storepasswd -new NEWPASSWORD -keystore esg-keystore # Finally, verify that the configuration is correct by listing the contents using the new password keytool -list -keystore esg-keystore #### Step 8: Import our new Certificate to our esg-truststore.ts Now that we have a new certificate generated and a shiny new keystore for our node, we need to import the certificate into our truststore. In ESG, a datanode or P2P node needs to trust itself for proper ORP ( OpenIdRelyingParty ) interactions with TDS (Thredds) and/or other local Java web services. Importing the certificate to our truststore is easy: keytool -import -alias HOSTNAME -file NEW-CERT-signed.der -keystore esg-truststore.ts Substitute in the HOSTNAME field above the fully qualified domain name of your server. It should match the CN field we carefully added in Step 1 above. The esg-truststore.ts file referenced should be the official one available from the ESG Certificate page here: [ http://esgf.org/esg-certs/index.html ](http://esgf.org/esg-certs/index.html) Direct link to truststore: [ https://rainbow.llnl.gov/dist/certs/esg-truststore.ts ](https://github.com/ESGF/esgf.github.io/wiki/|dist|certs|esg-truststore.ts) #### Step 9: Update your $CATALINA_HOME/conf/server.xml Tomcat needs pointers to both your new keystore and your new truststore. Since we just created new entries that it should be using at runtime, we need to edit this configuration. Add or edit the following to the Connector section that runs on port 443: truststoreFile="/usr/local/tomcat/conf/esg-truststore.ts" truststorePass="changeit" keyAlias="tomcat" keystoreFile="/usr/local/tomcat/conf/esg-keystore" keystorePass="NEWPASSWORD" Note that the default ESG truststore password is "changeit". If you download the latest from the official page, this is already set, so do not modify that value in the configuration. The keystore password should match the NEWPASSWORD you used in Step 6 above. At this point, the new certificate and key generation is complete. Your Tomcat server should now start properly using the new certificate. ### How do I regenerate my node's certificate? (Automated: via the esg-node script) The esg-node script has been a tool used throughout the setup of the ESGF Node. The script has become a valuable tool for performing various tasks associated with installation and maintenance of the node. The following describes using the script to perform the tasks described in the previous entry in a regimented way. For info on the installer (esg-node script) goto [ http://esgf.org/esgf-installer-site/ ](http://esgf.org/esgf-installer-site/) Please be sure to copy into a safe location the tomcat truststore, keystore, the exisiting pem files you may have, and the hostcert.pem and hostkey.pem files (located in /etc/grid-security) somewhere else safe; just as a precaution. These are the affected files so it is good to have a backup... I'll automate that too when I get a chance. ![;-\)](http://esgf.org/media/images/wiki/smile4.png) These steps may also be performed using the esg-node script. (step 1-2) To create a new ssl key and csr issue the following command. %> esg-node --generate-ssl-key-and-csr This results in the following files created in your tomcat's conf directory: hostkey.pem -esg-node.csr Then follow step 3 + 4 (above) When you get the signed csr back put it in tomcat's conf directory as well and do the following: (steps 5-8) %> esg-node --install-keypair Without any args the script will look in tomcat's conf directory for the files in needs. Namely it is looking specifically for the hostkey.pem and the signed csr with the same name as the .csr file but with the .pem extension (-esg-node.pem) You will be prompted for your CA's certificate chain file(s). For your convenience we have the VeriSign chain here: http://rainbow.llnl.gov/dist/etc/verisign_chain.cer Enter this url in directly. You may enter in your own chain file(s) and/or URLs to them. Do step 9 manually (when I get a chance I will automate that too ;-) ## X509 ### How do I view/retrieve a remote certificate from a SSL connection? (e.g. to https://pcmdi3.llnl.gov) The following line retrieves the certificate from the server: echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' It can be redirected to a file: echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > serv_cert.pem Or directly displayed if piped to ` openssl x509 -text ` : echo | openssl s_client -connect $server:$port 2>&1 | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -text ### How do I export a server x509 certificate? Like mentioned above, but you would also probably want to name it .0 so use this command: #get the certificate in a proper form (cert=$(echo | openssl s_client -connect $server:$port | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'); echo "$cert" > $(echo "$cert" | openssl x509 -noout -hash).0) This is only required for exporting the server SSL, normally you just need the CA signing the server SSL (this is the same one in case of self-signed certificates for testing purposes) ## OpenId ### How do I know the openId really exists? Just point your browser to your openID. There are three outcomes I'm aware of: Result Meaning You get an xrds(xml) file Everything is fine, that's your openId which tells the requesting server where it should go for verifying your id. ` HTTP Status 404 - /esgcet/myopenid/myname ` If the path gets displayed, the servlet couldn't be found. Check it's started (in the logs or the default one is found here: /esgcet/openid/provider.htm). This could be the /esgcet component is not required for you or it's missing, check the url is right. ` HTTP Status 404 \- ` In this case the path is not being displayed, this means the servlet was found, but the servlet itself returned a 404 (in this case, the user is not found). Check the Gateway DB to see it's there. ## gridFTP That's a Datanode specific issue, check it here: [ Data Node FAQ ](https://github.com/ESGF/esgf.github.io/wiki/Cmip5DataNode|FAQ) ## Others Security in a global environment is very closely tied to the time constrain. Be sure your server (Datanode and/or Gateway) runs an NTP service and that it's synchronized. You may check it by issuing the following command ntpstat It will report the maximum possible error, which should be less than a second. A wrong set locale or changes in DST might cause problems and reject a by all other means valid certificate. Just something to consider. # Other FAQs For other security issues which could be derived from either the Gateway or the Data-node refer to the [ Gateway FAQ ](https://github.com/ESGF/esgf.github.io/wiki/Cmip5Gateway|FAQ) (especially for publication problems) or the [ ESGF Node FAQ ](https://github.com/ESGF/esgf.github.io/wiki/ESGFNode|FAQ) (especially for gridFTP problems).