Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove dependency on apr by using c++11 / c++14 features #255

Closed
wants to merge 1 commit into from

Conversation

normanmaurer
Copy link
Member

Motivation:

We had some limited use of apr left in our tcnative fork which complicated the deployment. This apr usages can be replaced by using c++.

Modification:

Remove the dependency on apr by using c++11 / c++14 features

Result:

No apr dependeny

@normanmaurer
Copy link
Member Author

not 100 % done yet. Will ask for review once I am done

Copy link
Member

@nmittler nmittler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@normanmaurer looks great! Just a few nits

CRYPTO_set_dynlock_create_callback(ssl_dyn_create_function);
CRYPTO_set_dynlock_lock_callback(ssl_dyn_lock_function);
CRYPTO_set_dynlock_destroy_callback(ssl_dyn_destroy_function);

apr_pool_cleanup_register(p, NULL, ssl_thread_cleanup,
apr_pool_cleanup_null);
//apr_pool_cleanup_register(p, NULL, ssl_thread_cleanup,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking for a good way to call these again. So stay tuned

apr_pool_cleanup_register(tcn_global_pool, NULL,
ssl_init_cleanup,
apr_pool_cleanup_null);
//apr_pool_cleanup_register(tcn_global_pool, NULL,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking for a good way to call these again. So stay tuned

#include "ssl_private.h"
#include <string.h>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed? Was it previously brought in by another header?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep was brought it by another header before.

@normanmaurer
Copy link
Member Author

@nmittler @Scottmitch PTAL... it compiles here and on my linux / windows vm. Need to update the g++ version on the ci.

@normanmaurer normanmaurer force-pushed the remove_apr branch 3 times, most recently from d242b52 to 946520b Compare March 24, 2017 00:28
Copy link
Member

@nmittler nmittler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Goodbye APR! :)

@normanmaurer normanmaurer self-assigned this Mar 28, 2017
@normanmaurer
Copy link
Member Author

@trustin PTAL as well.

@@ -1987,3 +1918,8 @@ TCN_IMPLEMENT_CALL(void, SSL, freeX509Chain)(TCN_STDARGS, jlong x509Chain)
sk_X509_pop_free(chain, X509_free);
}

void tcn_ssl_unload() {
if (ssl_init_cleanup()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we just have a single method which does both cleanups? the ssl_init_cleanup method is also called in the initialize above, but initialize also calls ssl_thread_setup so it seems like we should cleanup both anyways. It also is awkward to split the cleanup amongst 2 methods in this way.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure can do

@@ -212,13 +213,13 @@ struct tcn_ssl_ctxt_t {
/* TLS ticket key session resumption statistics */

// The client did not present a ticket and we issued a new one.
apr_uint32_t ticket_keys_new;
tcn_atomic_uint32_t ticket_keys_new;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plz fix the alignment for the variable names so they line up with the other variable names like before this change.

}

void tcn_atomic_uint32_free(tcn_atomic_uint32_t atomic) {
auto *p = (std::atomic<uint32_t> *) atomic;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider removing the temp variable.

also consider passing a ** as a parameter so you can always set *atomic = NULL; in these methods.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment applies to all the cpp files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea


if ((c = apr_pcalloc(p, sizeof(tcn_ssl_ctxt_t))) == NULL) {
tcn_ThrowAPRException(e, apr_get_os_error());
if ((c = OPENSSL_malloc(sizeof(tcn_ssl_ctxt_t))) == NULL) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider just using calloc instead of OPENSSL_malloc. IIUC OpenSSL is not aware of this variable and therefore we don't need to use OpenSSL allocation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also make sure you use free instead of OPENSSL_free

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good... this way we can also remove the memset

return ssl_context_cleanup(c);
ssl_context_cleanup(c);

// TODO: Fix me
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we fix this now ... just make this function void?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like said before I wanted to do the pr without api breakage to make it easier to consume for now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing a followup PR is fine but the return value is not used in Netty. I don't think this will break anything.

@@ -29,12 +29,14 @@ AC_CANONICAL_HOST
AC_CANONICAL_SYSTEM

${CFLAGS="-O3 -Werror -fno-omit-frame-pointer -Wunused-variable"}
${CXXFLAGS="-O3 -Werror -fno-omit-frame-pointer -Wunused-variable"}
# Ensure we support c++11 for atomics etc
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c++11 -> c++14

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

extern "C" {
#endif // __cplusplus

typedef void* tcn_lock_rw_t;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have lock_rw, lock_reader, and lock_writer. Consider being consistent/complete with the naming:

  • lock_reader_writer, lock_reader, and lock_writer
  • lock_rw, lock_r, and lock_w

Also consider making the file name consistent if you go with the more verbose names.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

}

value = (struct CRYPTO_dynlock_value *)apr_palloc(p,
value = (struct CRYPTO_dynlock_value *)OPENSSL_malloc(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider declaring and initializing this variable in the same statement.

{
UNREFERENCED(data);
CRYPTO_set_locking_callback(NULL);
CRYPTO_THREADID_set_callback(NULL);
CRYPTO_set_dynlock_create_callback(NULL);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it doesn't look like this method takes ownership of the memory [1]. We should take more care to free this memory, and if we set it check if there already exists something and free it.

[1] https://github.com/openssl/openssl/blob/OpenSSL_1_0_2-stable/crypto/cryptlib.c#L376

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also be sure to use the ssl_dyn_destroy_function (should be accessible via CRYPTO_get_dynlock_destroy_callback) on the result of CRYPTO_get_dynlock_create_callback.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm why would this be needed if we not malloc the callbacks before?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you clarify? previously a apr_palloc was used to allocate the CRYPTO_dynlock_value structure. This PR uses OPENSSL_malloc instead of the apr_palloc method. Either way there is dynamic memory being allocated. I'm not sure where/if it was freed before, but unless you find that OpenSSL is taking ownership in this situation (just setting the variable to NULL) then we need to ensure this dynamic memory is cleaned up before we lose the reference to it.

if (rv != APR_SUCCESS) {
/* TODO log that fprintf(stderr, "Failed to destroy mutex for dynamic lock %s:%d", l->file, l->line); */
}
free((void *) l->file);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the cast necessary? can we just make the type char* in the structure definition if the const is confusing things?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me see if this works.

@normanmaurer
Copy link
Member Author

normanmaurer commented Mar 28, 2017 via email

@normanmaurer
Copy link
Member Author

normanmaurer commented Mar 28, 2017 via email

/* TODO log that fprintf(stderr, "Failed to create subpool for dynamic lock"); */
return NULL;
}
value = (struct CRYPTO_dynlock_value *)malloc(sizeof(struct CRYPTO_dynlock_value));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#255 (comment)

consider declaring and initializing this variable in the same statement.


CRYPTO_set_locking_callback(NULL);
CRYPTO_THREADID_set_callback(NULL);
CRYPTO_set_dynlock_create_callback(NULL);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#255 (comment)

Can you provide a link as to where the previous callback is freed if we just set the callback to NULL? When I looked at OpenSSL src it seemed like this created a memory leak.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is not yet, will do today (and it was not freed before in tcnative as well)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think all is good now. Please ping me once you are awake and we can talk about it in more detail if you have doubts. Maybe I am just missing something but from my look into the source etc its just fine now

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't track it all the way down but found that there are links to the callbacks retained in the engines ... so ENGINE_cleanup maybe the place where these callbacks are used to clean stuff up. my concern was we were missing a call which notifies OpenSSL to use these callbacks. @davidben - can you shed some light on this situation?

}
#endif

TCN_IMPLEMENT_CALL(jint, SSL, initialize)(TCN_STDARGS, jstring engine)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#255 (comment)

it looks like this can just be void because we throw an exception upon failure anyways.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like the method return type was changed in the java header, but not here ... seems like there will be a mismatch

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah... noe done yet.

@@ -2113,3 +2043,7 @@ TCN_IMPLEMENT_CALL(jbyteArray, SSL, getOcspResponse)(TCN_STDARGS, jlong ssl) {
return value;
#endif
}

void tcn_ssl_unload() {
ssl_init_cleanup();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you combine tcn_ssl_unload and ssl_init_cleanup? seems unnecessary to have a function which only calls a single function?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@@ -87,14 +84,19 @@ static apr_status_t ssl_context_cleanup(void *data)
free(c->password);
c->password = NULL;
}

tcn_atomic_uint32_free(&(c->ticket_keys_new));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

&(c->ticket_keys_new) -> &c->ticket_keys_new ? The parens shouldn't be necessary, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right


tcn_lock_rw_t tcn_lock_rw_new() {
// Once we switch to c++17 we should use std::shared_mutex
return (tcn_lock_rw_t) new std::shared_timed_mutex();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC in C++ it is ambiguous if you try to use () for the no argument constructor without the new keyword ... new may make the situation unambiguous but consider dropping the () which may be more C++ish.

http://stackoverflow.com/a/2318731


tcn_lock_w_t tcn_lock_w_acquire(tcn_lock_rw_t lock) {
auto *m = (std::shared_timed_mutex *) lock;
return (tcn_lock_w_t) new std::lock_guard<std::shared_timed_mutex>(*m);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#255 (comment)

why not std::unique_lock?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also consider removing the temp variable so you don't have to worry about auto or types:

return (tcn_lock_w_t) new std::lock_guard<std::shared_timed_mutex>(*((std::shared_timed_mutex*) lock));

}

tcn_lock_r_t tcn_lock_r_acquire(tcn_lock_rw_t lock) {
auto *m = (std::shared_timed_mutex *) lock;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider removing the temp variable so you don't have to worry about auto or types:

return (tcn_lock_r_t) new std::shared_lock<std::shared_timed_mutex>(*((std::shared_timed_mutex*) lock));

#ifndef TCN_THREAD_H
#define TCN_THREAD_H

#include <stddef.h>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you may want to be sure you include the cpp headers if you are in cpp, and the c headers if you are in c. They may define types differently or have different behaviors.

#ifdef __cplusplus
#include <cstddef>
extern "C" {
#else
#include <stddef.h>
#endif // __cplusplus

#ifndef TCN_ATOMIC_H
#define TCN_ATOMIC_H

#include <stdint.h>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you may want to be sure you include the cpp headers if you are in cpp, and the c headers if you are in c. They may define types differently or have different behaviors.

#ifdef __cplusplus
#include <cstdint>
extern "C" {
#else
#include <stdint.h>
#endif // __cplusplus

Copy link
Member

@Scottmitch Scottmitch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some small cleanup ... and just the memory management question left then lgtm.

/*
* Dynamic lock creation callback
*/
static struct CRYPTO_dynlock_value *ssl_dyn_create_function(const char *file,
int line)
{
struct CRYPTO_dynlock_value *value;
apr_pool_t *p;
apr_status_t rv;
value = (struct CRYPTO_dynlock_value *)malloc(sizeof(struct CRYPTO_dynlock_value));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#255 (comment)

consider declaring and initializing this variable in the same statement.

struct CRYPTO_dynlock_value* value = (struct CRYPTO_dynlock_value*) malloc(sizeof(struct CRYPTO_dynlock_value));

@@ -106,7 +99,7 @@ struct TCN_bio_bytebuffer {
* so that the return value for SSL_OP_FUTURE_WORKAROUND will only be
* reported by versions that actually support that specific workaround.
*/
static const jint supported_ssl_opts = 0
static const jlong supported_ssl_opts = 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets hold off on this change and just rebase when #258 is merged

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1


CRYPTO_set_locking_callback(NULL);
CRYPTO_THREADID_set_callback(NULL);
CRYPTO_set_dynlock_create_callback(NULL);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't track it all the way down but found that there are links to the callbacks retained in the engines ... so ENGINE_cleanup maybe the place where these callbacks are used to clean stuff up. my concern was we were missing a call which notifies OpenSSL to use these callbacks. @davidben - can you shed some light on this situation?

@@ -74,7 +71,7 @@ static apr_status_t ssl_context_cleanup(void *data)
}
c->alpn_proto_len = 0;

apr_thread_rwlock_destroy(c->mutex);
tcn_lock_rw_free(&(c->ticket_keys_lock));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

&(c->ticket_keys_lock) -> &c->ticket_keys_lock

@@ -2113,3 +2041,4 @@ TCN_IMPLEMENT_CALL(jbyteArray, SSL, getOcspResponse)(TCN_STDARGS, jlong ssl) {
return value;
#endif
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kill the extra new line

@normanmaurer normanmaurer force-pushed the remove_apr branch 3 times, most recently from 56646cd to 21c6ffb Compare March 29, 2017 20:29
@normanmaurer normanmaurer added this to the 2.0.1.Final milestone Mar 29, 2017
Copy link
Member

@trustin trustin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Way to go!

}

#if OPENSSL_VERSION_NUMBER < 0x10100000L
if (SSLeay() < 0x0090700L) {
TCN_FREE_CSTRING(engine);
tcn_ThrowAPRException(e, APR_EINVAL);
tcn_ThrowException(e, "openssl version too old");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about appending the version number and the minimum required version number?

@@ -29,12 +29,14 @@ AC_CANONICAL_HOST
AC_CANONICAL_SYSTEM

${CFLAGS="-O3 -Werror -fno-omit-frame-pointer -Wunused-variable"}
${CXXFLAGS="-O3 -Werror -fno-omit-frame-pointer -Wunused-variable"}
# Ensure we support c++14 for atomics etc
${CXXFLAGS="-O3 -Werror -fno-omit-frame-pointer -Wunused-variable -std=c++14"}
Copy link
Member

@trustin trustin Mar 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps should we static-link libgcc and libstdc++? -static-libgcc -static-libstdc++
Also, we may eventually want to link against musl or uClibc instead of GLIBC so our binary works on more distros. http://stackoverflow.com/a/26306630

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we not want to do any static linking in our dynamic build. Not sure about the "-static" ones tho. @Scottmitch @nmittler WDYT ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if the situation has changed, but I had an impression that libstdc++ has worse backward ABI compatibility historically. Maybe I'm completely wrong, though.

Copy link
Member

@Scottmitch Scottmitch Mar 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets handle this as a followup PR if it becomes an issue or is known to improve things.

Copy link
Member

@Scottmitch Scottmitch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few more comments then lgtm


#ifndef OPENSSL_NO_ENGINE
if (J2S(engine)) {
ENGINE *ee = NULL;
apr_status_t err = APR_SUCCESS;
int err = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just use bool?

@@ -721,19 +656,11 @@ TCN_IMPLEMENT_CALL(jint, SSL, initialize)(TCN_STDARGS, jstring engine)
if (r) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(not introduce by this PR, but while we are here) r is an integer type ... lets use it like an integer: r != 0

#ifndef TCN_ATOMIC_H
#define TCN_ATOMIC_H


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: kill line

#ifndef TCN_THREAD_H
#define TCN_THREAD_H


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: kill line

@Scottmitch
Copy link
Member

@normanmaurer - are you planning on pulling this in?

@normanmaurer
Copy link
Member Author

normanmaurer commented May 7, 2017 via email

Motivation:

We had some limited use of apr left in our tcnative fork which complicated the deployment. This apr usages can be replaced by using c++.

Modification:

Remove the dependency on apr by using c++11 / c++14 features

Result:

No apr dependeny
@normanmaurer
Copy link
Member Author

@Scottmitch PTAL again.. rebased on latest master and should are ready to get pulled in.

@normanmaurer
Copy link
Member Author

After @Scottmitch mentioned to me that centos 6 glibc may not support c++14 I checked in more detail and found out it even not support c++11 . So I guess we will need to life with apr ...

@Scottmitch WDYT ?

ssl_lock_cs = apr_palloc(p, ssl_lock_num_locks * sizeof(*ssl_lock_cs));
if (!ssl_initialized)
return;
ssl_initialized = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to consider concurrency/visibility for ssl_initialized? can we just remove ssl_initialized and consider it a programing error if initialized is called multiple times (can this happen in Netty)?

goto cleanup;
}

c->protocol = protocol;
c->mode = mode;
c->ctx = ctx;
c->pool = p;
c->ticket_keys = tcn_atomic_uint32_new();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this correct? ticket_keys is of type tcn_ssl_ticket_key_t but this is assigning it to a new atomic uint32 of type void*, right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the intention that this should be ticket_keys_new and ticket_keys should be initialized in some other way? It is freed above using OPENSSL_free.

@Scottmitch
Copy link
Member

@normanmaurer - I would like to still remove APR ... we should investigate building a glibc shared library, either publish these instructions or the library, and have folks on older distributions depend upon this instead of apr. Newer distributions which have glibc that support c++14 (eventually every one over time) will require not additional dependency. The statically compiled artifacts may be a bit more tricky due to licensing of glibc.

@kvc-code
Copy link

How does this affect the ability to build against musl C library on Alpine? At present we at least have the ability to build for ourselves on Alpine but sounds like this change may lock that out as well.

@normanmaurer normanmaurer modified the milestones: 2.0.7.Final, 2.0.8.Final Nov 3, 2017
@normanmaurer normanmaurer modified the milestones: 2.0.8.Final, 2.0.9.Final Apr 13, 2018
@normanmaurer normanmaurer force-pushed the master branch 2 times, most recently from 0d59905 to 4937622 Compare May 31, 2018 06:51
@normanmaurer normanmaurer removed this from the 2.0.11.Final milestone Mar 22, 2019
@normanmaurer
Copy link
Member Author

Let me close this for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants