pg gem fails to compile with chef 12.0 #212

jwmarshall · 2014-12-05T23:09:23Z

Just ran into this issue when my omnibus plugin for vagrant upgraded the chef-client from 11.16 to 12.0.

Fixed by pinning to last known 11.x version that worked.

config.omnibus.chef_version = '11.16.4'

The text was updated successfully, but these errors were encountered:

hulu1522 · 2014-12-06T00:17:42Z

Same here...

Somewhere something has changed with the embedded ruby in chef-12.0.0 and causes the whole thing to fail.

Here in the postgresql::ruby the check for omnibus is successful:
https://github.com/hw-cookbooks/postgresql/blob/develop/recipes/ruby.rb#L58
[2014-12-06T00:11:09+00:00] WARN: [["chef"]]

When it tries to do the omnibus fix it fails:

  [2014-12-06T00:11:09+00:00] WARN: Failed to properly build pg gem. Forcing properly linking and retrying (omnibus fix)
  * execute[generate pg gem Makefile] action run       [2014-12-06T00:11:09+00:00] INFO: Processing execute[generate pg gem Makefile] action run (postgresql::ruby line 86)
checking for pg_config...        yes
Using config values from /usr/bin/pg_config
checking for libpq-fe.h...        yes
       checking for libpq/libpq-fs.h...
       yes
       checking for pg_config_manual.h...
       yes
       checking for PQconnectdb() in -lpq...
       no
       checking for PQconnectdb() in -llibpq...
       no
       checking for PQconnectdb() in -lms/libpq...
       no
       Can't find the PostgreSQL client library (libpq)
       *** extconf.rb failed ***
       Could not create Makefile due to some reason, probably lack of necessary
       libraries and/or headers.  Check the mkmf.log file for more details.  You may
       need configuration options.

       Provided configuration options:
        --with-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/opt/chef/embedded/bin/ruby
        --with-pg
        --without-pg
        --with-pg-config
        --without-pg-config
        --with-pg_config
        --without-pg_config
        --with-pg-dir
        --without-pg-dir
        --with-pg-include
        --without-pg-include=${pg-dir}/include
        --with-pg-lib
        --without-pg-lib=${pg-dir}/lib
        --with-pqlib
        --without-pqlib
        --with-libpqlib
        --without-libpqlib
        --with-ms/libpqlib
        --without-ms/libpqlib

    ================================================================================
    Error executing action `run` on resource 'execute[generate pg gem Makefile]'
    ================================================================================

Sorry for this set back but this is a major issue unless people lock their chef version.

EDIT:
I also heard somewhere that chef-12.0.0 deprecated something to do with paths which might explain why this is failing: https://github.com/hw-cookbooks/postgresql/blob/develop/recipes/ruby.rb#L88

EDIT 2:
This is what I got according to the issue reported with chef (chef/chef#2545) but once I implement the change it fails to find generate in the command:

    lib_builder = execute 'generate pg gem Makefile' do
      # [COOK-3490] pg gem install requires full path on RHEL
      environment "PATH" => "#{ENV['PATH']}:/usr/pgsql-#{node['postgresql']['version']}/bin #{RbConfig.ruby} extconf.rb"
      cwd ext_dir
      action :nothing
    end
    lib_builder.run_action(:run)

jtimberman · 2014-12-06T22:07:54Z

As I explained, this is not due to the removal of the path attribute as the postgresql::source recipe doesn't use that at all.

Digging into this, I was able to reproduce the build failure. The mkmf.log has these errors:

conftest.c: In function ‘t’:
conftest.c:15:1: error: too few arguments to function ‘PQconnectdb’
 int t(void) { PQconnectdb(); return 0; }
 ^
In file included from conftest.c:3:0:
/usr/include/postgresql/libpq-fe.h:250:16: note: declared here
 extern PGconn *PQconnectdb(const char *conninfo);
                ^
checked program was:
/* begin */
 1: #include "ruby.h"
 2:
 3: #include <libpq-fe.h>
 4:
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12:
13:   return 0;
14: }
15: int t(void) { PQconnectdb(); return 0; }
/* end */

And later,

/usr/bin/ld: cannot find -llibpq
collect2: error: ld returned 1 exit status

These errors are repeated a couple times, I'm sure if you look at mkmf.log on your system (/opt/chef/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/pg-0.17.1/mkmf.log), you'll see similar issues.

Running chef 11.16.4 using the omnibus package worked, and I believe it's because that version uses Ruby 1.9.3. Compare Chef 11.16.4's embedded ruby:

vagrant@ruby-ubuntu-1404:~$ /opt/chef/embedded/bin/ruby --version
ruby 1.9.3p547 (2014-05-14 revision 45962) [x86_64-linux]

Versus Chef 12.0.0's:

vagrant@ruby-ubuntu-1404:~$ /opt/chef/embedded/bin/ruby --version
ruby 2.1.4p265 (2014-10-27 revision 48166) [x86_64-linux]

The requirements of the pg RubyGem mentions:

Ruby 1.9.3-p392, or 2.0.0-p0.

It's possible it hasn't been tested, and may not work, on Ruby 2.1.0+.

mshappe · 2014-12-09T00:40:10Z

The PG gem works fine with Ruby 2.1.2 -- I use it all the time.

It just doesn't work with Chef for some bizarre reason. I just ran into this, myself.

dpresling · 2014-12-09T02:58:26Z

Same issue here.

mshappe · 2014-12-09T16:51:05Z

Similar to the OP, I was able to work around it in .kitchen.yml by forcing Chef Omnibus back to 11.16.4:

driver:
  name: vagrant
  require_chef_omnibus: 11.16.4

Of course, that ONLY works for kitchen and not for, say, knife bootstrap.

One way or the other, moral of the story is to pin your versions...

While I haven't tried it, it's also possible that the issue could be worked around the "long way", for people who really need 12, by dropping the omnibus altogether and instead installing chef as a gem in sequence with others. I remember seeing this as an alternative while googling around. As a relative newbie, though, I don't much like the idea. The Omnibus is a good idea. It's just breaking stuff, here...

lamont-granquist · 2014-12-09T18:03:25Z

It seems like this is pretty clear about what the problem is:

       checking for PQconnectdb() in -lpq...
       no
       checking for PQconnectdb() in -llibpq...
       no
       checking for PQconnectdb() in -lms/libpq...
       no
       Can't find the PostgreSQL client library (libpq)

So what is responsible for installing the pg libs and where are they and why can't extconf.rb find them?

mshappe · 2014-12-09T18:09:03Z

What it seems like and what it is are two different universes :-)

The library is installed. I checked. /usr/lib/libpq.{a,so}* are all present. nm /usr/lib/libpq.a shows the "missing" symbol.

Also, this works fine with the 11.16.4 omnibus.

jdstuart · 2014-12-09T18:28:19Z

This is the same as chef/chef#2587. Where's the problem. Should the recipe be fixed, or is it a bug in Chef?

lamont-granquist · 2014-12-09T18:29:26Z

What is the complete output of mkmf.log?

mshappe · 2014-12-09T18:34:07Z

@lamont-granquist I'll unpin and reconverge and let you know in a little while. I do remember that the error didn't really seem to make a lot of sense, though 😄

lamont-granquist · 2014-12-09T18:47:33Z

Yeah mkmf.log never makes any sense, part of the reason why I need to see the whole thing in context...

mshappe · 2014-12-09T18:54:32Z

find_executable: checking for pg_config... -------------------- yes

--------------------

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -fstack-protector -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib          -L/opt/chef/embedded/lib -lruby  -lpthread -ldl -lcrypt -lm   -lc"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -fstack-protector -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib          -L/opt/chef/embedded/lib -lruby  -Wl,-rpath,/usr/lib -lpthread -ldl -lcrypt -lm   -lc"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main() {return 0;}
/* end */

find_header: checking for libpq-fe.h... -------------------- yes

"gcc -E -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC  conftest.c -o conftest.i"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <libpq-fe.h>
/* end */

--------------------

find_header: checking for libpq/libpq-fs.h... -------------------- yes

"gcc -E -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC  conftest.c -o conftest.i"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <libpq/libpq-fs.h>
/* end */

--------------------

find_header: checking for pg_config_manual.h... -------------------- yes

"gcc -E -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC  conftest.c -o conftest.i"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <pg_config_manual.h>
/* end */

--------------------

have_library: checking for PQconnectdb() in -lpq... -------------------- no

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -fstack-protector -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib    -Wl,-rpath,/usr/lib       -L/opt/chef/embedded/lib -lruby -lpq  -lpthread -ldl -lcrypt -lm   -lc"
/usr/lib/libpq.so: undefined reference to `SSL_CTX_use_certificate_chain_file@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_write@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_set_fd@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_use_PrivateKey_file@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `CRYPTO_set_locking_callback@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_NAME_get_text_by_NID@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_connect@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_init@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_STORE_load_locations@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_get_cert_store@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_ctrl@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_free@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_library_init@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_ctrl@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ERR_get_error@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_pending@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_free@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_get_subject_name@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_use_certificate_file@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_check_private_key@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_load_error_strings@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_by_id@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_get_peer_certificate@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_new@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `CRYPTO_num_locks@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_load_private_key@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_finish@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_set_verify@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_free@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `CRYPTO_set_id_callback@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_get_error@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_new@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_shutdown@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_use_PrivateKey@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `TLSv1_method@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_STORE_set_flags@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_read@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `OPENSSL_config@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_load_verify_locations@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ERR_reason_error_string@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_set_ex_data@OPENSSL_1.0.0'
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { void ((*volatile p)()); p = (void ((*)()))PQconnectdb; return 0; }
/* end */

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -fstack-protector -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib    -Wl,-rpath,/usr/lib       -L/opt/chef/embedded/lib -lruby -lpq  -lpthread -ldl -lcrypt -lm   -lc"
conftest.c: In function ‘t’:
conftest.c:15:1: error: too few arguments to function ‘PQconnectdb’
 int t(void) { PQconnectdb(); return 0; }
 ^
In file included from conftest.c:3:0:
/usr/include/postgresql/libpq-fe.h:250:16: note: declared here
 extern PGconn *PQconnectdb(const char *conninfo);
                ^
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { PQconnectdb(); return 0; }
/* end */

--------------------

have_library: checking for PQconnectdb() in -llibpq... -------------------- no

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -fstack-protector -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib    -Wl,-rpath,/usr/lib       -L/opt/chef/embedded/lib -lruby -llibpq  -lpthread -ldl -lcrypt -lm   -lc"
/usr/bin/ld: cannot find -llibpq
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { void ((*volatile p)()); p = (void ((*)()))PQconnectdb; return 0; }
/* end */

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -fstack-protector -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib    -Wl,-rpath,/usr/lib       -L/opt/chef/embedded/lib -lruby -llibpq  -lpthread -ldl -lcrypt -lm   -lc"
conftest.c: In function ‘t’:
conftest.c:15:1: error: too few arguments to function ‘PQconnectdb’
 int t(void) { PQconnectdb(); return 0; }
 ^
In file included from conftest.c:3:0:
/usr/include/postgresql/libpq-fe.h:250:16: note: declared here
 extern PGconn *PQconnectdb(const char *conninfo);
                ^
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { PQconnectdb(); return 0; }
/* end */

--------------------

have_library: checking for PQconnectdb() in -lms/libpq... -------------------- no

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -fstack-protector -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib    -Wl,-rpath,/usr/lib       -L/opt/chef/embedded/lib -lruby -lms/libpq  -lpthread -ldl -lcrypt -lm   -lc"
/usr/bin/ld: cannot find -lms/libpq
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { void ((*volatile p)()); p = (void ((*)()))PQconnectdb; return 0; }
/* end */

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -fstack-protector -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib    -Wl,-rpath,/usr/lib       -L/opt/chef/embedded/lib -lruby -lms/libpq  -lpthread -ldl -lcrypt -lm   -lc"
conftest.c: In function ‘t’:
conftest.c:15:1: error: too few arguments to function ‘PQconnectdb’
 int t(void) { PQconnectdb(); return 0; }
 ^
In file included from conftest.c:3:0:
/usr/include/postgresql/libpq-fe.h:250:16: note: declared here
 extern PGconn *PQconnectdb(const char *conninfo);
                ^
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { PQconnectdb(); return 0; }
/* end */

--------------------

hulu1522 · 2014-12-09T18:58:17Z

Here is a link to my mkmf.log also:
http://pastebin.com/P600g6p4

lamont-granquist · 2014-12-09T19:09:59Z

So its most likely this one I think:

have_library: checking for PQconnectdb() in -lpq... -------------------- no

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/chef/embedded/include   -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -fstack-protector -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib    -Wl,-rpath,/usr/lib       -L/opt/chef/embedded/lib -lruby -lpq  -lpthread -ldl -lcrypt -lm   -lc"
/usr/lib/libpq.so: undefined reference to `SSL_CTX_use_certificate_chain_file@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_write@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_set_fd@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_use_PrivateKey_file@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `CRYPTO_set_locking_callback@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_NAME_get_text_by_NID@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_connect@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_init@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_STORE_load_locations@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_get_cert_store@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_ctrl@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_free@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_library_init@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_ctrl@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ERR_get_error@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_pending@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_free@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_get_subject_name@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_use_certificate_file@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_check_private_key@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_load_error_strings@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_by_id@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_get_peer_certificate@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_new@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `CRYPTO_num_locks@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_load_private_key@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_finish@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_set_verify@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_free@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `CRYPTO_set_id_callback@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_get_error@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_new@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_shutdown@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_use_PrivateKey@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `TLSv1_method@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_STORE_set_flags@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_read@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `OPENSSL_config@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_load_verify_locations@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ERR_reason_error_string@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_set_ex_data@OPENSSL_1.0.0'
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { void ((*volatile p)()); p = (void ((*)()))PQconnectdb; return 0; }
/* end */

A lot of the tests aren't relevant, we need to look only at the -lpq ones, and most of the other ones blow up in the parser as ruby steps through different C syntaxes. This one parsed and compiled but failed to link. It looks like we're getting a conflict between the embedded openssl and the system one. Not sure how solvable that is, because ruby needs to load the openssl from embedded, but if the system libpg can't link against that one, you just can't load that into the same C process.

This smells like a resurrection of the old problem we had with this cookbook, details have paged out of my brain but there's a COOK ticket somewhere.

hulu1522 · 2014-12-09T19:13:22Z

Don't know if this is needed but I thought I would post a link to the successful build mkmf.log file when using Chef 11.16.4 and it's ruby 1.9.1

http://pastebin.com/HxFAAa51

lamont-granquist · 2014-12-09T19:18:05Z

Yeah @hulu1522 that's useful, here's the same test, passing:

have_library: checking for PQconnectdb() in -lpq... -------------------- yes

"gcc -o conftest -I/opt/chef/embedded/include/ruby-1.9.1/x86_64-linux -I/opt/chef/embedded/include/ruby-1.9.1/ruby/backward -I/opt/chef/embedded/include/ruby-1.9.1 -I. -I/usr/include/postgresql    -I/opt/chef/embedded/include -I/opt/chef/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/chef/embedded/lib -L/usr/lib -L/opt/chef/embedded/lib -L.  -L/opt/chef/embedded/lib -rdynamic  -L/opt/chef/embedded/lib    -Wl,-rpath,/usr/lib       -L/opt/chef/embedded/lib -lruby -lpq  -lpthread -lrt -ldl -lcrypt -lm   -lc"
checked program was:
/* begin */
 1: #include "ruby.h"
 2:
 3: #include <libpq-fe.h>
 4:
 5: /*top*/
 6: int t() { void ((*volatile p)()); p = (void ((*)()))PQconnectdb; return 0; }
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12:
13:   return 0;
14: }
/* end */

--------------------

wonder if this is related to the LIBPATH stuff in mkmf.rb that trolled AIX builds so hard...

mshappe · 2014-12-09T19:22:07Z

@lamont-granquist This absolutely looks like a resurrection of the old issue that turned out to be incompatible SSL libraries or something bizarre like that. I was at first really crabby because I assumed it was that issue and it still wasn't fixed, but pinning back to 11.16.4 suggests instead that this is a case of the issue being a hydra :-)

lamont-granquist · 2014-12-09T19:26:51Z

Yeah, I think it may be ruby 2.1 did some changes that re-introduced the problem. Wouldn't show up on normal non-omnibus-chef pg-gem on ruby 2.1.

And, the two gcc compile lines only differ by the new one having -fstack-protector added, which shouldn't have anything to do with this problem.

lamont-granquist · 2014-12-09T19:33:13Z

Try editing /opt/chef/embedded/lib/ruby/2.1.0/x86_64-linux/rbconfig.rb, and delete this line:

  CONFIG["LIBPATHENV"] = "LD_LIBRARY_PATH"

(or whatever its set to, but its probably set to that on linuxen)

lamont-granquist · 2014-12-09T21:26:50Z

We ship openssl 1.0.1j in chef-11 as well so that didn't change.

mshappe · 2014-12-09T21:28:26Z

Hey, I didn't say any of this actually made sense :-)

I've had other fish to fry this afternoon, but I'll try the above rbconfig.rb thing when I can.

mshappe · 2014-12-10T18:35:36Z

That solution works. Of course, that's not a practical solution for actually using, say, knife solo in an automatic way going forward, but it did make things work.

Full procedure:

ran knife solo bootstrap... and let it fail.
vagrant ssh and fix up the file as suggested above
knife solo cook... and it succeeded.

lamont-granquist · 2014-12-10T18:40:05Z

Heh,yeah, so we can work on ripping that out of the base ruby that we ship. I knew about that issue offhand because it already trolled us hard when we were building omnibus chef on AIX, so we already have some workarounds that we can just apply on every platform.

mshappe · 2014-12-10T18:40:38Z

...AIX... You are brave! 😄

lamont-granquist · 2014-12-10T18:47:02Z

HAHAHAHAHAHAHAHHAHAHA, luckily for me we hired some other sucker to take it over (@scotthain), but i still wind up learning more than i ever wanted to about it...

scotthain · 2014-12-10T18:49:01Z

I don't know what you're talking about @lamont-granquist ;)

hulu1522 · 2014-12-10T19:29:45Z

For those using Test-Kitchen this is a TEMP work around until the fix is merged:

fix = Chef::Util::FileEdit.new("/opt/chef/embedded/lib/ruby/2.1.0/x86_64-linux/rbconfig.rb")

fix.search_file_delete_line("^.*LIBPATHENV.*$")

fix.write_file

Just put it in your recipe.

mshappe · 2014-12-11T18:29:57Z

As workarounds go, that's not terrible, but it would be nice if the issue itself did not keep rearing its head every time a version changes! 😄 (I know, I'm demanding!)

nlopes · 2015-07-14T15:59:20Z

Can this be re-opened @amdprophet ?

patcon · 2015-07-14T18:22:42Z

FYI I was working on spiking out a pg-chef-gem cookbook as an analog to the approach of the mysql2_chef_gem cookbook.

It's not yet working and I had to move on to something else for now, but I can't see why it shouldn't be possible, if someone wants to submit PRs. A custom postgresql is required for now, and can be found in the berksfile. test-kitchen suite should be functional.

lazzarello · 2015-09-02T21:40:31Z

Pls reopen. Same issue as @nlopes Debian 7 with ChefDK released .deb.

Also, hi @patcon

vblessing · 2015-09-22T15:15:51Z

I'm seeing this issue with Ubuntu 14.04 and Chef Client 12.4.2, getting the same output as @sfroehler.
It's looking like no one really has a good answer to this?

epu · 2015-09-30T18:16:44Z

Hit this with chefdk 7.0 / chef-solo 12.4.1. Reverted to chef-solo 11.14.2 for testing, deploying.

jpeyret · 2015-10-07T02:27:13Z

Yeah, hitting this too. Maybe this hacky workaround will give someone more capable a bit of insight.

vagrant@vagrant:~$ chef-client -v
Chef: 12.4.3

Vagrant 1.7.4
Ubuntu 14.04 guest on Mac 10.9.5 host

audrey:ubuntu2 jluc$ chef -v
Chef Development Kit Version: 0.7.0
chef-client version: 12.4.1
berks version: 3.2.4
kitchen version: 1.4.2

workaround:
I've added build essential to my wrapper bemyerp-postgresql's metadata.rb

depends 'build-essential' #this was added
depends 'postgresql'

Now, this is not the only cookbook to require build-essential in my build, so I found it odd that I would encounter this problem.

Apparently however, if my bemyerp-postgresql cookbooks recipe are first in the sequence, before other cookbooks using build-essential, then things are ok. If other cookbooks using build_essential go first, then fail.

Could it be that postgresql's use of build_essential requires it to be 'just so'. Which it gets if it goes first, but not if it some other cookbooks gets there first?

JackChance · 2015-10-28T23:23:25Z

test-kitchen 1.4.2
chef-dk 0.8.0
hostbox: windows 7
guestbox: OEL 6.5
locked to omnibus version 12.3.0 (dsc resources fail nastily on 12.4+)

Running into this same issue on test kitchen when attempting to install berkflow 0.7.0

gem_package 'berkflow' do
  version '0.7.0'
  options('--no-document --no-user-install --install-dir /opt/chefdk/embedded/lib/ruby/gems/2.1.0')
  action :install
end

Attempted the fix mentioned above (no difference in behavior)

fix = Chef::Util::FileEdit.new('/opt/chef/embedded/lib/ruby/2.1.0/x86_64-linux/rbconfig.rb')
fix.search_file_delete_line('^.*LIBPATHENV.*$')
fix.write_file

The error claims that a file was made at /opt/chefdk/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/ffi-1.9.10/gem_make.out but the only one on the system was /opt/chefdk/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/ffi-1.9.8/gem_make.out, contents are

/opt/chef/embedded/bin/ruby -r ./siteconf20150428-27228-1lwal9u.rb extconf.rb
checking for ffi_call() in -lffi... yes
checking for ffi_prep_closure()... yes
checking for ffi_raw_call()... yes
checking for ffi_prep_raw_closure()... yes
checking for rb_thread_blocking_region()... yes
checking for rb_thread_call_with_gvl()... yes
checking for rb_thread_call_without_gvl()... yes
checking for ffi_prep_cif_var()... yes
creating extconf.h
creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling StructByValue.c
compiling FunctionInfo.c
compiling Type.c
compiling ClosurePool.c
compiling Buffer.c
compiling LastError.c
compiling Variadic.c
compiling Platform.c
compiling LongDouble.c
compiling Types.c
compiling Pointer.c
compiling AbstractMemory.c
compiling Function.c
compiling StructByReference.c
compiling MemoryPointer.c
compiling MappedType.c
compiling StructLayout.c
compiling DynamicLibrary.c
compiling DataConverter.c
compiling Call.c
compiling ArrayType.c
compiling ffi.c
compiling Struct.c
compiling MethodHandle.c
compiling Thread.c
linking shared-object ffi_c.so

make "DESTDIR=" install
/usr/bin/install -c -m 0755 ffi_c.so ./.gem.20150428-27228-sdvhfc
installing default ffi_c libraries

If I manually install the berkflow gem with

chef gem install berkflow -v 0.7.0 --no-user-install

it works as expected.

On a side note, execute 'chef gem install berkflow -v 0.7.0 --no-user-install' places it on a strange path under root when I would expect it to be under /opt/chefdk/.../bin

I'm trying to make a pre-configured chef workstation with the gems my org adds in for our workflow

acqant · 2015-12-04T19:17:00Z

+1 Seeing this on mysql gem.
chef 12.3.0
trying the "fix" now.

qhartman · 2015-12-04T23:23:02Z

Also hitting this on Ubuntu 14.04 w/ chef-client 12.3.0. Trying to install pg for an lwrp I wrote, have libpq-dev installed at the system level.

acqant · 2015-12-05T00:02:16Z

What's odd for me is it's failing running this
/opt/sensu/embedded/bin/ruby extconf.rb

It didn't even get to the part where it tried to compile it or at least didn't log it.
Running chef a second time works however.

I put ignore_failures true for the gem install just so jenkins would stop failing and the gem eventually gets installed but it's not a gem I need other that would be problematic.

patcon · 2015-12-05T00:54:12Z

Try setting default['apt']['compile_time_update'] = true. Build essential is required at compile time as part of the pg gem install process, but you need to manually make sure that apt-get update is running before that.

I seem to remember some reason this couldn't be done automatically, but maybe I'm wrong...

acqant · 2015-12-07T17:27:57Z

tried the compile time, didn't work.
tried downgrading chef, didn't work.

What's odd is it's not even getting to the point of compiling the gem it fails earlier than that.

qhartman · 2015-12-07T17:56:47Z

Yeah, I've tried all the recommended work-around to no avail. For me, it can't find the libpq-dev dependency even though it is installed and the pg gem installs for the system ruby just fine. In my case since there doesn't seem to be a direct solution to this readily available I'm going to rewrite my lwrp to shell out to psql commands. It's going to be a bit of a PITA, but it's better than not working at all.

patcon · 2015-12-07T20:19:24Z

Ah, just ran into this again yesterday when working on a new cookbook, and saw that I needed to set build-essentil['compile_time'' to true manually, even though it was supposedly in the postgresql::ruby recipe. But I did get it working. I seem to remember a related "normal" level attribute set in some other cookbook, so I'd recommend override, rather than the "default" level I suggested above.

acqant · 2015-12-08T14:20:42Z

I changed the order of the gem install and it's better. I got the same
error on another gem ( zookeeper ) but that hasn't repeated.

What is the root cause of all this? This isn't just a pg gem issue.

On Mon, Dec 7, 2015 at 3:19 PM, Patrick Connolly notifications@github.com
wrote:

Ah, just ran into this again yesterday when working on a new cookbook, and
saw that I needed to set build-essentil['compile_time'' to true manually,
even though it was supposedly in the postgresql::ruby recipe. But I did get
it working. I seem to remember a related "normal" level attribute set in
some other cookbook, so I'd recommend override, rather than the "default"
level I suggested above.

—
Reply to this email directly or view it on GitHub
#212 (comment)
.

qhartman · 2015-12-08T16:21:25Z

It has something to do with how the dependency paths for the omnibus components are set. I've lifted the solution from the postgres cookbook referenced above into my LWRP and it seems to work: https://github.com/hw-cookbooks/postgresql/blob/develop/recipes/ruby.rb#L32

patcon · 2015-12-08T17:57:17Z

What is the root cause of all this?

I think the mysql2_chef_gem cookbook has the proper future-oriented approach (see my comment above for link to abandonned WIP)

acqant · 2016-02-25T15:41:51Z

following up, in my case the problem was lack of memory. I changed my resource to run
gem install --debug and saw the error about not being able to allocate memory.

ashwinrayaprolu · 2016-08-28T03:21:10Z

I'm getting this. This is first time i'm trying to install mysql on ubuntu in vagrant environment using chef provisioner

Vagrant 1.8.5
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
Berks 4.3.5

depends 'apt', '> 2.9'
depends 'firewall', '> 2.4'
depends 'apache2', '> 3.2.2'
depends 'mysql', '> 8.0'
depends 'mysql2_chef_gem', '> 1.0'
depends 'database', '> 5.1'

"gcc -o conftest -I/opt/chef/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/chef/embedded/include/ruby-2.1.0/ruby/backward -I/opt/chef/embedded/include/ruby-2.1.0 -I. -I/usr/local/include -I/opt/chef/embedded/include -O2 -O3 -g -pipe -I/opt/chef/embedded/include -I/opt/chef/embedded/include -O2 -O3 -g -pipe -fPIC conftest.c -L. -L/opt/chef/embedded/lib -Wl,-R/opt/chef/embedded/lib -L/usr/local/lib -Wl,-R/usr/local/lib -L/opt/chef/embedded/lib -Wl,-R/opt/chef/embedded/lib -L. -Wl,-rpath,/opt/chef/embedded/lib -fstack-protector -L/opt/chef/embedded/lib -rdynamic -Wl,-export-dynamic -L/opt/chef/embedded/lib -Wl,-R/opt/chef/embedded/lib -lnsl -lz -lm -Wl,-R -Wl,/opt/chef/embedded/lib -L/opt/chef/embedded/lib -lruby -lmysqlclient -lnsl -lz -lm -lpthread -lrt -ldl -lcrypt -lm -lc"
/usr/bin/ld: cannot find -lmysqlclient
collect2: error: ld returned 1 exit status

Any solution?

patcon · 2016-08-29T18:09:34Z

You're in the wrong cookbook issue queue, @ashwinrayaprolu1984 ;)

Just came across this mysql, so you'll want to drop a +1 here:
sous-chefs/mysql#457

ashwinrayaprolu · 2016-08-29T21:13:01Z

I saw this just now . Thanks for reply i had fixed it in meantime though.

Allow to set LANG and LC_ALL in envvars

lock · 2018-07-24T10:26:23Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

lamont-granquist mentioned this issue Dec 9, 2014

Chef v12.0.0 Error executing action install on resource 'chef_gem[pg]' chef/chef#2587

Closed

lamont-granquist mentioned this issue Dec 10, 2014

remove libpath on all platforms chef/omnibus-software#346

Merged

tas50 mentioned this issue Jul 24, 2015

Error installing sensu-plugins-postgres sensu-plugins/sensu-plugins-postgres#1

Closed

lamont-granquist mentioned this issue Oct 23, 2015

Add additional native gems to omnibus-chef chef-boneyard/chef-rfc#162

Merged

aried3r mentioned this issue Nov 11, 2016

build-essential package is not installed using the ruby recipie #373

Closed

chris-rock pushed a commit to chris-rock/postgresql that referenced this issue Jan 4, 2017

Merge pull request sous-chefs#212 from bcs-de/configure_locale_envvars

2650e05

Allow to set LANG and LC_ALL in envvars

chris-rock pushed a commit to chris-rock/postgresql that referenced this issue Jan 4, 2017

Also adjust LANG for rhel systems sous-chefs#212

97bb175

chris-rock pushed a commit to chris-rock/postgresql that referenced this issue Jan 4, 2017

Update README and CHANGELOG sous-chefs#212

ad93bb1

chris-rock pushed a commit to chris-rock/postgresql that referenced this issue Jan 4, 2017

Adjust test to check for different contents sous-chefs#212

326f6ad

lock bot locked as resolved and limited conversation to collaborators Jul 24, 2018

pg gem fails to compile with chef 12.0 #212

pg gem fails to compile with chef 12.0 #212

Comments

jwmarshall commented Dec 5, 2014

hulu1522 commented Dec 6, 2014

jtimberman commented Dec 6, 2014

mshappe commented Dec 9, 2014

dpresling commented Dec 9, 2014

mshappe commented Dec 9, 2014

lamont-granquist commented Dec 9, 2014

mshappe commented Dec 9, 2014

jdstuart commented Dec 9, 2014

lamont-granquist commented Dec 9, 2014

mshappe commented Dec 9, 2014

lamont-granquist commented Dec 9, 2014

mshappe commented Dec 9, 2014

hulu1522 commented Dec 9, 2014

lamont-granquist commented Dec 9, 2014

hulu1522 commented Dec 9, 2014

lamont-granquist commented Dec 9, 2014

mshappe commented Dec 9, 2014

lamont-granquist commented Dec 9, 2014

lamont-granquist commented Dec 9, 2014

lamont-granquist commented Dec 9, 2014

mshappe commented Dec 9, 2014

mshappe commented Dec 10, 2014

lamont-granquist commented Dec 10, 2014

mshappe commented Dec 10, 2014

lamont-granquist commented Dec 10, 2014

scotthain commented Dec 10, 2014

hulu1522 commented Dec 10, 2014

mshappe commented Dec 11, 2014

nlopes commented Jul 14, 2015

patcon commented Jul 14, 2015

lazzarello commented Sep 2, 2015

vblessing commented Sep 22, 2015

epu commented Sep 30, 2015

jpeyret commented Oct 7, 2015

JackChance commented Oct 28, 2015

acqant commented Dec 4, 2015

qhartman commented Dec 4, 2015

acqant commented Dec 5, 2015

patcon commented Dec 5, 2015

acqant commented Dec 7, 2015

qhartman commented Dec 7, 2015

patcon commented Dec 7, 2015

acqant commented Dec 8, 2015

qhartman commented Dec 8, 2015

patcon commented Dec 8, 2015

acqant commented Feb 25, 2016

ashwinrayaprolu commented Aug 28, 2016

patcon commented Aug 29, 2016 • edited Loading

ashwinrayaprolu commented Aug 29, 2016

lock bot commented Jul 24, 2018

patcon commented Aug 29, 2016 •

edited

Loading