Can't find libsybdb.dylib on mac

[Paxa]

I downloaded package pgloader-3.2.0.pkg installed and try to run.

% pgloader --version

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {1006E666E3}>:
  Error opening shared object "libsybdb.dylib":
  dlopen(libsybdb.dylib, 10): image not found.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE       ] Skip this shared object and continue.
  1: [RETRY          ] Retry loading this shared object.
  2: [CHANGE-PATHNAME] Specify a different pathname to load the shared object from.

(SB-SYS:DLOPEN-OR-LOSE #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"libsybdb.dylib" :NAMESTRING "libsybdb.dylib" :HANDLE NIL :DONT-SAVE NIL))
0] 

Then if I try to migrate data from mysql to postgresq, I get this:

2015-01-28T06:48:07.627000+01:00 ERROR Database error 42601: syntax error at or near "3"
QUERY: CREATE TABLE merchants 
(
  id                     text not null,
  merchant_name          text,
  email                  text,
  phone                  text,
  url                    text,
  3d_secure_mpi_type     text,
  ....
);

Seems like postgres don't like column name starting with number

But it imports data correctly

[robins]

Well you can be sure that PostgreSQL accepts only column-names, starting with either an alphabet or an underscore (_) ... (More detail here) ... although I'd let @dimitri respond to what PgLoader does in such scenarios.

[dimitri]

The first bug has been fixed meanwhile, so please consider building from sources to see the bug disappear. If you're already using brew, building pgloader is quite easy and described in the docs, see INSTALL.md file.

Then the MySQL migration is another error entirely, you can work around it already by adding the option WITH quote identifiers so that case will be respected in the migration and column names will be protected with double-quotes. Working on a fix now so that it just work for that case.

[Paxa]

Thanks

[dimitri]

If you have the opportunity to try with the fix and with the option downcase identifiers (which is the default) so that you could then confirm the fix, it would be awesome. Thanks.

[Paxa]

When I compile from source:

  Unable to load foreign library (SYBDB).
  Error opening shared object "libsybdb.dylib":
  dlopen(libsybdb.dylib, 10): image not found.

press continue then:

build/bin/buildapp.sbcl      --logfile /tmp/build.log                \
                         --require sb-posix --require sb-bsd-sockets --require sb-rotate-byte                        \
                         --sbcl sbcl                            \
                         --asdf-path .                           \
                         --asdf-tree build/quicklisp/local-projects     \
                         --manifest-file build/manifest.ql             \
                         --asdf-tree build/quicklisp/dists              \
                         --asdf-path .                           \
                         --load-system pgloader               \
                         --load src/hooks.lisp                   \
                         --entry pgloader:main                   \
                         --dynamic-space-size 4096         \
                         --compress-core                    \
                         --output build/bin/pgloader.tmp
;; loading system "pgloader"
Fatal MISSING-DEPENDENCY:
  Component #:UUID not found, required by #<SYSTEM "pgloader">
make: *** [build/bin/pgloader] Error 1

[dimitri]

Ah, yes sorry, to compile from source yes you need to install freetds, brew install freetds.

[Paxa]

pgloader working without libsybdb.dylib well (I remove freetds after compilation)
table with column 3d_secure_mpi_type ok
Thnks

[dimitri]

Perfect, thanks for your feedback!

[Themanwithoutaplan]

I'm getting this error even with FreeTDS installed.

I'm using MacOS with MacPorts so the library is at /opt/local/lib/libsybdb.dylib

[dimitri]

Can you try to set your lib in the library path for MacOS that way:

DYLD_FORCE_FLAT_NAMESPACE=1 DYLD_INSERT_LIBRARIES=/path/to/your.dyld command

Note that I'm using a mac with brew myself and didn't need anything like that. Of course it might just be that brew is doing it for me (by just installing files in /usr/local/lib):

/usr/local/lib/libsybdb.5.dylib
/usr/local/lib/libsybdb.a
/usr/local/lib/libsybdb.dylib

[Themanwithoutaplan]

There are good reasons for using MacPorts over Brew. But each to their own. I normally don't have a problem compiling anything as a result of installing MacPorts.

What is the name of the dyld command I should be looking for? I only seem to have /usr/lib/system/libdyld.dylib but setting this does not help.

[dimitri]

Try:

DYLD_FORCE_FLAT_NAMESPACE=1 DYLD_INSERT_LIBRARIES=/opt/local/lib/libsybdb.dylib make

[Themanwithoutaplan]

Sorry, this doesn't solve the problem. What other components have you installed with Brew that might be relevant?

[dimitri]

Can you post the entire error message you get so that I can investigate better?

[Themanwithoutaplan]

Sure.

44,210 bytes in 0.19 seconds (230.88KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-mssql/2013-10-03/cl-mssql-20131003-git.tgz">
; 15.04KB
==================================================
15,398 bytes in 0.00 seconds (0.00KB/sec)
; Loading "mssql"
[package garbage-pools]...........................
[package mssql].
debugger invoked on a LOAD-FOREIGN-LIBRARY-ERROR in thread
#<THREAD "main thread" RUNNING {1002E06D43}>:
  Unable to load foreign library (SYBDB).
  Error opening shared object "libsybdb.dylib":
  dlopen(libsybdb.dylib, 10): image not found.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Try loading the foreign library again.
  1: [USE-VALUE                    ] Use another library instead.
  2: [SKIP                         ] Skip loading foreign library tds.
  3: [TRY-RECOMPILING              ] Recompile mssql and try loading it again
  4: [RETRY                        ] Retry
                                     loading FASL for #<CL-SOURCE-FILE "mssql" "src" "mssql">.
  5: [ACCEPT                       ] Continue, treating
                                     loading FASL for #<CL-SOURCE-FILE "mssql" "src" "mssql">
                                     as having been successful.
  6:                                 Retry ASDF operation.
  7: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  8: [ABORT                        ] Give up on "mssql"
  9:                                 Give up on "pgloader"
 10: [CONTINUE                     ] Ignore runtime option --eval "(ql:quickload \"pgloader\")".
 11:                                 Skip rest of --eval and --load options.
 12:                                 Skip to toplevel READ/EVAL/PRINT loop.
 13: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(CFFI::FL-ERROR "Unable to load foreign library (~A).~%  ~A" SYBDB "Error opening shared object \"libsybdb.dylib\":
  dlopen(libsybdb.dylib, 10): image not found.")
0] 13
;
; compilation unit aborted
;   caught 2 fatal ERROR conditions
make: *** [build/libs.stamp] Error 1

[dimitri]

I'm short of ideas now, I'm not versed into the details of the dynamic linker enough to help you further :(

[Themanwithoutaplan]

Me, neither. :-( It is possible to use pgloader by passing the path manually every time it's run. I tried this trick when compiling but it didn't work. Seems to be a difference between "evaluated" and "unevaluated": The value /OPT/LOCAL/LIB/LIBSYBDB.DYLIB is not of type LIST.

[psoo]

Have you tried DYLD_LIBRARY_PATH=/opt/local/lib make? That worked for me with macports.

[Themanwithoutaplan]

No, that works. Still have to set COMPRES_CORE=no and I get an error if I try and build the package.

fuchsia:pgloader charlieclark$ DYLD_FORCE_FLAT_NAMESPACE=1 make COMPRESS_CORE=no pkg
# intended for use on a MacOSX system
mkdir -p /tmp/pgloader && rm -rf /tmp/pgloader/*
mkdir -p /tmp/pgloader/usr/local/bin/
mkdir -p /tmp/pgloader/usr/local/share/man/man1/
cp ./pgloader.1 /tmp/pgloader/usr/local/share/man/man1/
cp ./build/bin/pgloader /tmp/pgloader/usr/local/bin/
pkgbuild --identifier org.tapoueh.pgloader \
             --root /tmp/pgloader            \
             --version 3.2.1.preview              \
                 ./build/pgloader-3.2.1.preview.pkg
dyld: Symbol not found: _iconv
  Referenced from: /usr/lib/libcups.2.dylib
  Expected in: flat namespace
 in /usr/lib/libcups.2.dylib
make: *** [pkg] Trace/BPT trap: 5

[psoo]

Ugh, you're right, i have a similar problem, though git errors out with the _iconv symbol lookup error. This worked a while ago, so not sure what changed....after playing around i got this working:

$ PATH=/opt/local/bin:/usr/bin:/bin DYLD_LIBRARY_PATH=/opt/local/lib make
[...]
mkdir -p /tmp/pgloader && rm -rf /tmp/pgloader/*
mkdir -p /tmp/pgloader/usr/local/bin/
mkdir -p /tmp/pgloader/usr/local/share/man/man1/
cp ./pgloader.1 /tmp/pgloader/usr/local/share/man/man1/
cp ./build/bin/pgloader /tmp/pgloader/usr/local/bin/
pkgbuild --identifier org.tapoueh.pgloader \
             --root /tmp/pgloader            \
             --version 3.2.1.preview              \
                 ./build/pgloader-3.2.1.preview.pkg
pkgbuild: Inferring bundle components from contents of /tmp/pgloader
pkgbuild: Wrote package to ./build/pgloader-3.2.1.preview.pkg

which at least compiles pgloader without errors on my OSX installation. It looks like the problem is in mixing binaries from OSX and macports with DYLD_LIBRARY_PATH in this specific case, but to get the real root cause someone would have to dig deeper.

[Themanwithoutaplan]

Having an install target to copy to /usr/local/lib would solve that conflict.

[Themanwithoutaplan]

@Enalmada pgloader is already known to work on systems that use brew but that's because the setup that Dmitri has. Fixing this properly unfortunately requires detailed knowledge of some Mac internals. :-/

[johnzimm]

Just following up here. I ran into the same issue today compiling using macports (haven't used brew on this system). I was able to get a clean build without having to disable compression after symlinking /usr/local/lib to /opt/local/lib. Note that I did not have anything installed to /usr/local/lib so I didn't lose anything here.

[dimitri]

Thanks for providing a solution here @johnzimm ! Meanwhile it's also possible to use the pgloader docker image, either from the Dockerfile in the source tree (just docker build -t pgloader .) or from the auto build at https://hub.docker.com/r/dimitri/pgloader/.

[Themanwithoutaplan]

Returning to this after a couple of years: seems that FreeTDS is an impicit dependency. The following worked for me:

sudo port install freetds sbcl
sudo ln -s /opt/local/lib/libsybdb.dylib /usr/local/lib

Then cd into the checkout, make and sudo mv buid/bin/pgloader /usr/local/bin (or /opt)
Should be possible to get this working in a portfile. Looks like the makefile only needs a single path change.

[martinstreicher]

On Mojave, I did:

$ brew install sbcl freetds 
$ git clone git@github.com:dimitri/pgloader.git --branch v3.5.1 --depth=1
$ cd pgloader
$ make pgloader

As a reference, this is the output of brew list prior to running make.

adns		gettext		heroku-node	libgpg-error	libusb		openssl		postgresql
autoconf	git		icu4c		libksba		libyaml		openssl@1.1	readline
automake	gmp		isl		libmpc		mpfr		p11-kit		redis
coreutils	gnupg		libassuan	libtasn1	mysql		pcre2		sbcl
freetds		gnutls		libffi		libtool		nettle		pinentry	unixodbc
gcc@6		heroku		libgcrypt	libunistring	npth		pkg-config

[Themanwithoutaplan]

Just trying again on MacOS and get the error below. I guess this is related to BerkleyDB or something similar.

; Loading "pgloader"
.;
; caught ERROR:
; READ error during COMPILE-FILE:
;
; Symbol "CLOSE-MEMO" not found in the DB3 package.
;
; Line: 26, Column: 17, File-Position: 816
;
; Stream: #<SB-INT:FORM-TRACKING-STREAM for "file /Users/charlieclark/temp/pgloader/src/sources/db3/db3-schema.lisp" {1004A67D83}>

debugger invoked on a UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread
#<THREAD "main thread" RUNNING {10005C85B3}>:
COMPILE-FILE-ERROR while
compiling #<CL-SOURCE-FILE "pgloader" "src" "sources" "db3" "db3-schema">

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry
                                     compiling #<CL-SOURCE-FILE "pgloader" "src" "sources" "db3" "db3-schema">.
  1: [ACCEPT                       ] Continue, treating
                                     compiling #<CL-SOURCE-FILE "pgloader" "src" "sources" "db3" "db3-schema">
                                     as having been successful.
  2:                                 Retry ASDF operation.
  3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  4:                                 Retry ASDF operation.
  5:                                 Retry ASDF operation after resetting the
                                     configuration.
  6: [ABORT                        ] Give up on "pgloader"
  7: [CONTINUE                     ] Ignore runtime option --eval "(ql:quickload \"pgloader\")".
  8:                                 Skip rest of --eval and --load options.
  9:                                 Skip to toplevel READ/EVAL/PRINT loop.
 10: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(UIOP/LISP-BUILD:CHECK-LISP-COMPILE-RESULTS NIL T T "~/asdf-action::format-action/" ((#<ASDF/LISP-ACTION:COMPILE-OP > . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "pgloader" "src" "sources" "db3" "db3-schema">)))
   error finding frame source: Bogus form-number: the source file has probably

Just seen that this error has been reported in #940 , will post any follow up there.

[dimitri]

I think you need to make clean and then make again.