Test failures on FreeBSD 9.1 #3360
Comments
|
are there other failures? |
|
I installed from the rc1 tarball, so I don't have test_fast.sh handy. Give me a moment to bring up a dev't environment and I'll get back to you about test_fast. In the mean time, here's the complete output from nose. I'm looking into the GFlags warnings now. I'm also working on a minimal failing test for But...
|
|
OK, when I I get the warnings, the failure in I don't know what to do next, but will gladly take direction and respond to requests. |
|
Please pull the latest master (I pushed some changes today), and post the output of also, what is the value of edit: Tha test name above is now fixed in master. |
And furthermore... |
|
Your terminal doesn't support utf-8/unicode, or at least doesn't report it in a way pandas Obviously, the test needs to fail more gracefuly in such a situation, but really it's signalling If you're only working with ASCII data, your should be fine in principle btw. |
|
But there's something else going on here. That should have been a I expected to see: x is a unicode string, which gets implicitly encoded into ASCII when decoded, Can you break in with pdb and print out x at the point of failure? also, what are the values of the $LC_ALL and $LANG envars? |
Neither $LC_ALL nor $LANG are defined in the environment. This has been vexing me but I just breathed a sigh of relief since you just pointed out the As for the UTF-8 support, I'll see about turning on support in my terminal so that I can run tests in a richer environment. I'm the pandas maintainer for FreeBSD so it's not just my data I need to worry about. |
|
I'm pretty sure the code that's misbehaving (although the wrong exception is puzzling) I can't repro any of this on my box, but if you figure out why the DecodeError is happening |
|
was the pickle issue a false alarm? |
|
Oops. Not sure about the pickle test. Will see if I can reproduce it tonight. |
|
fyi....do a clean git clone and try.....the error message on there, can't find multiarray is a numpy module (so also make sure that it is loading correct numpy), etc. |
|
Relavent perhaps: might you be using a 32bit compiled numpy on an amd64 system? |
|
OK, here's the reproducibility pattern I'm seeing for the test_legacy_pickle failure.
Next I'm going to try installing from git without assistance from the FreeBSD ports machinery and/or point FreeBSD ports at the repo instead of the tarball. Depending on the result I will probably suspect Cython or FreeBSD ports, but hopefully not both. As for word-length, I see no possibility that pandas and numpy are mismatched on this machine. |
|
so the py-pandas port is the FreeBSD packing mechanism? looks like it installs 0.10.1? if you install 0.10.1 from the pypi tarball does it work for you? |
|
@jreback: Yes, "port" in this context is like the .spec file and associated cruft to generate an RPM. I published a working draft for 0.11.0rc1 for testing to a mailing list. After 0.11.0 is released, I will revise the patch and submit it as a pandas maintainer update. All versions published to FreeBSD ports so far have had all tests passing, including 0.10.1. |
It's long past my bedtime, but when I get a chance tomorrow I will take a look at the diff between the tarball-provided .C files and my cython-generated .C files. That is, unless there is something else I should do first. At this point I think I have a fallback plan. In the event that we cannot determine and resolve the root cause of this issue prior to release, I could convert the port to build from github. I would rather not do that, though, because it adds a build-time dependency (cython), and increases the number of things that could vary across user build environments. |
|
what's your cython version? |
|
Cython 0.17.1. See above for other versions. |
|
any new leads on the unexplained |
|
@y-p: Sorry, but no. I've only had about an hour a day to work on this so far. I should have a good chunk of time this evening, so I hope to be able to report more substantial progress soon. |
|
I don't see this blocking 0.11, all the evidence points at the issue being somewhere along |
|
@y-p: Vagrant is a good idea, but it's not working out in practice. Using xironix/freebsd-vagrant, I got to |
|
@y-p: For the When the test passes, and the Next up: I need to find out what else could cause python to raise a different exception. |
|
I think I figured out the problem, the encoding detection routine was too rigid. That also possibly solves the failed unicode support detection on your |
|
@neirbowj , was the fix effective for you? would be good to clear this up prior to 0.11.0 |
|
@y-p: Sorry for the delay. No, 729d333, when applied in situ to my tarball build does not change the exception behaviour inside of This is what is currently bending my brain. In addition to applying 729d333, I have instrumented When When What I think this shows is that some internal state that is non-obvious to me is affecting the behaviour of the |
|
There's the you need to check the value of it's |
|
FreeBSD is off the hook for meddling with line endings. If you recall, |
|
Options for
|
|
On MPL issues. Changing sys.setdefaultencoding() back is a no go. might break GTK backend, and general Try #3409. I think it solves the problem. That doesn't solve the issue of MPL changing sys.getdefaultencoding, but the contract is that The rest of pandas shouldn't depend on getdefaultencoding() directly, so the end result |
|
#3409 does it for me. No more test failures. |
|
Good, merged. Now, about the pickle issues. @jreback what do you say? |
|
this code was not changed at all in 0.11, maybe the filename ha the embedded /r when it was generated? |
|
@neirbowj , when you says "remember that it passes when I built from GH", do you mean that you |
|
Ok, Github repo clone: tarball from (windows?) build box: NB the changed file size(s) of the pickle file The idea that windows would mangle binary files for line termination just cc @changhiskhan |
|
possibly related |
|
@y-p: When I talk about building from a tarball, I'm referring to the published sdist. When I refer to building from GH, FreeBSD fetches a GH-produced archive, and thereafter treats it like a regular sdist (i.e. extract into a working directory, configure, build, install). The GH method does not actually use git locally, but it can be configured to ask GH for a tarball from an arbitrary ref. |
|
Ok. looks like There's something wonky with the build box, don't think it's a pandas issue per se. |
|
I downloaded the tarball from http://pandas.pydata.org/pandas-build/dev/ |
|
those file sizes match what I get from the tarball as well, note also |
|
You should be able to replicate the test failure on any system just by import pickle
obj = pickle.load(open('multiindex_v1.pickle', 'r'))If your pickle has I'm not sure it really matters for anything other than this v0 pickle file, but wouldn't it be best to follow the common practice of shipping source code archives that match the likely convention of their intended platforms (.zip:CRLF, .tar.gz:LF)? If so, there should be a test. |
|
The contents of binary files should not change to match the platform you're on. |
|
Absolutely not, but a significant majority of what's in a source code archive is not binary. So there are two issues:
The former is the best way to resolve this issue. If you agree with the latter, I will open a new issue targeted to 0.12. |
|
FYI I am not sure when/how the multindex.pickle was generated (it is possible it was written not in binary mode) we have in place pickle compat tests going forward |
|
@y-p oh I see. I think you flipped the file sizes in your original comparison? What I got for the sizes on linux is what you posted under windows and vice versa. |
|
at the very least we should switch to creating the tarball/zip on linux instead. |
|
I sure did. sorry. doesn't cure it? |
|
Sorry for the trouble I caused by building the source distros on Windows, had been all linux til now =) Chang is uploading new rc1 tarballs and I'm going to work on cutting the 0.11 final now |
|
@changhiskhan You've probably noticed that I have what some might call an unhealthy tolerance for pain. I appreciate you and @y-p hanging in and resolving these test failures. @wesm No trouble. Just another opportunity for me to learn something about something, and a new corner case that might admit a new test or two. Congrats on the latest release. All tests passing (skipped 115) on FreeBSD 9.1-STABLE (r248078), with: Off to update the port for 0.11.0 release. |
|
hooray! thank @y-p for the fix. edit: yp -> y-p |
|
no damn it. thank me. |
|
My work here is done (for now). Good night. |
ghost
mentioned this issue
The text was updated successfully, but these errors were encountered: