More comprehensive test coverage

[timholy]

Updated Feb 23, 2015.

This would make a whole bunch of good "first contribution" projects, and does not require deep insider knowledge of the language.

The basic idea is to expand the test suite to make sure that julia's base code works as promised. Here is one recommended way to contribute toward this goal:

The easy way

1. Go visit https://coveralls.io/r/JuliaLang/julia. (At the time of this writing, coverage numbers are in the 70s%. If you see numbers that are much lower than this, don't trust the reports; there is likely a problem in measuring coverage.)
2. Browse through the source files and find some untested functionality (highlighted in red) that you think you might be able to write a test for.
3. Write a test that exercises this functionality---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in test/runtests.jl. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see CONTRIBUTING.md).

The manual method

1. Make sure you have an up-to-date git checkout and are on the master branch. Build julia with make
2. Copy/paste the following script into an editor window and save it. Here I'll use the filename /tmp/coverage_tests.jl:

using Base.Test
testnames = ["core", "keywordargs", "numbers", "strings", "dates",
    "hashing", "remote", "iobuffer", "staged", "arrayops",
    "subarray", "reduce", "reducedim", "random", "intfuncs",
    "simdloop", "blas", "fft", "dsp", "sparse", "bitarray", "copy", "math",
    "functional", "bigint", "sorting", "statistics", "spawn",
    "backtrace", "priorityqueue", "arpack", "file", "suitesparse", "version",
    "pollfd", "mpfr", "broadcast", "complex", "socket",
    "floatapprox", "readdlm", "reflection", "regex", "float16", "combinatorics",
    "sysinfo", "rounding", "ranges", "mod2pi", "euler", "show",
    "lineedit", "replcompletions", "repl", "test", "goto",
    "llvmcall", "grisu", "nullable", "meta", "profile",
    "libgit2", "docs", "base64", "pkg", "linalg1", "linalg2",
    "linalg3", "linalg4", "linalg/lapack", "linalg/triangular", "linalg/tridiag", 
    "linalg/pinv", "linalg/cholmod", "linalg/umfpack", "linalg/givens"
]
for tst in testnames
    println(tst)
    include(joinpath(JULIA_HOME,Base.DATAROOTDIR,"julia","test","$tst.jl"))
end

(This is the list of tests currently in test/runtests.jl, with 3 omissions (resolve, reflection, and meta) and one addition (pkg). The omitted tests explicitly test inlining, which we're going to disable, or are problematic when inlining is disabled.)

1. From the top-level directory, type rm usr/lib/julia/sys.so. Deleting sys.so will prevent julia from using any pre-compiled functions, increasing the accuracy of the results. (This also makes startup a bit slower, but that's fine for this test---and once you're done, simply typing make will cause this file to be rebuilt).
2. Via the command prompt, navigate to julia's test/ directory.
3. Start julia with julia --code-coverage=all --inline=no. This turns on code-coverage and prevents inlining (inlining makes it difficult to accurately assess whether a function has been tested).
4. At the julia prompt, type include("/tmp/coverage_tests.jl").
5. Once the tests complete successfully, quit julia
6. Navigate to the base/ directory.
7. Browse through the *.jl.cov files, and look for functions (or branches within functions) that either have 0 in front of them (indicating that they were run 0 times), or have - in front of them (indicating that they were never compiled).
8. Write a test that exercises some function that is not adequately covered---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in test/runtests.jl. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see CONTRIBUTING.md).

[JeffBezanson]

💯

[sbromberger]

This sounds like a fantastic idea, but I think I'm missing something: I fire up julia* from test/ with
--code-coverage=all and follow the instructions (the tests all appear to run correctly; that is, I get SUCCESS at the end), but then when I go to base/, I don't see any .cov files.

*I've tried using /usr/local/bin/julia which was installed via make install, as well as the one in the source tree under usr/bin.

julia> VERSION
v"0.4.0-dev+2251"

[IainNZ]

I just did it, and it worked for me. Are you definitely using the julia you are building from source?

[sbromberger]

Pretty sure. See https://gist.github.com/sbromberger/282ed8d69695b34d4279 for the log.

ETA: I do see a runtests.jl.cov in test/:

-rw-r--r--  1 seth  staff  2675 Dec 30 01:51 runtests.jl.cov

[IainNZ]

Why are you doing ../usr/bin/julia, or is that symlinked to /Users/seth/dev/julia/julia/julia?

Anyway, I thought I'd try to give this a go, but I'm not getting sensible results from the coverage. For example, I got that intersect and setdiff weren't run - sure enough they're in the tests. I added some printlns just to be sure, and they do get run, but I see

        - function intersect(s::Set, sets::Set...)
        1     println(:intersect, s, sets)
        0     i = similar(s)
        0     for x in s
        0         inall = true
        0         for t in sets
        0             if !in(x,t)
        0                 inall = false
        0                 break
        -             end
        -         end
        0         inall && push!(i, x)
        -     end
        0     return i
        - end

which is even harder to explain ¯\_(ツ)_/¯.

[sbromberger]

Why are you doing ../usr/bin/julia, or is that symlinked to /Users/seth/dev/julia/julia/julia?

Because that's where the compiled julia (pre-make install) lives. There is no /Users/seth/dev/julia/julia/julia. It's

-rwxr-xr-x  1 seth  staff  20404 Dec 24 01:19 /Users/seth/dev/julia/julia/usr/bin/julia*

(On my system, $SRCROOT/julia was a broken symlink to $SRCROOT/usr/bin/julia-debug, which broke when I moved the directory. Even after fixing the broken symlink, I did not get the expected results.)

[sbromberger]

OK, mystery solved. Because I moved the source root after the build, I have broken symlinks all over the place. Specifically, $SRCROOT/usr/share/julia/base is a symlink that is not relative (that is, it points to a full path starting with /), which ties it to the filesystem at the time of the build. Since I moved the source root, that absolute symlink now points to an invalid directory, which (I guess) forces $SRCROOT/usr/bin/julia to try to find some other base/ (though I don't know where).

Why the symlinks within the build directory are absolute-pathed (at least on OSX) is a mystery to me. Far better, instead of having

lrwxr-xr-x  1 seth  staff  26 Dec 24 00:43 base@ -> /Users/seth/dev/julia/base

in $SRCROOT/usr/share/julia, we should have

lrwxr-xr-x  1 seth  staff  26 Dec 24 00:43 base@ -> ../../../base

so that the build directory is independent of the overall filesystem and the source root (/Users/seth/dev/julia) can be moved without problem.

[waldyrious]

@timholy, step 7 appears to be less detailed than the other ones. Perhaps it should at least link to http://julia.readthedocs.org/en/latest/stdlib/test/ for those new to unit testing. (The rest of the list is crystal clear, thanks for doing that!)

Additionally, CONTRIBUTING.md mentions "Make sure you test your code as described here", but the "here" is kinda ambiguous. Does it means the list that follows shortly afterwards, or was it meant to be a link, perhaps to the docs I pointed to above?

[timholy]

@IainNZ: wow. OK, I'll take a peek and see if I can figure out what's happening.

@waldyrious: very good suggestions. I've edited the post up top to include your link, and will look into editing CONTRIBUTING.md.

[timholy]

@IainNZ, the following test script works for me:

using Base.Test
s1 = Set(1,2,3)
s2 = Set(2,5,7)
s = intersect(s1, s2)
@test s == Set(2)

I get useful coverage results with or without #9354 and --inline=no:

        - function intersect(s::Set, sets::Set...)
        1     i = similar(s)
        1     for x in s
        3         inall = true
        3         for t in sets
        3             if !in(x,t)
        2                 inall = false
        2                 break
        -             end
        -         end
        3         inall && push!(i, x)
        -     end
        1     return i
        - end

Does that work for you? If not, what platform and commit are you on?

[svaksha]

Hi,
@timholy, I ran the tests for commit: 5219315, which threw this error:

julia> include("runtests.jl")
From worker 3: * linalg2
From worker 2: * linalg1
From worker 2: * linalg3
From worker 3: * linalg4
From worker 2: * linalg/lapack
From worker 2: * linalg/triangular
exception on 3: ERROR: test failed: transpose(qrfact(randn(3,3))) did not throw ErrorException
in expression: transpose(qrfact(randn(3,3)))
in error at error.jl:21
in default_handler at test.jl:27
in do_test_throws at test.jl:69
in runtests at /home/mom/julia/test/testdefs.jl:5
in anonymous at multi.jl:852
in run_work_thunk at multi.jl:603
in anonymous at task.jl:852
while loading linalg4.jl, in expression starting on line 205
ERROR: test failed: transpose(qrfact(randn(3,3))) did not throw ErrorException
in expression: transpose(qrfact(randn(3,3)))
in anonymous at task.jl:1616
while loading linalg4.jl, in expression starting on line 205
while loading /home/mom/julia/test/runtests.jl, in expression starting on line 42

I had another question regarding running these tests: Currently, for personal devel stuff I prefer to use the nightly julia build from Elliot/staticfloat for Ubuntu-14.04-LTS. But, for the test coverage, is it kosher to use and build julia in the /home/user/julia directory? I figured that since the nightlies run off a different space, this arrangement will give me a sandbox (not exactly, but then julia does not have an Anaconda / Canopy/ virtualenv env like python does) space. What are the drawbacks of doing this, if any? TIA.

[IainNZ]

@timholy everything seems to be working fine, not sure what was happening before. Woops!

[timholy]

@svaksha, thanks for giving this a spin. Does make test-linalg4 fail reliably for you?

is it kosher to use and build julia in the /home/user/julia directory?

I am not anything remotely resembling a build guru, so don't listen to me 😄. I'm not even sure where @staticfloat's binary installs. But, given those caveats: it should be OK to run code_coverage even from a binary install. However, when you add to files in test/, it's better to use a git checkout, just to make it easier to contribute. I don't see any problem with having two julia installs---I have one 0.4 and one 0.3 on my machine, and except for history and some package symlinks I've set up deliberately, the two don't interact at all.

[timholy]

@IainNZ, very glad to hear that! It is weird that it was giving you something so strange; if you can replicate, please do file an issue.

[staticfloat]

@svaksha I routinely keep a "stable" version installed via package manager, (whether it be Homebrew on OSX, or aptitude on Ubuntu) and an "unstable" version just sitting in ~/src/julia. Absolute paths shouldn't make a difference here, just be sure to invoke the proper julia executable when playing around in the unstable julia directory. (I can't tell you how many times I've scratched my head before realizing that I needed to ./julia to use the unstable version, rather than just julia to use the stable version)

[svaksha]

On Wed, Dec 31, 2014 at 12:56 AM, Tim Holy notifications@github.com wrote:

@svaksha, thanks for giving this a spin. Does make test-linalg4 fail reliably for you?

Hi and Thanks for the reply @timholy! If "fail reliably" means it ran
successfully, then it did (with a small linker warning) :
https://gist.github.com/svaksha/aa25f53bf540e3fd52d6

is it kosher to use and build julia in the /home/user/julia directory?

I am not anything remotely resembling a build guru, so don't listen to me . I'm not even sure where @staticfloat's binary installs.

Hah, same here - my ppa install gets called via
/etc/apt/sources.list.d is the limit of my interest in system level
internals, unless an OS issue crops up breaking stuff that I have to
fix so that I can get back to work :-)

However, when you add to files in test/, it's better to use a git checkout, just to make it easier to contribute.

Good point - will keep this in mind.

I don't see any problem with having two julia installs---I have one 0.4 and one 0.3 on my machine, and except for history and some package symlinks I've set up deliberately, the two don't interact at all.

True, but I'm a huge fan of sandboxed testing/development which makes
it easier to catch the bug source. </thinking out loud>.

PS: @timholy, Would you mind sharing (in a blog post or a gist) how
you installed, maintain and update the two julia installs? I fail at
this for code that was written for older julia versions.

Thanks, SVAKSHA ॥ http://about.me/svaksha ॥

[svaksha]

Hi and Thanks for the reply @staticfloat.

On Wed, Dec 31, 2014 at 1:00 AM, Elliot Saba notifications@github.com wrote:

@svaksha I routinely keep a "stable" version installed via package manager, (whether it be Homebrew on OSX, or aptitude on Ubuntu) and an "unstable" version just sitting in ~/src/julia.

Got that. What isnt clear is how (if) the bash modifications affect
the 'unstable' version on Ubuntu-LTS.

Absolute paths shouldn't make a difference here, just be sure to invoke the proper julia executable when playing around in the
unstable julia directory. (I can't tell you how many times I've scratched my head before realizing that I needed to ./julia to use the
unstable version, rather than just julia to use the stable version)

Could you elaborate a little more on what you mean by needing "./julia
to use the unstable version, rather than just julia to use the stable
version"? I'm not sure I understand this entirely. This may not be
entirely relevant to this testing thread so feel free to reply in a
gist/email.

On an unrelated note, I recall seeing a thread about running Julia
within Anaconda - are you involved with that or do you happen to know
its progress?

@staticfloat, Also wanted to thank you for the nightlies - they make
life so simple that I've forgotten the harsh world of compile failures
:-). Just last week I was caught trying to figure out why the latest
git co was failing and yet showing a 3-day old version on the REPL.

Thanks, SVAKSHA ॥ http://about.me/svaksha ॥

[tkelman]

On an unrelated note, I recall seeing a thread about running Julia
within Anaconda - are you involved with that or do you happen to know
its progress?

I don't know that anyone has specifically tried this. It's been mentioned a few times, such as #4853 (comment) or a few times in relation to making the installation of IJulia easier, but I don't know if anyone has done any work in this direction. Maybe some people more familiar with the SciPy packaging ecosystem would know if it's happened yet from that side. I did once try using their BinStar binary packaging/building service, but it was (is?) very young and not too well documented and I couldn't figure it out.

[staticfloat]

What I mean is this: My setup is to have a stable version in /usr/local/bin (that's my Homebrew installation of OSX, using the Homebrew tap) or /usr/bin (that's the apt installation on Ubuntu using the stable PPA). Those directories are both listed in the default PATH environment variable, so if I type julia at the command prompt, it will look in the folders that are on my PATH, and when it finds a julia executable, it will run that. However, if I type ./julia, it only will execute a file called julia in the current directory. (I'm providing an unambiguous path to a file in this case; . represents the current directory, so ./julia means "the file named julia in the current directory")

My setup is most useful if you do most of your work in a stable version of julia, but want to try something out on master every now and then. Running julia anywhere will always start the stable version of Julia, but running, for instance, ~/src/julia/julia will run the Julia executable sitting in ~/src/julia, which happens to be the master checkout. (Alternatively, if you're already sitting in ~/src/julia, you can just run ./julia)

Does that make more sense?

On an unrelated note, I recall seeing a thread about running Julia
within Anaconda - are you involved with that or do you happen to know
its progress?

I haven't heard anything about that recently, so if people are working on that right now, our paths haven't crossed.

I'm a huge fan of sandboxed testing/development which makes
it easier to catch the bug source.

I agree. Julia (and I'm only talking about things in this git repository; packages are another story) doesn't touch anything outside of ~/julia/X.Y, where X.Y is the major/minor version of julia. Two versions of Julia can coeexist very peacefully, and if you'd like to have control over where your Julia stores its package state, you can do so via the JULIA_PKGDIR environment variable. This allows for a (somewhat labor-intensive) method of pretending like we have a virtualenv for Julia; you can put the Julia executable, and all its packages in a directory, and then just call Julia with the correct environment variables setup to have your own little packaged, isolated Julia installation.

Packages can do whatever they want, however. For instance, every time I run Pkg.build() on 0.3, the IJulia.jl package sets up an iPython notebook profile for IJulia, which gets overwritten every time I run Pkg.build() on 0.4. So those two fight every now and then, but that's something that is more of a problem in IPython (and which they are addressing in IPython 3.0 I believe) than it is a problem in Julia.

[svaksha]

On Wed, Dec 31, 2014 at 5:56 AM, Elliot Saba notifications@github.com wrote:

My setup is most useful if you do most of your work in a stable version of julia, but want to try something out on master every now and then. Running julia anywhere will always start the stable version of Julia, but running, for instance, ~/src/julia/julia will run the Julia executable sitting in ~/src/julia, which happens to be the master checkout. (Alternatively, if you're already sitting in ~/src/julia, you can just run ./julia)

Does that make more sense?

Thanks for explaining that. Its what I'd like to (and was trying to)
do but for some reason, its not working as I imagined it would. In
/home/user/julia, when I try Tim's instructions it complains about
being unable to find the directory:

~/julia/test$ ./julia --code-coverage=all
bash: ./julia: No such file or directory

If I run the above command without the ./ it runs off your
nightlies, which isnt what I want. I'd like to keep the packaged
nightlies disjunct from the git master, the latter being the sandbox
to break stuff.

I agree. Julia (and I'm only talking about things in this git repository; packages are another story) doesn't touch anything outside of ~/julia/X.Y, where X.Y is the major/minor version of julia. Two versions of Julia can coeexist very peacefully, and if you'd like to have control over where your Julia stores its package state, you can do so via the JULIA_PKGDIR environment variable. This allows for a (somewhat labor-intensive) method of pretending like we have a virtualenv for Julia; you can put the Julia executable, and all its packages in a directory, and then just call Julia with the correct environment variables setup to have your own little packaged, isolated Julia installation.

Is there any documentation on how one can setup such an isolated environment?
I grepped http://docs.julialang.org/en/latest/ for JULIA_PKGDIR but
it does not explain how to setup this env.

SVAKSHA ॥ http://about.me/svaksha ॥

[svaksha]

On Wed, Dec 31, 2014 at 5:38 AM, Tony Kelman notifications@github.com wrote:

I don't know that anyone has specifically tried this. It's been mentioned a few times, such as #4853 (comment) or a few times in relation to making the installation of IJulia easier, but I don't know if anyone has done any work in this direction. Maybe some people more familiar with the SciPy packaging ecosystem would know if it's happened yet from that side. I did once try using their BinStar binary packaging/building service, but it was (is?) very young and not too well documented and I couldn't figure it out.

Thanks for the reply. Found the SO thread:
http://stackoverflow.com/questions/25373577/how-to-install-julia-in-an-anaconda-environment
I prefer the Anaconda env's more (than say, virtualenv) as it acts
like an OS inside an OS (for lack of a better term) for Python, making
it easier to sanitize the development work. As for pure julia
development, the current method of running off other directories with
a path defined works well but while developing for a mixed codebase it
would be nicer if Julia would run off the Anaconda env too.

SVAKSHA ॥ http://about.me/svaksha ॥

[timholy]

@svaksha, from inside test/ just say ../julia --code-coverage=all. Note there are two dots in front of julia, so you go up a directory.

What I do is have julia refer to master, and julia-old be a symlink (on my PATH) to a v0.3 checkout. Most users will want to flip that, and make julia be for stable and julia-next for master. As long as both are on your PATH, you're pretty much done.

[timholy]

@svaksha, I remembered your linalg4 issue; that gist you posted showed success, so clearly it's not "failing reliably." You might want to try a second time running the whole test suite as described above.

If you experience the same transpose(qrfact(... error again, it would be great to open a separate issue.

[timholy]

OK, I've substantially edited the instructions above; with these changes, I think we get reasonably-accurate results, as long as a human parses the *.jl.cov files.

However, I'm running into #9536, so (for me) currently this doesn't work.

[kshyatt]

I would personally prefer a green badge, but I'm happy to defer to others. There are areas (the REPL, especially) that have overall poor test coverage so maybe we could open individual issues for those?

[timholy]

I think this is one of those issues that's counterproductive to close until we hit 100%. After all, it is a great entry point for new developers, and we don't have that many good examples of that kind of issue.

I'll be super-excited to see that color change to yellow!

[ivarne]

I think this is one of those issues that's counterproductive to close until we hit 100%

I agree, but when the number of comments becomes becomes high, it's probably time to archive the discussion and continue the work in a new issue with an updated summary on top.

[ViralBShah]

I agree. For other areas, it would be nice to have specific issues to direct people to write tests for those, and close the umbrella issue.

[kshyatt]

I'd like to have a few separate issues like "Improve REPL tests" or "Flesh out LAPACK tests" and keep the information in the OP here as a Gist/README somewhere. It's probably pretty intimidating for some people to go to the "more tests" issue and see 150+ comments.
E: sorry if this is a dumb question - why don't we have the Coveralls badge on the README?

[timholy]

If people prefer that, it's all fine by me.

[ViralBShah]

@kshyatt Great idea to add the Coveralls badge to the README. Go for it!

[kshyatt]

It's me being a pain again - Coveralls hasn't run in 4 days. Are the buildbots still down?

[staticfloat]

Thank you for the ping, I've restarted the latest builds after clearing out the build directory following multiple failures in a row over the last few days. We'll see if things shake themselves out.

[staticfloat]

Things did not shake themselves out, I had to rebuild the CentOS 5 buildbots. The good news is, that's almost completely automated now. We've got a new coverage build up, but alas, still no git info making up there.

@kshyatt I seem to be having difficulty communicating git commit information to Coveralls. My mini script for submitting coverage results is:

r1 = load("coverage_noninlined.jld", "results")
r2 = load("coverage_inlined.jld", "results")
r = CoverageBase.merge_coverage(r1, r2)
git_info = @compat Dict(
    "branch" => Base.GIT_VERSION_INFO.branch,
    "remotes" => [
        @compat Dict(
            "name" => "origin",
            "url" => "https://github.com/JuliaLang/julia.git"
        )
    ],
    "head" => @compat Dict(
        "id" => Base.GIT_VERSION_INFO.commit,
        "message" => "Merge pull request #11390 from ScottPJones/spj/deputf32"
    )
)
println("git_info: ")
println(git_info)
Coveralls.submit_token(r, git_info)

Which results in: (You can also look at the actual execution log)

WARNING: char(x) is deprecated, use Char(x) instead.
 in depwarn at deprecated.jl:44
 in char at deprecated.jl:31
 in include at boot.jl:252
 in include_from_node1 at loading.jl:134
 in include at boot.jl:252
 in include_from_node1 at loading.jl:134
 in reload_path at loading.jl:158
 in _require at loading.jl:70
 in require at loading.jl:56
 in include at boot.jl:252
 in include_from_node1 at loading.jl:134
 in reload_path at loading.jl:158
 in _require at loading.jl:70
 in require at loading.jl:53
 in process_options at client.jl:288
 in _start at client.jl:409
while loading /home/ubuntu/.julia/v0.4/JuliaParser/src/lexer.jl, in expression starting on line 12
Warning: could not import Base.put into Requests
Coverage.MallocInfo[]
git_info: 
Dict{ASCIIString,Any}("branch"=>"master","head"=>Dict("message"=>"Merge pull request #11390 from ScottPJones/spj/deputf32","id"=>"04051e96993cb53c8556691ee40e49bd054c7a5b"),"remotes"=>[Dict("name"=>"origin","url"=>"https://github.com/JuliaLang/julia.git")])
WARNING: [a,b,...] concatenation is deprecated; use [a;b;...] instead
 in depwarn at deprecated.jl:44
 in oldstyle_vcat_warning at abstractarray.jl:28
 in vect at abstractarray.jl:37
 in render at /home/ubuntu/.julia/v0.4/Requests/src/Requests.jl:23
 in open_stream at /home/ubuntu/.julia/v0.4/Requests/src/Requests.jl:224
 in send_multipart at /home/ubuntu/.julia/v0.4/Requests/src/Requests.jl:519
 in do_request at /home/ubuntu/.julia/v0.4/Requests/src/Requests.jl:566
 in post at /home/ubuntu/.julia/v0.4/Requests/src/Requests.jl:583
 in submit_token at /home/ubuntu/.julia/v0.4/Coverage/src/Coverage.jl:236
 in process_options at client.jl:288
 in _start at client.jl:409
while loading no file, in expression starting on line 0
WARNING: int(x) is deprecated, use Int(x) instead.
 in depwarn at deprecated.jl:44
 in int at deprecated.jl:31
 in on_header_value at /home/ubuntu/.julia/v0.4/Requests/src/Requests.jl:122
while loading no file, in expression starting on line 0
ASCIIString "{\"message\":\"Job #15453.1\",\"url\":\"https://coveralls.io/jobs/6174181\"}"

Are you doing anything differently when you run things locally in order to generate the git information? What is the payload you submit to Coveralls in order to get it to accept the git information?

[kshyatt]

I'm just using CoverageBase.jl. It's whatever method that package uses, which is presumably the same thing that's running on the buildbots! I've run coverage on julia.mit.edu and had it work - something might be weird about the buildbot?

[staticfloat]

Can you show me the results of your coverage run?

[ViralBShah]

I think we should close this issue - and have new issues for some of these discussions.

[samuelpowell]

A number of more general purpose parts of base are covered only by virtue of their use in other tests in the suite.

Is it fair to say that each module should be tested fully by its own test script, such that the modification of unrelated tests does not have the unintended consequence of leaving untested code?

[timholy]

I find it helpful to have tests that fail for obvious reasons rather than for indirect reasons. For example, if there's a problem with type inference, it's much nicer if it fails a direct test in core.jl than having it caught because some call dispatched to the wrong method. The second is much harder to deduce at the level of what went wrong and what needs changing.

[ScottPJones]

👍 @samuelpowell @timholy Definitely saves a lot of time to have pinpoint tests.

[samuelpowell]

Excellent, I'll see what I can do.

[tkelman]

Let's close this in favor of #11885 and continuing targeted pushes. The instructions in the top post are still valuable, but the nearly-200 comments discussion is not so relevant going forward and takes a while to load.