enable slim on windows

[petrelharp]

Hopefully it is this simple!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.77%. Comparing base (f3769ae) to head (5b89040).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1571      +/-   ##
==========================================
- Coverage   99.83%   99.77%   -0.07%     
==========================================
  Files         130      130              
  Lines        4354     4359       +5     
  Branches      597      596       -1     
==========================================
+ Hits         4347     4349       +2     
- Misses          3        5       +2     
- Partials        4        5       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[petrelharp]

Hm - getting errors like

stdpopsim\slim_engine.py:1665: in simulate
    self._run_slim(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = _SLiMEngine()
script_file = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmppghxj46m.slim'
slim_path = 'slim', seed = None, dry_run = True, verbosity = None

    def _run_slim(
        self, script_file, slim_path=None, seed=None, dry_run=False, verbosity=None
    ):
        """
        Run SLiM.
    
        We capture the output using Popen's line-oriented text buffering
        (bufsize=1, universal_newlines=True) and redirect all messages to
        Python's logging module.
        By convention, messages from SLiM prefixed with "ERROR: " or
        "WARNING: " are treated as ERROR or WARN loglevels respectively.
        All other output on stdout is given the DEBUG loglevel.
        ERROR messages will raise a SLiMException here too, because
        they are always generated by the `stop()` eidos function which
        makes SLiM exit with a non-zero return code.
        """
        if slim_path is None:
            slim_path = self.slim_path()
    
        # SLiM v3.6 sends `stop()` output to stderr, which we rely upon.
        self._assert_min_version("4.0", slim_path)
    
        slim_cmd = [slim_path]
        if seed is not None:
            slim_cmd.extend(["-s", f"{seed}"])
        if dry_run:
            slim_cmd.extend(["-d", "dry_run=T"])
        if verbosity is not None:
            slim_cmd.extend(["-d", f"verbosity={verbosity}"])
        slim_cmd.append(script_file)
    
        with subprocess.Popen(
            slim_cmd,
            bufsize=1,
            universal_newlines=True,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        ) as proc:
            for line in proc.stdout:
                line = line.rstrip()
                if line.startswith("WARNING: "):
                    warnings.warn(
                        stdpopsim.UnspecifiedSLiMWarning(line[len("WARNING: ") :])
                    )
                else:
                    # filter `dbg` function calls that generate output
                    line = line.replace("dbg(self.source); ", "")
                    logger.debug(line)
    
            stderr = proc.stderr.read()
            for line in stderr.splitlines():
                if line.startswith("ERROR: "):
                    logger.error(line[len("ERROR: ") :])
    
        if proc.returncode != 0:
>           raise SLiMException(
                f"{slim_path} exited with code {proc.returncode}.\n{stderr}"
            )
E           stdpopsim.slim_engine.SLiMException: slim exited with code 1.
E           
E           ERROR (main): could not open input file: C:\Users\RUNNER~1\AppData\Local\Temp\tmppghxj46m.slim.

[petrelharp]

Uh-oh, and errors like this, which looks like a SLiM bug:

Error on script line 584, character 36:

    defineConstant("log", community.createLogFile("C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_sweep_meets_min_freq_at_s0\\sweep_start_af.logfile", logInterval=NULL));
                                    ^^^^^^^^^^^^^
FAILED tests/test_slim_engine.py::TestSelectiveSweep::test_sweep_meets_min_freq_at_end - stdpopsim.slim_engine.SLiMException: slim exited with code 1.
#ERROR (Eidos_WriteToFile): could not write to file at path D:\a\stdpopsim\stdpopsim/C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_sweep_meets_min_freq_at_e0\sweep_end_af.logfile.

[petrelharp]

Okay, there are Issues with temporary files on Windows. In the tests there's various ways to deal with temporary files, but it's unclear what's going on because sometimes the github logs don't show us all the output (e.g. "slim exited with code 3" but not showing us the error).

To debug I've got a minimal but not-minimal-enough test set-up in #1574.

Also for some reason conda cacheing seems to not be working.

[petrelharp]

Note: to make sure you see the full output from the tests (since the "Details" link tends to cut it off):

1. scroll to the bottom, making sure you see the summary like ================== 27 failed, 9 passed, 1 warning in 12.00s =================== so you know For Reals you're at the bottom
2. select a bit
3. scroll to the top, shift-click to select the whole thing
4. slowly scroll back through to the bottom so it all appears on the screen
5. copy-paste to elsewhere

[petrelharp]

Another note: we have in stdpopsim/slim_engine.py the _escape_eidos() function, that doubles up the backslashes in paths. This is necessary since otherwise we get errors like:

ERROR (EidosScript::Tokenize): illegal escape \U in string literal "C:".

Error on script line 21, character 36:

    defineConstant("trees_file", "C:\Users\RUNNER~1\AppData\Local\Temp\stdpopsim_bqckmhjx\8bcb71.trees");

[petrelharp]

The logfile issue looks to be a SLiM bug: MesserLab/SLiM#452 maybe

[petrelharp]

NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\stdpopsim_672p1gzn\\6c1381.slim

There seems to be some kind of issue with the temporary file code? (Note: this code was already changed from previously to deal with Windows not really doing temporary files in a reasonable or consistent way...)