Compilation error on Windows - missing directory

[jqluisneumann]

I am trying read and compile a Lightgbm model but I am having issues with temporary files. My code is as follows:

import treelite
model = treelite.Model.load('lgb_model.txt', model_format='lightgbm')
model.export_lib(toolchain='msvc', libpath='./lgb_model.dll', verbose=True)

And I get the following output:

[00:41:46] c:\projects\treelite-wheels\treelite\src\compiler\ast_native.cc:22: Using ASTNativeCompiler
[00:41:47] c:\projects\treelite-wheels\treelite\src\compiler\ast\split.cc:10: Parallel compilation disabled; all member trees will be dumped to a single source file. This may increase compilation time and memory usage.
[00:41:48] c:\projects\treelite-wheels\treelite\src\c_api\c_api.cc:297: Code generation finished. Writing code to files...
[00:41:48] c:\projects\treelite-wheels\treelite\src\c_api\c_api.cc:314: Writing file header.h...
[00:41:48] c:\projects\treelite-wheels\treelite\src\c_api\c_api.cc:314: Writing file main.c...
[00:41:49] c:\projects\treelite-wheels\treelite\src\c_api\c_api.cc:314: Writing file recipe.json...
[00:41:49] C:\Users\myusername\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\treelite\contrib\__init__.py:208: WARNING: some of the source files are long. Expect long compilation time. You may want to adjust the parameter parallel_comp.

[00:41:49] C:\Users\myusername\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\treelite\contrib\util.py:98: Compiling sources files in directory C:\Users\myusername\AppData\Local\Temp\tmp2fs01b49 into object files (*.obj)...

But the compilation fails because it can't write to a temp file. The tempfolder never gets created, and the retcode_cpu0.txt file gets created on the same location of the lightgbm model and Jupyter notebook (I am running this in Jupyter Lab).

FileNotFoundError                         Traceback (most recent call last)
<ipython-input-4-e9892c80d16e> in <module>
----> 1 model.export_lib(toolchain='msvc', libpath='./lgb_model.dll', verbose=True)

~\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\treelite\frontend.py in export_lib(self, toolchain, libpath, params, compiler, verbose, nthread, options)
     97       self.compile(temp_dir, params, compiler, verbose)
     98       temp_libpath = create_shared(toolchain, temp_dir, nthread,
---> 99                                    verbose, options)
    100       shutil.move(temp_libpath, libpath)
    101 

~\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\treelite\contrib\__init__.py in create_shared(toolchain, dirpath, nthread, verbose, options)
    219     from .gcc import _create_shared, _openmp_supported
    220   libpath = \
--> 221     _create_shared(dirpath, toolchain, recipe, nthread, options, verbose)
    222   if verbose:
    223     log_info(__file__, lineno(),

~\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\treelite\contrib\msvc.py in _create_shared(dirpath, toolchain, recipe, nthread, options, verbose)
     86                           .format(_varsall_bat_path(),
     87                                   'amd64' if _is_64bit_windows() else 'x86')
---> 88   return _create_shared_base(dirpath, recipe, nthread, verbose)
     89 
     90 def _check_ext(dllpath):

~\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\treelite\contrib\util.py in _create_shared_base(dirpath, recipe, nthread, verbose)
    114   result = []
    115   for tid in range(ncpu):
--> 116     result.append(_wait(proc[tid], workqueue[tid]))
    117 
    118   for tid in range(ncpu):

~\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\treelite\contrib\util.py in _wait(proc, args)
     81   dirpath = args['dirpath']
     82   stdout, _ = proc.communicate()
---> 83   with open(os.path.join(dirpath, 'retcode_cpu{}.txt'.format(tid)), 'r') as f:
     84     retcode = [int(line) for line in f]
     85   return {'stdout':_str_decode(stdout), 'retcode':retcode}

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\myusername\\AppData\\Local\\Temp\\tmp2fs01b49\\retcode_cpu0.txt'

[hcho3]

@jqluisneumann I'm not too familiar with how temporary directories work on Windows. Do you know the best practices for handling temporaries on Windows?

[jqluisneumann]

Not really, I am afraid. But I have managed to find a workaround while looking at the frontend code. As export lib is just a convenience function, I managed to compile a model using "model.compile" and "treelite.shared", as long as i set dirpath in model.compile to the same directory where I have my jupyter notebook, lightgbm model, etc.

So most likely a bug in the tempdir allocation.

[hcho3]

@jqluisneumann That's good to hear. Yes, I'd like to spend some time to learn more about how temporaries work in Windows.

[hcho3]

@jqluisneumann Update: I fixed the issue with temporary directories in #101. From now on, Treelite will always create temporary directories in the same directory as the libpath specified in export_lib().