Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sporadic cleanup issues on AppVeyor after upload #14

Closed
jakirkham opened this issue May 1, 2018 · 7 comments · Fixed by #20 · May be fixed by #17
Closed

Sporadic cleanup issues on AppVeyor after upload #14

jakirkham opened this issue May 1, 2018 · 7 comments · Fixed by #20 · May be fixed by #17

Comments

@jakirkham
Copy link
Member

As noted in this comment and again in this comment, it seems that sometimes we lack permissions to do a little bit of cleanup on AppVeyor (possibly due to resource contention), which causes the build to fail after uploading packages. Since this appears to be occurring after upload it isn't actually messing the uploads, but it does give one pause. Restarting seems to make it go away, but it does show up too often for comfort and it does make users nervous. Maybe sleeping for a bit before cleaning up would avoid this issue. There may be better fixes though.

Traceback (most recent call last):
  File "C:\Miniconda36-x64\Scripts\upload_or_check_non_existence.py", line 131, in <module>
    main()
  File "C:\Miniconda36-x64\Scripts\upload_or_check_non_existence.py", line 121, in main
    upload(cli, fname, owner, channel)
  File "C:\Miniconda36-x64\Scripts\upload_or_check_non_existence.py", line 68, in upload
    env=os.environ)
  File "C:\Miniconda36-x64\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\Miniconda36-x64\Scripts\upload_or_check_non_existence.py", line 27, in get_temp_token
    shutil.rmtree(dn)
  File "C:\Miniconda36-x64\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Miniconda36-x64\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Miniconda36-x64\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmphmihw0df\\binstar.token'

ref: https://ci.appveyor.com/project/conda-forge/conda-feedstock/build/1.0.122
ref: https://ci.appveyor.com/project/conda-forge/r-base-feedstock/build/1.0.90

@nehaljwani
Copy link
Member

But not all pkgs have been uploaded! Current status for windows:

  • C:\bld\win-64\go-core-1.10.3-h41a7c8f_3.tar.bz2
  • C:\bld\win-64\go_linux-64-1.10.3-h0f5337a_3.tar.bz2
  • C:\bld\win-64\go_osx-64-1.10.3-h144c09f_3.tar.bz2
  • C:\bld\win-64\go_win-64-1.10.3-h2df250f_3.tar.bz2
  • C:\bld\win-64\go-1.10.3-3.tar.bz2

@jakirkham
Copy link
Member Author

This issue only occurs after all uploads were attempted. It actually has nothing to do with uploading itself, but some finalization step after uploading.

Looking at the log, it says the following...

Attempting to finalize metadata for go_osx-64
Attempting to finalize metadata for go_win-64
Attempting to finalize metadata for go

Normally these are followed by a Solving environment: ...working... done line or so, but these were not.

IOW it tried to upload those packages, but failed for some reason. Unfortunately the log doesn't give us enough information to know what, but it sounds like it failed to render. Maybe a bug in the recipe or the tooling. Would check the former first.

@jakirkham
Copy link
Member Author

The rendering happens on this line. So that is the best thing to test for issues.

If that works ok, maybe there was a networking issue on this line, but would expect a traceback.

@nehaljwani
Copy link
Member

I don't think the problem is with rendering. I wrote a small snippet to test this out:

import conda_build.api

metas = conda_build.api.render('.', variant_config_files=['..\.ci_support\win_.yaml'])
for meta, _, _ in metas:
    fnames = conda_build.api.get_output_file_paths(meta)
    for fname in fnames:
        print(fname)

Although it says 'Solving environment' only three times, it still prints all the names.

@nehaljwani
Copy link
Member

The upload function is called each time for each tarball, so even if one calls returns the Access Denied error, the rest of the uploads are ignored.

@jakirkham
Copy link
Member Author

What I find interesting is after restarting the CI it keeps failing on those 3 specifically. It may very well be a network issue, but it seems a little strange that it doesn't fail elsewhere. 🤔

@nehaljwani
Copy link
Member

Anaconda Client does this:

def get_server_api(token=None, site=None, cls=None, config=None, **kwargs):
    """
    Get the anaconda server api class
    """
    if not cls:
        from binstar_client import Binstar
        cls = Binstar

    config = config if config is not None else get_config(site=site)

    url = config.get('url', DEFAULT_URL)

    logger.info("Using Anaconda API: %s", url)

    if token:
        logger.debug("Using token from command line args")
    elif 'BINSTAR_API_TOKEN' in os.environ:
        logger.debug("Using token from environment variable BINSTAR_API_TOKEN")
        token = os.environ['BINSTAR_API_TOKEN']
    elif 'ANACONDA_API_TOKEN' in os.environ:
        logger.debug("Using token from environment variable ANACONDA_API_TOKEN")
        token = os.environ['ANACONDA_API_TOKEN']
    else:
        token = load_token(url)

    verify = config.get('ssl_verify', config.get('verify_ssl', True))

    return cls(token, domain=url, verify=verify, **kwargs)

Why not just rely on the environment variable? Why create a file and then read from it and and delete it later?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants