Unexpected OpenSSL dll files in C:\Windows\System32

[MSP-Greg]

While working recently on a Ruby trunk build using OpenSSL master (vers 1.1.1), I had build problems, and the issue was that two OpenSSL files (libcrypto-1_1-x64.dll & libssl-1_1-x64.dll) are in C:\Windows\System32.

I've never looked for MSFT policy, but I thought only MSFT files are located in System32. Below is file info. It doesn't look like these are MSFT. If they aren't, will they remain?

Thanks Greg

(Get-Item -Path 'C:\Windows\System32\libssl-1_1-x64.dll').VersionInfo | Format-List -Force

OriginalFilename  : libssl-1_1-x64.dll
FileDescription   : OpenSSL shared library
ProductName       : The OpenSSL Toolkit
Comments          : 
CompanyName       : The OpenSSL Project, http://www.openssl.org/
FileName          : C:\Windows\System32\libssl-1_1-x64.dll
FileVersion       : 1.1.0f
ProductVersion    : 1.1.0f
IsDebug           : False
IsPatched         : False
IsPreRelease      : False
IsPrivateBuild    : False
IsSpecialBuild    : False
Language          : English (United States)
LegalCopyright    : Copyright 1998-2016 The OpenSSL Authors. All rights reserved.
LegalTrademarks   : 
PrivateBuild      : 
SpecialBuild      : 
FileVersionRaw    : 1.1.0.6
ProductVersionRaw : 1.1.0.6


(Get-Item -Path 'C:\Windows\System32\libcrypto-1_1-x64.dll').VersionInfo | Format-List -Force

OriginalFilename  : libcrypto-1_1-x64.dll
FileDescription   : OpenSSL shared library
ProductName       : The OpenSSL Toolkit
Comments          : 
CompanyName       : The OpenSSL Project, http://www.openssl.org/
FileName          : C:\Windows\System32\libcrypto-1_1-x64.dll
FileVersion       : 1.1.0f
ProductVersion    : 1.1.0f
IsDebug           : False
IsPatched         : False
IsPreRelease      : False
IsPrivateBuild    : False
IsSpecialBuild    : False
Language          : English (United States)
LegalCopyright    : Copyright 1998-2016 The OpenSSL Authors. All rights reserved.
LegalTrademarks   : 
PrivateBuild      : 
SpecialBuild      : 
FileVersionRaw    : 1.1.0.6
ProductVersionRaw : 1.1.0.6

[IlyaFinkelshteyn]

AFAIK those files are not included into default Windows Server installation. However some of numerous software installations applied to AppVeyor worker images could put them into %Windir%\System32\ just because this is the easiest way to make files discover-able without modifying path environment variable. Try just delete them at init or install stage and see if this unblock you without breaking any other part of build. Let us know if this helps.

[MSP-Greg]

@IlyaFinkelshteyn,

First, sorry. I forgot to mention that I renamed the files immediately before they're problematic, and 'restored' them after.

some of numerous software installations applied to AppVeyor worker images could put them into %Windir%\System32\

Not good. I've been using MSFT OS's since DOS days, and coding for longer. This took me a while to find, and there are some Appveyor users that are 80% *nix/MacOS, 20% Windows. I'm sure some would give up.

I might suggest locking off %Windir% when doing the installs/updates. If they're not MSFT files, they shouldn't be there, and PATH should include their location. Also, there are ways for apps to find dll's without using PATH.

IOW, one should be able to resolve dll conflicts by modifying PATH, without needing to rename files. I can't remove %Windir%\System32 from PATH...

Thanks, Greg

[IlyaFinkelshteyn]

@MSP-Greg good thing about AppVeyor builds is that each runs in fresh transient VM, which is killed immediately after build finished. So you should not be careful about 'restoring' those files and should not afraid to delete them :)

[MSP-Greg]

@IlyaFinkelshteyn

I wasn't sure whether any Appveyor code used them...

Dumb question. I have a few artifacts that are used by other projects. Is there a different URL to use in
a situation like that?

BTW, I do try to be a good user and flush my old artifacts...

[IlyaFinkelshteyn]

If I understand your question correctly, you need permalink?

Right now we do not have any UI or API to delete old artifacts. However recently implemented retention policy forcibly makes all users good :)

[MSP-Greg]

Sorry. Poorly written.

Permalink are described somewhere in the docs.

What I'm referring to is in a build script for ProjA, I want to use an artifact from a build in ProjB.

Is the a different URL that can be used, as one might say it's part of the same network?

[IlyaFinkelshteyn]

No, the same URL. Build VMs are isolated from each other and anyway artifacts are being stored on separate clouds as Azure and/or GCE.

[MSP-Greg]

@IlyaFinkelshteyn

As long as I'm on odd questions, is the build/job metadata (json, strings shown in the tests or messages tabs, etc) on a separate cloud also?

[IlyaFinkelshteyn]

Build VMs are isolated and transient, all metadata is stored outside of actual build environment.

[Ede123]

I just hit this bug, too:

I'm building Inkscape for Windows in an MSYS2/mingw-w64 environment (https://ci.appveyor.com/project/inkscape/inkscape/history). Recently builds suddenly started to fail as the executable was crashing.

I first suspected an issue on the MSYS2 side as OpenSSL was recently updated to version 1.1.1.
It turns out however that my executable was loading the conflicting DLLs in the system32 directory which are older and therefore incompatible.

The reason is that Microsoft's DLL load order dictates applications to look into their own path first, in the System directories second and only after that consult the PATH variable.

Therefore copying non-system DLLs into system32 will result in applications to prefer them over their own working DLLs (unless they are copied into the executable's directory), even if the PATH variable is set up properly.
This makes it hard to impossible to get the desired behavior, as the load order can not be easily overridden.

The problematic part is that such DLL issues are extremely hard to debug (it's not without reason the name "DLL hell" formed for this) which is why I'd also strongly recommend not to put any non-system DLLs into system locations.

[MSP-Greg]

The problematic part is that such DLL issues are extremely hard to debug

Yeah, I wasn't to happy when I found the dll's there. It took a while, because (at least for me), it wasn't expected, and it broke CI code that had worked for a long time.

I think the installer for the ShiningLight OpenSSL package allows for the dll's to be placed in system32. I don't know if the option existed previously, but either way, as you stated, copying non-system DLLs into system32 is a very bad idea.

[FeodorFitsner]

The question is what "software" put those DLLs into System32 in the first place and what if that "software" relies on them to properly work?

I think we should just take that risk and remove mentioned DLLs from System32. If someone's build is broken they would be able to xcopy DLLs from OpenSSL installation.

[MSP-Greg]

@FeodorFitsner

While recently working some scripting to set up systems, I needed to find out what vc runtime was used in OpenSSL-Win64.

As I recall, like 7Zip, there isn't a simple 'zip type' file of the app, it's either an exe or an msi. When I ran it, there was an option as to where to install (what I assume were) the dll files. So, you might check if a script you've got does that by default. Also, not knowing how up on OpenSSL you are, the dll files changed names between 1.0.2 and 1.1.0...

I think we should just take that risk and remove mentioned DLLs from System32.

Agreed.

[MSP-Greg]

@FeodorFitsner

JFYI, working in my Ruby fork today, and the files are still in System32...

[FeodorFitsner]

Sorry for the delay with images update! It's going to be early next week.

[FeodorFitsner]

While looking into this issue on master VMs I've determined that those lib* DLLs were installed into C:\Windows\System32 by MongoDB installer.

Even its installer file name suggests it has to do something with SSL: mongodb-win32-x86_64-2008plus-ssl-4.0.3-signed.msi

[MSP-Greg]

Thanks for taking the time to research this. I hate to ask, but you might double check that.

I downloaded mongodb-win32-x86_64-2008plus-ssl-4.0.3-signed.msi and mongodb-win32-x86_64-2008plus-ssl-4.0.3.zip, and both seem to have OpenSSL 1.0.2 files (libeay32.dll & ssleay32.dll, same name for both 32 & 64).

The files in System32 are 1.1.+ files (32 bit - libcrypto-1_1.dll & libssl-1_1.dll, 64 bit - libcrypto-1_1-x64.dll & libssl-1_1-x64.dll). These files exists in the SLProWeb.com OpenSSL Win32/Win64 builds (vc140).

So, these files are only 'conflicting' when another app is using OpenSSL 1.1.0 or 1.1.1.

1.1.1 was just released in September, and MSYS2 apps only recently converted from 1.0.2 to 1.1.1, bypassing 1.1.0. JFYI, ruby-loco & RubyInstaller2 Ruby 2.5.x have used 1.1.0 since it's release (we build 'custom' MSYS2 compatible packages).

[FeodorFitsner]

Yeah, I saw libeay32.dll & ssleay32.dll in MongoDB's bin folder. However, comparing file timestamps in System32 with the date of Mongo installation it looks like they were added by Mongo.

Anyway, we are removing them in the next release.

[MSP-Greg]

Thanks for double checking. I don't mean to be a PITA, I've kind of got my hands in a few repos' appveyor.yml files...

I hate to ask, and I haven't checked previously, but can you close the milestone when the new images are active?

Off topic - I've got a repo that has code used for CI in other repos (tests & builds a gem on every CI), and I need to update it for the new OpenSSL versions used in Ruby 2.4 & 2.5. I could make the code a lot messier with Ruby version checks, but I really don't want to do that...

[iseegr8tfuldeadppl]

I fixed this by installing Win64 OpenSSL v1.1.1b.exe from https://slproweb.com/products/Win32OpenSSL.html
scroll all the way to the bottom for it.
while installing just make sure u don't modify anything specified by the installer just click next next next.

[albertosottile]

We just hit this again when trying to test the packing of Python SSL module with py2exe. See here for further details. Are these new images live already?