Issue with extensions when Lib.Dir is modifiend in /etc/waagent.conf #2949
Comments
|
@cacheguard - Unfortunately there is no easy way to change the installation path for the agent, mainly for the reason you are pointing out: extensions are hardcoded to look for certificates under /var/lib/waagent. There is a protocol between the agent and extensions to communicate important file system paths that the extensions need in order to run, but certificates were never included in that protocol, so over the years extensions have hardcoded the use of /var/lib/waagent to get the certificates. Some extensions also hardcode this path for other purposes (although they shouldn't). We are aware of the restrictions on /var, and we discussed the issue of the installation path when we started our current development milestone, but did not pick it up during prioritization. I'll leave this issue open and bring it up during our next milestone. However, even if we take it, this will be a very slow process, given the large number of extensions that have hardcoded this path. There is a large number of development teams involved in the extension ecosystem, both within Microsoft and outside. |
|
Hi Narrieta, Thank you for your prompt response. For many reasons, we were not already convinced to use extensions but your clarifications allow us to take the final decision : we are not going to activate extensions in CacheGuard-OS. I think that even resetting the admin password in extensions is hardcoded (and do not use chpasswd() in the DefaultOSUtil class). Do you confirm? Best Regards, |
|
Hello @cacheguard
This would depend on the extension. Development of extensions and the Agent is done by different teams. If you are referring to VmAccess, you could ask this in their repo: https://github.com/Azure/azure-linux-extensions
This may be a perfectly valid decision for your scenario, but be aware that many features in Azure depend on extensions (backup, disk encryption, etc). Are you going to disable extensions setting "Extensions.Enabled=n" in waagent.conf? Doing it this way will, at least in recent agent versions, produce a good error message for anybody trying to use an extension. Otherwise extension operations may fail only after a long timeout (90 min) and end up failing with a very cryptic error message. |
|
Yes, I'm referring to VmAccess... We disabled extensions by setting "Extensions.Enabled=n" but that wasn't enough because the extension sub process is started regardless of Extensions.Enabled setting. To completely stop the extension sub process we had to disable allowExtensionOperations and provisionVMAgent in our ARM template as follows (as the extension sub process is started even with "Extensions.Enabled=n"):
It's a pity that there are so many inconsistencies between the agent, extensions and the way MS interprets settings in ARM templates. Sometimes we are completely lost... You know we decided to use waagent rather than cloud-init especially to be able to use extensions (mainly for password resetting as other features are rather useless in our case) and now we notice that using extensions is not going to be so easy. Also integrating the waagent seemed to us less complicated than integrating cloud-init. For AWS we developed our own minimalist interface with the cloud side without integrating cloud-init in our OS (which is another nightmare). Anyway, thanks a lot for your help ; greatly appreciated! Best Regards, |
|
@cacheguard - The extension subprocess is also in charge of allowing JIT (Just In Time) access to the VM, so it is started even if extensions are disabled. If you do not need that functionality either, it is totally possible to provision VMs without installing the Agent. This document describes that option and others, like using cloud-init, or even your own provisioning tool. https://learn.microsoft.com/en-us/azure/virtual-machines/linux/imaging As far as provisioning, cloud-init is the recommended way to do it. The provisioning code in the Agent is there only for legacy reasons and is no longer being maintained. Yes, "provisionVMAgent" is somehow misleading on Linux VMs. Its semantics are correct on Windows VMs, but in the case of Linux, the Agent comes preinstalled on most images in the marketplace and this flag just makes the agent process disable itself after provisioning. The above document also describes how to remove the agent altogether when using this flag. |
|
Now that we integrated the waagent into our OS, we are going to keep it (even if we are going to use it today for the provisioning only). Based on your clarifications, we finally set the provisionVMAgent to "true" to do not disable the agent. You know, CacheGuard-OS is a network appliance oriented OS and not a classical OS and integrating cloud-init and/or waagent isn't a straightforward task... We need to fine-tune every parameter and the testing process is very long. It's really a lack that MS does not propose a sandbox to make our tests more quickly (and without being obliged to pay Azure for inter continent bandwidth usage whenever we publish our VM image). Bes Regards, |
Hi,
As the default value for Lib.Dir variable in /etc/waagent.conf is /var/lib/waagent and /var is mounted with no exec rights in our OS, we decided to modify its location as follows:
Lib.Dir=/usr/local/waagentThe issue now is that we can't use extensions because the file /usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16//Utils/constants.py contains the following line:
LibDir = "/var/lib/waagent"Let's note in passing that the default value /var/lib/waagent for LibDir is not a good idea as executable extensions code are placed there. By doing so you create a security hole in a Linux system (and that's why secure Linux systems normally mount /var with noexec rights).
Below the content of the /var/log/azure/Microsoft.OSTCExtensions.VMAccessForLinux/extension.log file:
2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] sequence number is 0 2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] setting file path is/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/config/0.settings 2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON config: {"runtimeSettings": [{"handlerSettings": {"publicSettings": null, "protectedSettings": "*** REDACTED ***", "protectedSettingsCertThumbprint": "*** REDACTED ***"}}]} 2023/10/19 15:49:31 ERROR: CalledProcessError. Error Code is 2 2023/10/19 15:49:31 ERROR: CalledProcessError. Command was ['openssl', 'smime', '-inform', 'DER', '-decrypt', '-recip', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt', '-inkey', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.prv'] 2023/10/19 15:49:31 ERROR: CalledProcessError. Command result was Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory 2023/10/19 15:49:31 ERROR: 139972641794944:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r') 2023/10/19 15:49:31 ERROR: 139972641794944:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76: 2023/10/19 15:49:31 ERROR: unable to load certificate 2023/10/19 15:49:31 ERROR: 2023/10/19 15:49:31 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON exception decoding Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory 2023/10/19 15:49:31 ERROR: 139972641794944:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r') 2023/10/19 15:49:31 ERROR: 139972641794944:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76: 2023/10/19 15:49:31 ERROR: unable to load certificate 2023/10/19 15:49:31 ERROR: 2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Config decoded correctly. 2023/10/19 15:49:31 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Install,success,0,Install Succeeded 2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] sequence number is 0 2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] setting file path is/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/config/0.settings 2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON config: {"runtimeSettings": [{"handlerSettings": {"publicSettings": null, "protectedSettings": "*** REDACTED ***", "protectedSettingsCertThumbprint": "*** REDACTED ***"}}]} 2023/10/19 15:49:33 ERROR: CalledProcessError. Error Code is 2 2023/10/19 15:49:33 ERROR: CalledProcessError. Command was ['openssl', 'smime', '-inform', 'DER', '-decrypt', '-recip', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt', '-inkey', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.prv'] 2023/10/19 15:49:33 ERROR: CalledProcessError. Command result was Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory 2023/10/19 15:49:33 ERROR: 140583363873664:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r') 2023/10/19 15:49:33 ERROR: 140583363873664:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76: 2023/10/19 15:49:33 ERROR: unable to load certificate 2023/10/19 15:49:33 ERROR: 2023/10/19 15:49:33 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON exception decoding Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory 2023/10/19 15:49:33 ERROR: 140583363873664:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r') 2023/10/19 15:49:33 ERROR: 140583363873664:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76: 2023/10/19 15:49:33 ERROR: unable to load certificate 2023/10/19 15:49:33 ERROR: 2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Config decoded correctly. 2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] set most recent sequence number to 0 2023/10/19 15:49:33 ERROR: Error Traceback (most recent call last): 2023/10/19 15:49:33 ERROR: File "/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/Utils/extensionutils.py", line 353, in add_extension_event 2023/10/19 15:49:33 ERROR: event.save() 2023/10/19 15:49:33 ERROR: File "/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/Utils/extensionutils.py", line 315, in save 2023/10/19 15:49:33 ERROR: os.mkdir(event_folder) 2023/10/19 15:49:33 ERROR: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/waagent/events' 2023/10/19 15:49:33 ERROR: 2023/10/19 15:49:33 ERROR: CalledProcessError. Error message is [Errno 2] No such file or directory: '/usr/sbin/service' 2023/10/19 15:49:33 ERROR: Failed to restart SSH service with return code:2 2023/10/19 15:49:33 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Failed to enable the extension with error: 'str' object has no attribute 'get', stack trace: Traceback (most recent call last): 2023/10/19 15:49:33 ERROR: File "/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/./vmaccess.py", line 156, in enable 2023/10/19 15:49:33 ERROR: if _is_sshd_config_modified(protect_settings): 2023/10/19 15:49:33 ERROR: File "/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/./vmaccess.py", line 201, in _is_sshd_config_modified 2023/10/19 15:49:33 ERROR: result = protected_settings.get('reset_ssh') or protected_settings.get('password') 2023/10/19 15:49:33 ERROR: AttributeError: 'str' object has no attribute 'get' 2023/10/19 15:49:33 ERROR: 2023/10/19 15:49:33 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Enable,error,0,Enable failed: 'str' object has no attribute 'get' 2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] sequence number is 0 2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] setting file path is/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/config/0.settings 2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON config: {"runtimeSettings": [{"handlerSettings": {"publicSettings": null, "protectedSettings": "*** REDACTED ***", "protectedSettingsCertThumbprint": "*** REDACTED ***"}}]} 2023/10/19 15:52:35 ERROR: CalledProcessError. Error Code is 2 2023/10/19 15:52:35 ERROR: CalledProcessError. Command was ['openssl', 'smime', '-inform', 'DER', '-decrypt', '-recip', '/var/lib/waagent/*** REDACTED ***.crt', '-inkey', '/var/lib/waagent/*** REDACTED ***.prv'] 2023/10/19 15:52:35 ERROR: CalledProcessError. Command result was Can't open /var/lib/waagent/*** REDACTED ***.crt for reading, No such file or directory 2023/10/19 15:52:35 ERROR: 140075000892288:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/*** REDACTED ***.crt','r') 2023/10/19 15:52:35 ERROR: 140075000892288:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76: 2023/10/19 15:52:35 ERROR: unable to load certificate 2023/10/19 15:52:35 ERROR: 2023/10/19 15:52:35 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON exception decoding Can't open /var/lib/waagent/*** REDACTED ***.crt for reading, No such file or directory 2023/10/19 15:52:35 ERROR: 140075000892288:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/*** REDACTED ***.crt','r') 2023/10/19 15:52:35 ERROR: 140075000892288:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76: 2023/10/19 15:52:35 ERROR: unable to load certificate 2023/10/19 15:52:35 ERROR: 2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Config decoded correctly. 2023/10/19 15:52:35 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Current sequence number, 0, is not greater than the sequence number of the most recent executed configuration. Exiting... 2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] sequence number is 0 2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] setting file path is/usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16/config/0.settings 2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON config: {"runtimeSettings": [{"handlerSettings": {"publicSettings": null, "protectedSettings": "*** REDACTED ***", "protectedSettingsCertThumbprint": "*** REDACTED ***"}}]} 2023/10/19 15:52:58 ERROR: CalledProcessError. Error Code is 2 2023/10/19 15:52:58 ERROR: CalledProcessError. Command was ['openssl', 'smime', '-inform', 'DER', '-decrypt', '-recip', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt', '-inkey', '/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.prv'] 2023/10/19 15:52:58 ERROR: CalledProcessError. Command result was Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory 2023/10/19 15:52:58 ERROR: 140299211979648:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r') 2023/10/19 15:52:58 ERROR: 140299211979648:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76: 2023/10/19 15:52:58 ERROR: unable to load certificate 2023/10/19 15:52:58 ERROR: 2023/10/19 15:52:58 ERROR: [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] JSON exception decoding Can't open /var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt for reading, No such file or directory 2023/10/19 15:52:58 ERROR: 140299211979648:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/var/lib/waagent/28FFE9210CC03B8D16BD9A5C29DCD34CD09E825E.crt','r') 2023/10/19 15:52:58 ERROR: 140299211979648:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76: 2023/10/19 15:52:58 ERROR: unable to load certificate 2023/10/19 15:52:58 ERROR: 2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Config decoded correctly. 2023/10/19 15:52:58 [Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16] Current sequence number, 0, is not greater than the sequence number of the most recent executed configuration. Exiting...As you can see extensions codes can't even find the certificate...
Is there a way to configure the agent to place extensions in another location? Or modify the Lib.Dir value in /usr/local/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.5.16//Utils/constants.py during the initialization phase?
Best Regards,
The CacheGuard Dev Team
The text was updated successfully, but these errors were encountered: