Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hashicorp Vault component #1348

Open
samar-elsayed opened this issue Aug 5, 2024 · 35 comments
Open

Hashicorp Vault component #1348

samar-elsayed opened this issue Aug 5, 2024 · 35 comments
Labels
bug Something isn't working
Milestone
4.9.0

Comments

@samar-elsayed
Copy link

samar-elsayed commented Aug 5, 2024 •
edited
Loading

Describe the bug

I have tried to add a vault component as shown below :

image

but once I run Jbang run, it gives me the following error

image

Steps to reproduce the behavior

as mentioned above

Variant

Vs Code Extension

Container Management (if applicable)

None

Operating System (if applicable)

Windows

Version

4.6.0

Relevant log output

No response
@samar-elsayed samar-elsayed added the bug Something isn't working label Aug 5, 2024
@mgubaidullin
Copy link
Contributor

I think you need to configure Hashicorp Vault Property Function
https://camel.apache.org/components/4.4.x/hashicorp-vault-component.html#_using_hashicorp_vault_property_function

@samar-elsayed
Copy link
Author

Shouldn't these properties be set from the UI itself ?

I have set them except for secretPath but I faced that error:
java.lang.IllegalArgumentException: Secret Path must be specified

so I tried setting the below property:
camel.vault.hashicorp.secretPath

but it also failed with:

Error binding property (camel.vault.hashicorp.secretPath=online) with name: secretPath on bean: org.apache.camel.vault.HashicorpVaultConfiguration@147e0734

@mgubaidullin so please let me know how to specify the secretPath

@oscerd
Copy link
Contributor

oscerd commented Aug 11, 2024

Secret properties function is a thing, the producer methods are a different thing. If you use secret path, you need to set an header. When you set camel.vault.hashicorp.* properties you're setting up the secret properties function. The function will resolve parameters for you inside a route with the following syntax {{hashicorp:engine:secretname}}

@samar-elsayed
Copy link
Author

@oscerd can you share with me if is there a default name for the header that hashicorp component supports and uses to set the secretPath ?

@oscerd
Copy link
Contributor

oscerd commented Aug 11, 2024

It is CamelHashicorpVaultSecretPath

@mgubaidullin
Copy link
Contributor

@samar-elsayed you can always find component headers in the property panel

headers

@samar-elsayed
Copy link
Author

@mgubaidullin

While trying with Karavan vscode 4.7.0, I found that camel.vault.hashicorp.engine is not recognized and the below error happens although I didn't face this issue while using 4.6.0

org.apache.camel.PropertyBindingException: Error binding property (camel.vault.hashicorp.engine=secret) with name: engine on bean: org.apache.camel.vault.HashicorpVaultConfiguration@6ddd1c51 with value: secret

@oscerd
Copy link
Contributor

oscerd commented Aug 23, 2024 via email

@oscerd
Copy link
Contributor

oscerd commented Aug 23, 2024

https://camel.apache.org/components/next/hashicorp-vault-component.html#_using_hashicorp_vault_property_function

@samar-elsayed
Copy link
Author

@oscerd could you give me an example to log a value of a secret ?

for instance, I have a secret that is located in that path main/test/apiKey

I have tried to use the below but it just prints main/test/apiKey literally

              - log:
                  id: log-a4d7
                  disabled: false
                  message: "{{hashicorp-vault:secret:main/test/apiKey}}"

@oscerd
Copy link
Contributor

oscerd commented Aug 26, 2024 •
edited
Loading

Do you have hashicorp vault component in your classpath? Because otherwise it won't resolve anything.

@samar-elsayed
Copy link
Author

@oscerd yes I have it already inside my karavan project

I noticed while apache camel was connecting to vault, I found hashicorp-vault component adds the following https://host:port/v1/secret/data to my secret path

but at my company we are using v2 secret engine so I think the issue may be related to the secret engine version

@oscerd
Copy link
Contributor

oscerd commented Aug 26, 2024

spring-projects/spring-vault#738

@oscerd
Copy link
Contributor

oscerd commented Aug 26, 2024

I think we need to revisit this and check compatibility with v1 and v2

@samar-elsayed
Copy link
Author

@oscerd I got a chance to try this with v1 but I faced the below error

org.apache.camel.FailedToCreateRouteException: Failed to create route Inquiry at: >>> step -> [[Log[Inquiry Req body from BE is ${exchangeProperty.InquiryReqBody}], SetHeader[CamelHashicorpVaultSecretPath, simple{onlineMidTakseet/test}], To[hashicorp-vault:secret?host=10.90.3.108&operation=getSecret&port=443&scheme=https&token=xxxxxx], Log[{{hashicorp:secret:onlineMidTakseet/test}}], process[ref:InquiryReq], Log[Inquiry Req to SP is  === ${body}], To[https://check_app_properities/?bridgeEndpoint=false&httpMethod=POST&skipRequestHeaders=false], Log[Inquiry Res from SP is === ${body}], process[ref:InquiryRes], Log[Inquiry Res to BE === ${body}]]] <<< in route: Route(Inquiry)[From[direct:Inquiry?bridgeErrorHandler=false]... because of Something went wrong while recovering onlineMidTakseet from vault
        at org.apache.camel.reifier.RouteReifier.doCreateRoute(RouteReifier.java:235)        
        at org.apache.camel.reifier.RouteReifier.createRoute(RouteReifier.java:86)
        at org.apache.camel.impl.DefaultModelReifierFactory.createRoute(DefaultModelReifierFactory.java:49)
        at org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:720)
        at org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:587)
        at org.apache.camel.impl.engine.AbstractCamelContext.doInit(AbstractCamelContext.java:2390)
        at org.apache.camel.support.service.BaseService.init(BaseService.java:78)
        at org.apache.camel.impl.engine.AbstractCamelContext.init(AbstractCamelContext.java:2024)
        at org.apache.camel.support.service.BaseService.start(BaseService.java:105)
        at org.apache.camel.impl.engine.AbstractCamelContext.start(AbstractCamelContext.java:2043)
        at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:211)     
        at org.apache.camel.main.KameletMain.doStart(KameletMain.java:364)
        at org.apache.camel.support.service.BaseService.start(BaseService.java:113)
        at org.apache.camel.dsl.jbang.core.commands.Run.runKameletMain(Run.java:1336)        
        at org.apache.camel.dsl.jbang.core.commands.Run.run(Run.java:873)
        at org.apache.camel.dsl.jbang.core.commands.Run.doCall(Run.java:320)
        at org.apache.camel.dsl.jbang.core.commands.CamelCommand.call(CamelCommand.java:71)  
        at org.apache.camel.dsl.jbang.core.commands.CamelCommand.call(CamelCommand.java:37)  
        at picocli.CommandLine.executeUserObject(CommandLine.java:2045)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)     
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
        at picocli.CommandLine.execute(CommandLine.java:2174)
        at org.apache.camel.dsl.jbang.core.commands.CamelJBangMain.run(CamelJBangMain.java:168)
        at org.apache.camel.dsl.jbang.core.commands.CamelJBangMain.run(CamelJBangMain.java:61)
        at main.CamelJBang.main(CamelJBang.java:36)
Caused by: org.apache.camel.RuntimeCamelException: Something went wrong while recovering onlineMidTakseet from vault
        at org.apache.camel.component.hashicorp.vault.HashicorpVaultPropertiesFunction.apply(HashicorpVaultPropertiesFunction.java:178)
        at org.apache.camel.component.properties.DefaultPropertiesParser$ParsingContext.getPropertyValue(DefaultPropertiesParser.java:338)
        at org.apache.camel.component.properties.DefaultPropertiesParser$ParsingContext.readProperty(DefaultPropertiesParser.java:232)
        at org.apache.camel.component.properties.DefaultPropertiesParser$ParsingContext.doParseNested(DefaultPropertiesParser.java:169)
        at org.apache.camel.component.properties.DefaultPropertiesParser$ParsingContext.parse(DefaultPropertiesParser.java:118)
        at org.apache.camel.component.properties.DefaultPropertiesParser.parseUri(DefaultPropertiesParser.java:70)
        at org.apache.camel.component.properties.PropertiesComponent.parseUri(PropertiesComponent.java:336)
        at org.apache.camel.component.properties.PropertiesComponent.parseUri(PropertiesComponent.java:172)
        at org.apache.camel.impl.engine.DefaultCamelContextExtension.resolvePropertyPlaceholders(DefaultCamelContextExtension.java:327)
        at org.apache.camel.impl.engine.AbstractCamelContext.resolvePropertyPlaceholders(AbstractCamelContext.java:1584)
        at org.apache.camel.support.CamelContextHelper.parseText(CamelContextHelper.java:388)        at org.apache.camel.reifier.AbstractReifier.parseString(AbstractReifier.java:58)     
        at org.apache.camel.reifier.LogReifier.createProcessor(LogReifier.java:49)
        at org.apache.camel.reifier.ProcessorReifier.createProcessor(ProcessorReifier.java:863)
        at org.apache.camel.reifier.ProcessorReifier.createOutputsProcessor(ProcessorReifier.java:816)
        at org.apache.camel.reifier.StepReifier.createOutputsProcessor(StepReifier.java:42)  
        at org.apache.camel.reifier.ProcessorReifier.createOutputsProcessor(ProcessorReifier.java:590)
        at org.apache.camel.reifier.ProcessorReifier.createChildProcessor(ProcessorReifier.java:612)
        at org.apache.camel.reifier.StepReifier.createProcessor(StepReifier.java:36)
        at org.apache.camel.reifier.ProcessorReifier.makeProcessor(ProcessorReifier.java:886)        at org.apache.camel.reifier.ProcessorReifier.addRoutes(ProcessorReifier.java:622)    
        at org.apache.camel.reifier.RouteReifier.doCreateRoute(RouteReifier.java:231)        
        ... 28 more

@oscerd
Copy link
Contributor

oscerd commented Aug 27, 2024

It's impossible to say without knowing the configuration of the vault and how you configured the properties.

@samar-elsayed
Copy link
Author

@oscerd you mean the properties I have configured for vault component ? or the configuration we have for the vault instance itself? and if it is the latter then which part of the vault instance configuration is needed for troubleshooting?

@oscerd
Copy link
Contributor

oscerd commented Aug 29, 2024

Can you try to enable debug logging level and see what happens?

@oscerd
Copy link
Contributor

oscerd commented Sep 3, 2024

With the debug log level, you should be able to see a bit better what is happening.

@samar-elsayed
Copy link
Author

@oscerd well I was not able to simulate the last issue I shared here again but I am back to the first issue I faced which is not being able to get the value of the fields under /onlineMidTakseet/test?version=1 ( I have 2 fields (key=value) under this path and I want to at least get one of them )
I have enabled the debug mode and the last highlighted line is the log step where I used {{hashicorp-vault:secret:onlineMidTakseet/test@1}} but as you can see in the below screenshot it only logged secret:onlineMidTakseet/test@1= not the value itself

image

so can you please help me to just get the field value from the vault ?
thanks

@oscerd
Copy link
Contributor

oscerd commented Sep 3, 2024

It should be something like

{{hashicorp-vault:secret:onlineMidTakseet/test:<key_name>@1}}

@samar-elsayed
Copy link
Author

samar-elsayed commented Sep 3, 2024 •
edited
Loading

@oscerd still the same result I cant get its value

the log step only give me the below

image

@oscerd
Copy link
Contributor

oscerd commented Sep 3, 2024

well, the call seems to be done, I really have a no idea. The function is working in the local tests and even by testing through Kamelet. I cannot reproduce.

@samar-elsayed
Copy link
Author

samar-elsayed commented Sep 3, 2024 •
edited
Loading

In this case, I will need to use k8 secrets the normal way and exclude hashicorp-vault for now until a further solution is available

Thanks @oscerd for following up with me

@oscerd
Copy link
Contributor

oscerd commented Sep 3, 2024

The solution is working in 4.7.0, there are no further solutions to be find, I cannot access your environment so I cannot really reproduce.

@oscerd
Copy link
Contributor

oscerd commented Sep 3, 2024

I can try to setup a route with pure Kamelets and an Hashicorp Vault docker image to show it will log the secret if you want.

@samar-elsayed
Copy link
Author

@oscerd I am using 4.7.0

I have shared here the vault config as a last resort

image

image

@oscerd
Copy link
Contributor

oscerd commented Sep 3, 2024

I'll try with v2 and see, maybe it's because of that.

@oscerd
Copy link
Contributor

oscerd commented Sep 5, 2024

I found the time to reproduce. Essentially the problem is with the secret path. The Hashicorp properties function is not able to distinguish between the slash for the subkey and the slash in your secret path. So if you something like onlineMidTakseet as path and not onlineMidTaskseet/test it will work.

This is because the last slash should represent the separator to dive into the secret value and get a key. So what is happening is that the function is looking for secret onlineMidTakseet with key apiKey while the secret is onlineMidTakseet/test.

We can work on improving this or add a note about this.

@samar-elsayed
Copy link
Author

Hmm ok so it is not related the kv version right?

I hope you can improve it since in my team we need to use / to separate and organize multiple environments secret per application name

@oscerd
Copy link
Contributor

oscerd commented Sep 5, 2024

No, it has nothing to do with engine version.

For sure this won't go in 4.8.0, the Camel release will be during the weekend and there is no time.

@oscerd
Copy link
Contributor

oscerd commented Sep 5, 2024

Thinking about it, it's also hard to implement because if the slash is used to determine where the secret path ends and the after slash is the subkey, you'll never know if you are looking for a full key or a particular field of the secret. Even without investigating, this is not really possible, unless we change the syntax.

@oscerd
Copy link
Contributor

oscerd commented Sep 5, 2024

https://issues.apache.org/jira/browse/CAMEL-21179

@oscerd
Copy link
Contributor

oscerd commented Sep 10, 2024

This is now supported and it will be available in 4.9.0

@samar-elsayed
Copy link
Author

samar-elsayed commented Sep 10, 2024 •
edited
Loading

Great!
Many Thanks @oscerd for your support and effort!

@mgubaidullin mgubaidullin added this to the 4.9.0 milestone Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
4.9.0
Development

No branches or pull requests
3 participants
@mgubaidullin @oscerd @samar-elsayed