Sample stream pods deployed on SCDF do not start up on Openshift

[yookore]

I have been able to successfully deploy SCDF on Openshift (Release 1.2.0) using the patch template supplied.
However, running the example ticktock stream does not deploy correctly on OS. I see the pods being created but the deployment never completes. The logs do not seem to tell me much but here is a log output from the pod container as it being created....

2017-09-26 14:22:17.370 INFO 1 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at: http://localhost:8888 2017-09-26 14:22:18.676 WARN 1 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/time-source/default": Connection refused; nested exception is java.net.ConnectException: Connection refused 2017-09-26 14:22:18.689 INFO 1 --- [ main] .c.s.a.t.s.r.TimeSourceRabbitApplication : No active profile set, falling back to default profiles: default 2017-09-26 14:22:18.869 INFO 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@7d0587f1: startup date [Tue Sep 26 14:22:18 GMT 2017]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1edf1c96 2017-09-26 14:22:27.098 INFO 1 --- [ main] o.s.i.config.IntegrationRegistrar : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created. 2017-09-26 14:22:32.273 INFO 1 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=e55f14dd-8408-3458-9724-3b5b1fc018c0 2017-09-26 14:22:32.489 INFO 1 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created. 2017-09-26 14:22:32.503 INFO 1 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created. 2017-09-26 14:22:33.868 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.amqp.rabbit.annotation.RabbitBootstrapConfiguration' of type [org.springframework.amqp.rabbit.annotation.RabbitBootstrapConfiguration$$EnhancerBySpringCGLIB$$901c9cb8] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:34.409 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.stream.config.BindingServiceConfiguration$PostProcessorConfiguration' of type [org.springframework.cloud.stream.config.BindingServiceConfiguration$PostProcessorConfiguration$$EnhancerBySpringCGLIB$$8b787c8f] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:35.464 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationGlobalProperties' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:35.468 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationGlobalProperties' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:35.793 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$e5cae187] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:35.889 INFO 1 --- [ main] o.s.aop.framework.CglibAopProxy : Unable to proxy method [protected final org.springframework.core.env.Environment org.springframework.boot.actuate.endpoint.AbstractEndpoint.getEnvironment()] because it is final: All calls to this method via a proxy will NOT be routed to the target instance. 2017-09-26 14:22:36.956 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperConfiguration$$EnhancerBySpringCGLIB$$25445e1b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:37.196 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.stream.config.SpelExpressionConverterConfiguration' of type [org.springframework.cloud.stream.config.SpelExpressionConverterConfiguration$$EnhancerBySpringCGLIB$$c9ed367f] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:37.353 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'spelConverter' of type [org.springframework.cloud.stream.config.SpelExpressionConverterConfiguration$SpelConverter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:37.469 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jackson-org.springframework.boot.autoconfigure.jackson.JacksonProperties' of type [org.springframework.boot.autoconfigure.jackson.JacksonProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:37.557 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$Jackson2ObjectMapperBuilderCustomizerConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$Jackson2ObjectMapperBuilderCustomizerConfiguration$$EnhancerBySpringCGLIB$$92a80623] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:37.684 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'standardJacksonObjectMapperBuilderCustomizer' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$Jackson2ObjectMapperBuilderCustomizerConfiguration$StandardJackson2ObjectMapperBuilderCustomizer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:37.698 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperBuilderConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperBuilderConfiguration$$EnhancerBySpringCGLIB$$d7912034] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:37.793 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$$EnhancerBySpringCGLIB$$74ffbaa] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:38.488 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.stream.metrics.config.BinderMetricsAutoConfiguration' of type [org.springframework.cloud.stream.metrics.config.BinderMetricsAutoConfiguration$$EnhancerBySpringCGLIB$$49db1830] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:38.618 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'metricJsonSerializer' of type [org.springframework.cloud.stream.metrics.config.MetricJsonSerializer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:38.661 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'jsonComponentModule' of type [org.springframework.boot.jackson.JsonComponentModule] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:38.741 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'jacksonObjectMapperBuilder' of type [org.springframework.http.converter.json.Jackson2ObjectMapperBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:39.182 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'jacksonObjectMapper' of type [com.fasterxml.jackson.databind.ObjectMapper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:39.426 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationConversionService' of type [org.springframework.integration.context.CustomConversionServiceFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:39.453 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationConversionService' of type [org.springframework.core.convert.support.DefaultConversionService] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:39.466 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'datatypeChannelMessageConverter' of type [org.springframework.integration.support.converter.DefaultDatatypeChannelMessageConverter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:39.473 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.stream.config.BindingServiceConfiguration' of type [org.springframework.cloud.stream.config.BindingServiceConfiguration$$EnhancerBySpringCGLIB$$efb270d7] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:40.088 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.cloud.stream-org.springframework.cloud.stream.config.BindingServiceProperties' of type [org.springframework.cloud.stream.config.BindingServiceProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:40.316 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'compositeMessageConverterFactory' of type [org.springframework.cloud.stream.converter.CompositeMessageConverterFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:40.340 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'messageConverterConfigurer' of type [org.springframework.cloud.stream.binding.MessageConverterConfigurer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:40.353 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'compositeMessageChannelConfigurer' of type [org.springframework.cloud.stream.binding.CompositeMessageChannelConfigurer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:40.366 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'channelFactory' of type [org.springframework.cloud.stream.binding.SubscribableChannelBindingTargetFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:40.506 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.stream.metrics.config.Emitter' of type [org.springframework.cloud.stream.binding.BindableProxyFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:40.565 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.stream.metrics.config.Emitter' of type [com.sun.proxy.$Proxy104] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:40.690 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.actuate.autoconfigure.MetricExportAutoConfiguration$MetricExportPropertiesConfiguration' of type [org.springframework.boot.actuate.autoconfigure.MetricExportAutoConfiguration$MetricExportPropertiesConfiguration$$EnhancerBySpringCGLIB$$b9ab0142] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:40.912 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.metrics.export-org.springframework.boot.actuate.metrics.export.MetricExportProperties' of type [org.springframework.boot.actuate.metrics.export.MetricExportProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:41.083 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.cloud.stream.metrics-org.springframework.cloud.stream.metrics.ApplicationMetricsProperties' of type [org.springframework.cloud.stream.metrics.ApplicationMetricsProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2017-09-26 14:22:44.674 INFO 1 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 8080 (http) 2017-09-26 14:22:44.816 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service Tomcat 2017-09-26 14:22:44.818 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.11 2017-09-26 14:22:46.098 INFO 1 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2017-09-26 14:22:46.100 INFO 1 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 27231 ms 2017-09-26 14:22:50.792 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'metricsFilter' to: [/*] 2017-09-26 14:22:50.793 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'characterEncodingFilter' to: [/*] 2017-09-26 14:22:50.794 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'hiddenHttpMethodFilter' to: [/*] 2017-09-26 14:22:50.794 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'httpPutFormContentFilter' to: [/*] 2017-09-26 14:22:50.853 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'requestContextFilter' to: [/*] 2017-09-26 14:22:50.894 INFO 1 --- [ost-startStop-1] .s.DelegatingFilterProxyRegistrationBean : Mapping filter: 'springSecurityFilterChain' to: [/*] 2017-09-26 14:22:50.895 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'webRequestLoggingFilter' to: [/*] 2017-09-26 14:22:50.896 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'applicationContextIdFilter' to: [/*] 2017-09-26 14:22:50.896 INFO 1 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean : Mapping servlet: 'dispatcherServlet' to [/] 2017-09-26 14:22:54.274 INFO 1 --- [ main] o.s.c.s.a.trigger.TriggerConfiguration : Trigger type: org.springframework.scheduling.support.PeriodicTrigger@90a5 2017-09-26 14:22:55.292 INFO 1 --- [ main] o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService 'taskScheduler' 2017-09-26 14:23:00.270 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@7d0587f1: startup date [Tue Sep 26 14:22:18 GMT 2017]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1edf1c96 2017-09-26 14:23:01.966 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), 2017-09-26 14:23:01.973 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest) 2017-09-26 14:23:02.806 INFO 1 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler] 2017-09-26 14:23:02.809 INFO 1 --- [ main] o.s.w.s.handler.SimpleUrlHandlerMapping : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]

That's where is always stops...
Any ideas what the issues could be. I am running all of this on a OS Origin single master, multiple node setup.
The versions of Openshift and Kubernetes are
Version

OpenShift Master:
v3.6.0+c4dd4cf
Kubernetes Master:
v1.6.1+5115d708d7

[donovanmuller]

Hi, thanks for the feedback.

Based on previous experience, if the app just "stops" it's generally due to:

• The app is taking too long to startup and the liveness/readiness probes will fail and the Pod will be killed and another started.
• The app is reaching the memory limit and being OOMKilled by OpenShift and another started.

In both cases this should be visible in the events for the Pod.
Could you please provide the events for the affected Pods?

[donovanmuller]

Closing due to inactivity.

[rwenz3l]

This seems to be an issue bound to scdf or kubernetes since I'm getting the same result in rancher. No idea how to fix it - just want to state that this is not fixed.

[donovanmuller]

Did you try increasing the resource limits or bypassing them completely?
I.e. as per this section in the Kubernetes reference documentation

Try passing the following deployment properties:

• to increase the resource limits:

deployer.<app>.kubernetes.limits.cpu=2000m
deployer.<app>.kubernetes.limits.memory=1024Mi

• to bypass the limits:

deployer.<app>.kubernetes.limits.cpu=0m
deployer.<app>.kubernetes.limits.memory=0Mi

[rwenz3l]

Yes that is indeed what helped - I found the Solution in another GitHub Issue. For simple apps like 'time' and 'log' a cpu value of 1000 is enough as well as a memory value of 1024.

Thanks for pointing to the docs for the limits. I think those need to be set in spring cloud dataflow directly to circumvent this issue in all deployments.

[donovanmuller]

I think those need to be set in spring cloud dataflow directly to circumvent this issue in all deployments.

Yes, you could add the limits/requests to the Config Map which would give you better defaults.