Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A lot of house-keeping messages in logs #1490

Closed
mfojtik opened this issue Mar 30, 2015 · 13 comments
Closed

A lot of house-keeping messages in logs #1490

mfojtik opened this issue Mar 30, 2015 · 13 comments
Labels
component/install kind/bug Categorizes issue or PR as related to a bug. priority/P3

Comments

@mfojtik
Copy link
Contributor

mfojtik commented Mar 30, 2015

Running Vagrant with libvirt driver causes OpenShift to produce a lot of these messages:

I0330 10:07:33.083587   14187 manager.go:550] Added container: "/system.slice/dev-disk-by\\x2did-dm\\x2duuid\\x2dLVM\\x2dNqXeSdtIuAKvYoSNHsosxOgJcYXbCYNmqc9NkGPsumGQoW6wP5rVpIdQdNVif1eS.swap" (aliases: [], namespace: "")
I0330 10:07:33.084012   14187 container.go:185] Start housekeeping for container "/system.slice/dev-disk-by\\x2did-dm\\x2duuid\\x2dLVM\\x2dNqXeSdtIuAKvYoSNHsosxOgJcYXbCYNmqc9NkGPsumGQoW6wP5rVpIdQdNVif1eS.swap"
I0330 10:07:33.084817   14187 manager.go:550] Added container: "/system.slice/dev-disk-by\\x2duuid-31221702\\x2dd950\\x2d4501\\x2d93ca\\x2d1377cf988999.swap" (aliases: [], namespace: "")
I0330 10:07:33.085256   14187 container.go:185] Start housekeeping for container "/system.slice/dev-disk-by\\x2duuid-31221702\\x2dd950\\x2d4501\\x2d93ca\\x2d1377cf988999.swap"
I0330 10:07:33.086031   14187 manager.go:550] Added container: "/system.slice/dev-vg_vagrant-lv_swap.swap" (aliases: [], namespace: "")
I0330 10:07:33.086379   14187 container.go:185] Start housekeeping for container "/system.slice/dev-vg_vagrant-lv_swap.swap"
I0330 10:07:33.087114   14187 manager.go:550] Added container: "/system.slice/fedora-readonly.service" (aliases: [], namespace: "")
I0330 10:07:33.087476   14187 container.go:185] Start housekeeping for container "/system.slice/fedora-readonly.service"
I0330 10:07:33.088230   14187 manager.go:550] Added container: "/system.slice/systemd-journal-flush.service" (aliases: [], namespace: "")
I0330 10:07:33.088594   14187 container.go:185] Start housekeeping for container "/system.slice/systemd-journal-flush.service"
I0330 10:07:33.089344   14187 manager.go:550] Added container: "/system.slice/systemd-sysctl.service" (aliases: [], namespace: "")
I0330 10:07:33.089679   14187 container.go:185] Start housekeeping for container "/system.slice/systemd-sysctl.service"
I0330 10:07:33.090377   14187 manager.go:550] Added container: "/system.slice/systemd-tmpfiles-setup-dev.service" (aliases: [], namespace: "")
I0330 10:07:33.090709   14187 container.go:185] Start housekeeping for container "/system.slice/systemd-tmpfiles-setup-dev.service"
I0330 10:07:33.091381   14187 manager.go:550] Added container: "/system.slice/lvm2-lvmetad.service" (aliases: [], namespace: "")
I0330 10:07:33.091691   14187 container.go:185] Start housekeeping for container "/system.slice/lvm2-lvmetad.service"
I0330 10:07:33.092365   14187 manager.go:550] Added container: "/system.slice/sys-kernel-debug.mount" (aliases: [], namespace: "")
I0330 10:07:33.092698   14187 container.go:185] Start housekeeping for container "/system.slice/sys-kernel-debug.mount"
I0330 10:07:33.093369   14187 manager.go:550] Added container: "/system.slice/system-systemd\\x2dfsck.slice" (aliases: [], namespace: "")
I0330 10:07:33.093684   14187 container.go:185] Start housekeeping for container "/system.slice/system-systemd\\x2dfsck.slice"
I0330 10:07:33.094361   14187 manager.go:550] Added container: "/system.slice/systemd-logind.service" (aliases: [], namespace: "")
I0330 10:07:33.094689   14187 container.go:185] Start housekeeping for container "/system.slice/systemd-logind.service"
I0330 10:07:33.095353   14187 manager.go:550] Added container: "/system.slice/tmp.mount" (aliases: [], namespace: "")

(there is much more)....

Also I can see that pods gets stuck in Pending state and then keep restarting (docker-registry for example).
@mfojtik
Copy link
Contributor Author

mfojtik commented Mar 30, 2015

@smarterclayton @deads2k any clues?

@mfojtik
Copy link
Contributor Author

mfojtik commented Mar 30, 2015

fyi, I'm running up-to-date f21 box in vagrant.... these messages started to appear last week, so I assume something in k8s changed and it is touching things it should not touch...

@mfojtik
Copy link
Contributor Author

mfojtik commented Mar 30, 2015

another weirdness is that right now my 'docker ps' command get stuck... no output, just hangs...

@soltysh
Copy link
Contributor

soltysh commented Mar 30, 2015

I'm seeing those messages as well (I'm using f21 vbox vagrant) whenever starting a pod (either through builds or deployments). But once pods are all started up the logs are gone. Additionally I can still run any command eg. docker ps.

@mfojtik
Copy link
Contributor Author

mfojtik commented Mar 30, 2015

In libvirt I can see only 'pod' container running, any other container is ran and then stopped and restarted... There is nothing in logs...

@smarterclayton
Copy link
Contributor

These are from cAdvisor, you'll want to engage Victor or Vish upstream about it.

@mfojtik
Copy link
Contributor Author

mfojtik commented Mar 30, 2015

@smarterclayton can we make them more silent? ;-)

@deads2k
Copy link
Contributor

deads2k commented Mar 30, 2015

It came in with the last rebase. Kube had it for a while, but our kubelet start up has drifted from upstream.

@smarterclayton
Copy link
Contributor

I believe some of these have been squelched by my pull to cAdvisor last week.

@mfojtik
Copy link
Contributor Author

mfojtik commented Mar 31, 2015

@smarterclayton what is interesting is that this message cause the openshift to hang:

I0331 12:34:21.699526   26346 container.go:165] Raising housekeeping interval for "/system.slice/gssproxy.service" to 16s
I0331 12:34:21.700068   26346 container.go:165] Raising housekeeping interval for "/system.slice/crond.service" to 4s
W0331 12:34:33.119486   26346 handler.go:529] Error while processing event ("/sys/fs/cgroup/memory/system.slice/NetworkManager-dispatcher.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/system.slice/NetworkManager-dispatcher.service: no such file or directory
W0331 12:36:36.117301   26346 handler.go:529] Error while processing event ("/sys/fs/cgroup/memory/system.slice/NetworkManager-dispatcher.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/system.slice/NetworkManager-dispatcher.service: no such file or directory

(there is no log output after this point)

The osc:

[root@openshiftdev origin]# osc get -n secret780444235 builds
F0331 12:36:53.221940    5711 get.go:67] Client error processing command: Get https://192.168.121.47:51538/osapi/v1beta1/builds?namespace=secret780444235: net/http: TLS handshake timeout

@smarterclayton
Copy link
Contributor

What does CTRL+\ tell you?

On Mar 31, 2015, at 8:38 AM, Michal Fojtik notifications@github.com wrote:

@smarterclayton what is interesting is that this message cause the openshift to hang:

I0331 12:34:21.699526 26346 container.go:165] Raising housekeeping interval for "/system.slice/gssproxy.service" to 16s
I0331 12:34:21.700068 26346 container.go:165] Raising housekeeping interval for "/system.slice/crond.service" to 4s
W0331 12:34:33.119486 26346 handler.go:529] Error while processing event ("/sys/fs/cgroup/memory/system.slice/NetworkManager-dispatcher.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/system.slice/NetworkManager-dispatcher.service: no such file or directory
W0331 12:36:36.117301 26346 handler.go:529] Error while processing event ("/sys/fs/cgroup/memory/system.slice/NetworkManager-dispatcher.service": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/system.slice/NetworkManager-dispatcher.service: no such file or directory
(there is no log output after this point)

The osc:

[root@openshiftdev origin]# osc get -n secret780444235 builds
F0331 12:36:53.221940 5711 get.go:67] Client error processing command: Get https://192.168.121.47:51538/osapi/v1beta1/builds?namespace=secret780444235: net/http: TLS handshake timeout

—
Reply to this email directly or view it on GitHub.

@mfojtik
Copy link
Contributor Author

mfojtik commented Mar 31, 2015

@smarterclayton I killed it :-( but next time I will provide that output

@danmcp danmcp added kind/bug Categorizes issue or PR as related to a bug. priority/P3 component/install labels May 11, 2015
@smarterclayton
Copy link
Contributor

These have all been muted.

jboyd01 pushed a commit to jboyd01/origin that referenced this issue Nov 3, 2017
Squashed 'cmd/service-catalog/go/src/github.com/kubernetes-incubator/…
23a8ca3 
…service-catalog/' changes from 892b0368f0..3064247d05

3064247d05 origin build: add origin tooling
48ecff1 Chart changes for 0.1.2 (openshift#1527)
8727247 Fix change validator to look up serviceclass properly (openshift#1518)
b0b138b Broker Reconciliation occurs too frequently (openshift#1514)
cc816eb When a SI is unbindable, mark the binding request failed. (openshift#1522)
41290e9 Clarify semantics around RelistDuration (openshift#1516)
6bcb593 Send Context with UpdateInstanceRequest (openshift#1517)
c3b72d8 Enforce stricly increasing broker relistRequests (openshift#1515)
01d81e5 Follow up from openshift#1450 and openshift#1444 (openshift#1504)
5e77882 Retry failed deprovision requests (openshift#1505)
f6c891d Allow updates of instances that failed a previous update. (openshift#1502)
5107086 Use Plan ID from ExternalProperties when deprovisioning instance (openshift#1501)
d897c60 Adding message builder for expected event strings  (openshift#1465)
9f6152e update osb client and freeze gorilla context (openshift#1496)
c63277e Add unit tests verifying deleting a resource with an on-going operation or in orphan mitigation. (openshift#1490)
4ecca16 Pretty logging for controller_instance.go (openshift#1472)
c232db4 Register qemu statics when building non-amd64 images (openshift#1494)
0e54c57 fix 'Spring Cloud Services -> Configuring with Vault' link in docs (openshift#1495)
REVERT: 892b0368f0 origin build: add origin tooling

git-subtree-dir: cmd/service-catalog/go/src/github.com/kubernetes-incubator/service-catalog
git-subtree-split: 3064247d0544aa43f976d93caea0a11771434ef7
jpeeler pushed a commit to jpeeler/origin that referenced this issue Feb 1, 2018
@staebler @kibbles-n-bytes
Add unit tests verifying deleting a resource with an on-going operati…
c63277e 
…on or in orphan mitigation. (openshift#1490)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/install kind/bug Categorizes issue or PR as related to a bug. priority/P3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mfojtik @danmcp @soltysh @smarterclayton @deads2k