Skip to content
This repository has been archived by the owner on Jun 6, 2024. It is now read-only.

export vc/node related metrics from yarn #2289

Merged
merged 6 commits into from
Mar 11, 2019
Merged

export vc/node related metrics from yarn #2289

merged 6 commits into from
Mar 11, 2019

Conversation

xudifsd
Copy link
Member

@xudifsd xudifsd commented Mar 8, 2019

export following metrics:

  • yarn_queue_cpu_available
  • yarn_queue_cpu_total
  • yarn_queue_mem_available
  • yarn_queue_mem_total
  • yarn_queue_gpu_available
  • yarn_queue_gpu_total
  • yarn_queue_running_job
  • yarn_queue_pending_job
  • yarn_queue_running_container
  • yarn_queue_pending_container
  • yarn_node_cpu_total
  • yarn_node_cpu_available
  • yarn_node_mem_total
  • yarn_node_mem_available
  • yarn_node_gpu_total
  • yarn_node_gpu_available

@xudifsd xudifsd requested a review from mzmssg March 8, 2019 07:19
@coveralls
Copy link

coveralls commented Mar 8, 2019 •
edited
Loading

Coverage Status

Coverage remained the same at 52.637% when pulling f0a9cd7 on dixu/yarn-metrics into c885637 on master.

@xudifsd
Copy link
Member Author

xudifsd commented Mar 8, 2019

fixed #2231

@xudifsd xudifsd merged commit 6e0a52a into master Mar 11, 2019
@xudifsd xudifsd deleted the dixu/yarn-metrics branch March 11, 2019 06:38
@xudifsd xudifsd mentioned this pull request Mar 12, 2019
Closed
sunqinzheng added a commit that referenced this pull request Mar 20, 2019
@sunqinzheng
[VS Code] view container (#2301)
021a645 
* add a dashboard in grafana to list all tasks in node (#2197)

* Fix format in issue templates (#2233)

Fix format in issue templates:
- remove trailing spaces
- change chinese colon into english

* Fix auto retries when out of memory. (#1108)

* Distinguish cgroup OOM from dmesg.

* Remove cgroup OOM detection

Make all OOM cause exiting by 5

* Exit 55 when OOM

* Refine homepage for new users (#2155)

Updated first level bullets, to add more content for administrators and users, who is first time touch OpenPAI, or computing platform.

* Fix yarn container failed when docker container exited quickly. (#2256)

* REST server: remove expires in JWT payload of unit test (#2263)

* Deploy: add explicit config field in webportal  plugin (#2251)

* Deploy: add explicit config field in webportal  plugin

* Fix json.dumps

* t

* fix

* Update PLUGINS.md

* Update webportal.md

* alert on unhealthy gpu (#2209)

* Pylon: fix double start query in yarn redirect (#2258)

* Pylon: fix double start query in yarn redirect

* Hide debug info in docker-compose.yaml

* adapt user transfer script to new config (#2266)

* Webportal: add pai-version attribute to <pai-plugin> (#2245)

* Webportal: add pai-version attribute to <pai-plugin>

* Use preprocess to apply window.PAI_VERSION

* set version in layout.html

* Fix ib drivers bug (#2269)

* FIx ib installation script bug (#2271)

* [BUG] Fix hadoop ai build path (#2262)

* fix hadoop ai build bugs

* refine

* Web portal submit job: support init json from sessionStorage. (#2253)

* YARN and HDFS log persistence  (#2244)

* rm log persist

* change log dir to host

* persist nm log to host

* resolve conflict

* persist namenode log

* persist data node log

* add comments

* move log path to common pai storage

* use twisted in yarn-exporter (#2273)

* [Job Debugging] Basic Implement Of Job Debugging. (#2272)

* Refine document for new user to submit job (#2278)

1. add new guidance to submit job for beginners.
2. refine homepage to connect with new guidance.
3. reorganize content of troubleshooting for next refactoring.
4. fix links in md files.

* [Drivers] Fix the issue when installing IB drivers.  (#2275)

* fix can not report zombie process using gpu error (#2279)

* fix external process error

* add debug log

* fix short ID and long ID do not match

* use time based atomic ref to exchange info between threads

* add test case for AtomicRef

* fix bug in file remove (#2288)

* fix hadoop build error (#2296)

* export vc/node related metrics from yarn (#2289)

* 720

* open hdfs explorer in view container
enable tslint rule "ordered-imports"

* add tslint rule for indent

* add home button to hdfs explorer's navigation;
adjust octicon's color

* fix lint error

* [VS Code] Add job list (#2160)

* add job list view to pai extension

* [VS Code] joblist fix (#2185)

* eager load recent jobs when job submitted

* avoid eager getChildren, and let vscode treeview.reveal do it implicitly

* fix lint error
sunqinzheng added a commit that referenced this pull request Mar 28, 2019
@sunqinzheng
[VS Code] view container (#2301)
f1b1535 
* add a dashboard in grafana to list all tasks in node (#2197)

* Fix format in issue templates (#2233)

Fix format in issue templates:
- remove trailing spaces
- change chinese colon into english

* Fix auto retries when out of memory. (#1108)

* Distinguish cgroup OOM from dmesg.

* Remove cgroup OOM detection

Make all OOM cause exiting by 5

* Exit 55 when OOM

* Refine homepage for new users (#2155)

Updated first level bullets, to add more content for administrators and users, who is first time touch OpenPAI, or computing platform.

* Fix yarn container failed when docker container exited quickly. (#2256)

* REST server: remove expires in JWT payload of unit test (#2263)

* Deploy: add explicit config field in webportal  plugin (#2251)

* Deploy: add explicit config field in webportal  plugin

* Fix json.dumps

* t

* fix

* Update PLUGINS.md

* Update webportal.md

* alert on unhealthy gpu (#2209)

* Pylon: fix double start query in yarn redirect (#2258)

* Pylon: fix double start query in yarn redirect

* Hide debug info in docker-compose.yaml

* adapt user transfer script to new config (#2266)

* Webportal: add pai-version attribute to <pai-plugin> (#2245)

* Webportal: add pai-version attribute to <pai-plugin>

* Use preprocess to apply window.PAI_VERSION

* set version in layout.html

* Fix ib drivers bug (#2269)

* FIx ib installation script bug (#2271)

* [BUG] Fix hadoop ai build path (#2262)

* fix hadoop ai build bugs

* refine

* Web portal submit job: support init json from sessionStorage. (#2253)

* YARN and HDFS log persistence  (#2244)

* rm log persist

* change log dir to host

* persist nm log to host

* resolve conflict

* persist namenode log

* persist data node log

* add comments

* move log path to common pai storage

* use twisted in yarn-exporter (#2273)

* [Job Debugging] Basic Implement Of Job Debugging. (#2272)

* Refine document for new user to submit job (#2278)

1. add new guidance to submit job for beginners.
2. refine homepage to connect with new guidance.
3. reorganize content of troubleshooting for next refactoring.
4. fix links in md files.

* [Drivers] Fix the issue when installing IB drivers.  (#2275)

* fix can not report zombie process using gpu error (#2279)

* fix external process error

* add debug log

* fix short ID and long ID do not match

* use time based atomic ref to exchange info between threads

* add test case for AtomicRef

* fix bug in file remove (#2288)

* fix hadoop build error (#2296)

* export vc/node related metrics from yarn (#2289)

* 720

* open hdfs explorer in view container
enable tslint rule "ordered-imports"

* add tslint rule for indent

* add home button to hdfs explorer's navigation;
adjust octicon's color

* fix lint error

* [VS Code] Add job list (#2160)

* add job list view to pai extension

* [VS Code] joblist fix (#2185)

* eager load recent jobs when job submitted

* avoid eager getChildren, and let vscode treeview.reveal do it implicitly

* fix lint error
qfyin pushed a commit that referenced this pull request Apr 1, 2019
@sunqinzheng @qfyin
[VS Code] version 0.2.0 (#2442)
efdca68 
* add installation guide for VS code extension (#2223)

* add installation guide for VS code extension

* [VS Code] view container (#2301)

* add a dashboard in grafana to list all tasks in node (#2197)

* Fix format in issue templates (#2233)

Fix format in issue templates:
- remove trailing spaces
- change chinese colon into english

* Fix auto retries when out of memory. (#1108)

* Distinguish cgroup OOM from dmesg.

* Remove cgroup OOM detection

Make all OOM cause exiting by 5

* Exit 55 when OOM

* Refine homepage for new users (#2155)

Updated first level bullets, to add more content for administrators and users, who is first time touch OpenPAI, or computing platform.

* Fix yarn container failed when docker container exited quickly. (#2256)

* REST server: remove expires in JWT payload of unit test (#2263)

* Deploy: add explicit config field in webportal  plugin (#2251)

* Deploy: add explicit config field in webportal  plugin

* Fix json.dumps

* t

* fix

* Update PLUGINS.md

* Update webportal.md

* alert on unhealthy gpu (#2209)

* Pylon: fix double start query in yarn redirect (#2258)

* Pylon: fix double start query in yarn redirect

* Hide debug info in docker-compose.yaml

* adapt user transfer script to new config (#2266)

* Webportal: add pai-version attribute to <pai-plugin> (#2245)

* Webportal: add pai-version attribute to <pai-plugin>

* Use preprocess to apply window.PAI_VERSION

* set version in layout.html

* Fix ib drivers bug (#2269)

* FIx ib installation script bug (#2271)

* [BUG] Fix hadoop ai build path (#2262)

* fix hadoop ai build bugs

* refine

* Web portal submit job: support init json from sessionStorage. (#2253)

* YARN and HDFS log persistence  (#2244)

* rm log persist

* change log dir to host

* persist nm log to host

* resolve conflict

* persist namenode log

* persist data node log

* add comments

* move log path to common pai storage

* use twisted in yarn-exporter (#2273)

* [Job Debugging] Basic Implement Of Job Debugging. (#2272)

* Refine document for new user to submit job (#2278)

1. add new guidance to submit job for beginners.
2. refine homepage to connect with new guidance.
3. reorganize content of troubleshooting for next refactoring.
4. fix links in md files.

* [Drivers] Fix the issue when installing IB drivers.  (#2275)

* fix can not report zombie process using gpu error (#2279)

* fix external process error

* add debug log

* fix short ID and long ID do not match

* use time based atomic ref to exchange info between threads

* add test case for AtomicRef

* fix bug in file remove (#2288)

* fix hadoop build error (#2296)

* export vc/node related metrics from yarn (#2289)

* 720

* open hdfs explorer in view container
enable tslint rule "ordered-imports"

* add tslint rule for indent

* add home button to hdfs explorer's navigation;
adjust octicon's color

* fix lint error

* [VS Code] Add job list (#2160)

* add job list view to pai extension

* [VS Code] joblist fix (#2185)

* eager load recent jobs when job submitted

* avoid eager getChildren, and let vscode treeview.reveal do it implicitly

* fix lint error

* [VS Code] default to generate jsonc job config file  (#2368)

* 720

* open hdfs explorer in view container
enable tslint rule "ordered-imports"

* add tslint rule for indent

* add home button to hdfs explorer's navigation;
adjust octicon's color

* fix lint error

* [VS Code] Add job list (#2160)

* add job list view to pai extension

* [VS Code] joblist fix (#2185)

* eager load recent jobs when job submitted

* avoid eager getChildren, and let vscode treeview.reveal do it implicitly

* default to generate jsonc job config file

* [VS Code] Refine error messages; Fix Cluster Explorer's bug

* [VS Code] changelog and readme (#2429)

* 539

* 712

* 452

* [VS Code] v0.11 compatible issue (#2457)

* 536

* 600

* [VS Code] fix cluster explorer's right-click menu (#2463)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@mzmssg mzmssg mzmssg approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xudifsd @coveralls @mzmssg