[Bug]: OpenSearch Dashboards needs to rebuild core every time (make --component work) #606

dblock · 2021-09-27T14:17:34Z

Describe the bug

OpenSearch dashboards requires to rebuild core every single time to build any of the components, which makes it very slow to debug component build problems.

To reproduce

./build-workflow/build.sh manifests/1.1.0/openseach-dashboards-1.1.0.yml

Succeeds for all components.

./build-workflow/build.sh manifests/1.1.0/openseach-dashboards-1.1.0.yml --component=alertingDashboards

Fails with

+ PLUGIN_NAME=alertingDashboards
+ cp -r ../alertingDashboards/ ../OpenSearch-Dashboards/plugins
cp: cannot create directory '../OpenSearch-Dashboards/plugins': No such file or directory
2021-09-27 14:17:21 ERROR    Error building alertingDashboards, retry with: ./bundle-workflow/build.sh manifests/1.1.0/opensearch-dashboards-1.1.0.yml --component alertingDashboards
Traceback (most recent call last):
  File "./bundle-workflow/src/run_build.py", line 80, in <module>
    sys.exit(main())
  File "./bundle-workflow/src/run_build.py", line 66, in main
    builder.build(target)
  File "/home/ubuntu/source/opensearch-project/opensearch-build/dblock-opensearch-build/bundle-workflow/src/build_workflow/builder.py", line 41, in build
    self.git_repo.execute(build_command)
  File "/home/ubuntu/source/opensearch-project/opensearch-build/dblock-opensearch-build/bundle-workflow/src/git/git_repository.py", line 70, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/home/ubuntu/.pyenv/versions/3.7.11/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/home/ubuntu/source/opensearch-project/opensearch-build/dblock-opensearch-build/bundle-workflow/scripts/default/build.sh -v 1.1.0 -a x64 -s false -o artifacts -d true' returned non-zero exit status 1.

Expected behavior

We could adopt something similar to how OpenSearch builds into ~/.m2 for dependencies with npm local, so that the output from the dashboards core could be published and preserved.

The text was updated successfully, but these errors were encountered:

kavilla · 2021-10-06T22:00:03Z

Subtask: #680

Tengda-He · 2021-10-26T17:17:28Z

Had a offline sync with kavilla, thinking the right solution should be letting plugin owner figure out what modules are required from OSD for plugin build. The current situation is we have many plugins, and each has a different dependency hierarchy with OSD modules. (And some plugin have import module with relative path pointing to OSD src folder). This makes it quite hard to develop a fit-all solution from OSD side. Prefer to hand this work to plugin owner to achieve.

…ilding the OpenSearch-Dashboards component. Also create the load_dashboard_from_cache function for plugin build to reuse OpenSearch-Dashboards source code. This way allows individual plugin to build independently, opensearch-project#606 Signed-off-by: Tengda He <tengh@amazon.com>

kavilla · 2021-11-05T23:57:10Z

We were thinking about to solve this issue we will allow for multiple components to be passed so it builds together

peterzhuamazon · 2021-12-09T18:36:40Z

#1278 provides a way to get it run.
We can try to use this for the fixes.

gaiksaya · 2021-12-09T19:05:28Z

One way might be to use local-npm (similar to maven local). Build dashboards and push to local-npm (one time) and then build the components on top of it. I have not tested this myself but can be one of the option.

bbarani · 2022-10-18T20:33:51Z

@kavilla @tianleh Is this still an open issue?

peterzhuamazon · 2022-10-18T23:45:51Z

Since we already implement the --component on both build and jenkins I think this can be closed.
Thanks.

dblock · 2022-10-19T13:38:41Z

This problem is about developers, not Jenkins, see description of the issue. I don't think it should be closed.

bbarani · 2023-01-30T06:12:54Z

@seanneumann @kavilla This needs to be prioritized to improve the build generation velocity. Can you please prioritize this as soon as possible?

seanneumann · 2023-01-30T19:39:18Z

Hey @bbarani. Taking a look with the team now.

kavilla · 2023-01-30T20:56:49Z

@seanneumann @kavilla This needs to be prioritized to improve the build generation velocity. Can you please prioritize this as soon as possible?

Right now it does re-use the build from OpenSearch Dashboards in the default script:

I don't think this will improve build generation velocity besides if we only just want to build a singular component which based on Peter's comment it is supported on Jenkins. Solving this would allow for developers locally to push out if they wanted to build out --component securityDashboards.

Is there a case that this impacting @bbarani?

bbarani · 2023-01-30T21:51:11Z

@kavilla As far as I know, this is needed for granular build generation process work that is described here. @peterzhuamazon, you mentioned that we can overcome this blocker by Bootstrapping the core, is that a long term solution?

bbarani · 2023-01-31T18:30:44Z

The bootstrapping also causes the core build to fail if any of the plugins fail to build as well

ERROR [bootstrap] failed:
ERROR Error: Command failed with exit code 127: /usr/share/opensearch/.nvm/versions/node/v14.20.1/lib/node_modules/yarn/bin/yarn.js install --non-interactive
          at makeError (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:25150:11)
          at handlePromise (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:24085:26)
          at processTicksAndRejections (internal/process/task_queues.js:95:5)
          at async installInDir (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:23821:3)
          at async Project.installDependencies (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:14780:5)
          at async Object.run (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:9003:11)
          at async runCommand (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:57870:5)
          at async Module.run (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:263:3)
error Command failed with exit code 1.

gaiksaya · 2023-12-18T23:47:39Z

I found a couple of solutions in this article https://medium.com/@debshish.pal/publish-a-npm-package-locally-for-testing-9a00015eb9fd which can be used to publish opensearch-dashboards as a local npm package rather than bootstraping everytime (if I understand if correctly). Please take a look and see if this can be helpful. Also not sure if @Tengda-He is still working on this issue. If not please assign a new assignee. Removing the assignee for now.

Thanks!

gaiksaya · 2024-08-23T19:44:07Z

@kavilla @AMoo-Miki @ashwin-pc Wondering if you have any inputs for this issue?
Thanks!

AMoo-Miki · 2024-10-16T05:45:06Z

I am not sure I understand the problem enough or the historical context of this issue.

dblock added bug Something isn't working opensearch-dashboards enhancement New Enhancement and removed bug Something isn't working labels Sep 27, 2021

dblock changed the title ~~[Bug]: OpenSearch Dashboards needs to rebuild core every time~~ [Bug]: OpenSearch Dashboards needs to rebuild core every time (make --component work) Sep 30, 2021

kavilla added the v1.2.0 OpenSearch 1.2 version label Oct 6, 2021

kavilla mentioned this issue Oct 6, 2021

Make dashboards plugins' build independent #680

Open

peternied assigned kavilla Oct 12, 2021

Tengda-He assigned Tengda-He and unassigned kavilla Oct 14, 2021

Tengda-He mentioned this issue Oct 15, 2021

save OpenSearch Dashboards for plugin to reuse #755

Closed

1 task

Tengda-He mentioned this issue Oct 26, 2021

temp solution for component build not require dashboards core rebuild in advance #812

Closed

Tengda-He mentioned this issue Oct 27, 2021

Copy the OpenSearch-Dashboards src package to cache for plugin build reuse #815

Closed

1 task

peternied removed the v1.2.0 OpenSearch 1.2 version label Nov 16, 2021

peterzhuamazon mentioned this issue Dec 9, 2021

Update dashboards-reports build.sh so it builds with dashboards #1278

Merged

1 task

dblock mentioned this issue Oct 19, 2022

[PROPOSAL] Introduce an OpenSearch-Dashboards SDK opensearch-project/OpenSearch-Dashboards#2608

Open

wbeckler mentioned this issue Jul 18, 2023

[Decouple] Make the build process the same for dev and production opensearch-project/OpenSearch-Dashboards#4582

Open

zelinh mentioned this issue Nov 6, 2023

[Incremental Build] Differentiate the commits changes and finalize rebuild components #4201

Closed

gaiksaya unassigned Tengda-He Dec 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: OpenSearch Dashboards needs to rebuild core every time (make --component work) #606

[Bug]: OpenSearch Dashboards needs to rebuild core every time (make --component work) #606

dblock commented Sep 27, 2021 •

edited by peternied

Loading

kavilla commented Oct 6, 2021

Tengda-He commented Oct 26, 2021 •

edited

Loading

kavilla commented Nov 5, 2021

peterzhuamazon commented Dec 9, 2021

gaiksaya commented Dec 9, 2021 •

edited

Loading

bbarani commented Oct 18, 2022

peterzhuamazon commented Oct 18, 2022

dblock commented Oct 19, 2022 •

edited

Loading

bbarani commented Jan 30, 2023

seanneumann commented Jan 30, 2023

kavilla commented Jan 30, 2023

bbarani commented Jan 30, 2023

bbarani commented Jan 31, 2023

gaiksaya commented Dec 18, 2023

gaiksaya commented Aug 23, 2024

AMoo-Miki commented Oct 16, 2024

[Bug]: OpenSearch Dashboards needs to rebuild core every time (make --component work) #606

[Bug]: OpenSearch Dashboards needs to rebuild core every time (make --component work) #606

Comments

dblock commented Sep 27, 2021 • edited by peternied Loading

Describe the bug

To reproduce

Expected behavior

kavilla commented Oct 6, 2021

Tengda-He commented Oct 26, 2021 • edited Loading

kavilla commented Nov 5, 2021

peterzhuamazon commented Dec 9, 2021

gaiksaya commented Dec 9, 2021 • edited Loading

bbarani commented Oct 18, 2022

peterzhuamazon commented Oct 18, 2022

dblock commented Oct 19, 2022 • edited Loading

bbarani commented Jan 30, 2023

seanneumann commented Jan 30, 2023

kavilla commented Jan 30, 2023

bbarani commented Jan 30, 2023

bbarani commented Jan 31, 2023

gaiksaya commented Dec 18, 2023

gaiksaya commented Aug 23, 2024

AMoo-Miki commented Oct 16, 2024

dblock commented Sep 27, 2021 •

edited by peternied

Loading

Tengda-He commented Oct 26, 2021 •

edited

Loading

gaiksaya commented Dec 9, 2021 •

edited

Loading

dblock commented Oct 19, 2022 •

edited

Loading