bit export is extremely slow

[davidfirst]

Edit: If you use node >=14.1.0, as a workaround, downgrade your node version. More details in #2844.

Describe the bug

This happens when a workspace has lots of components and/or each component has lots of versions. Especially when exporting a dependency that has lots of dependent components.
The process of the "auto-tag" makes sure that all dependents are tagged and then when running "bit export" it exports them as well.
Currently, the export process in some scenarios may take a while to complete.

Steps to Reproduce

1. you'll need a large amount of data in the workspace.
2. run "bit export"

Expected Behavior

Exporting components should be a quick process.

Specifications

• Bit version: 14.7.3
• Node version:
• npm / yarn version:
• Platform:
• Bit compiler (include version):
• Bit tester (include version):

Additional context

There are two issues with the current implementation of "bit export", which need to be addressed.

1. Bit sends lots of data to the server that is already on the server. The process needs to be more efficient, check what's missing and send only the diff.
2. The transfer rate seems to be very slow (about 100KB per second).

[ranm8]

@davidfirst @GiladShoham I know we have done a lot of progress on this. Can you update on our status here or what is left to improve?

[davidfirst]

@ranm8 , sure.
According to what I wrote in the description:

There are two issues with the current implementation of "bit export", which need to be addressed.

1. Bit sends lots of data to the server that is already on the server. The process needs to be more efficient, check what's missing and send only the diff.
2. The transfer rate seems to be very slow (about 100KB per second).

The first issue has been fixed and it's part of the latest version.
The second issue is in-progress. I found the root cause, opened a PR but didn't finish it.

[blocknomad]

@davidfirst Is there any workaround for the second issue or an estimation for when we are going to have a more performant version of this command available?

Currently, I have to export the components manually, one by one, because the bulk export seems to never finish and it has slowed down quite a lot our productivity using the tool.

[davidfirst]

@yurijean , during the investigating of this issue back then, I found that components compiled with typescript compiler on Windows were extremely heavy due to a bug that added the entire node_modules directory into the component objects.
If this is the case, and you can start fresh, meaning, you can delete the scopes and re-create the components, that would be the best solution.
Other than that, I don't have an estimation at the moment of fixing the second issue, it's not as easy as I thought it would be.

[blocknomad]

@davidfirst I'm on Linux but the same issue happens. I'm using TypeScript, however, I don't compile them using Bit, but I export them in this format. I have around 60 components in my local and starting fresh doesn't solve my issue. I tried turning them into JavaScript and exporting them this way, but still no success.

Unfortunately, I'll have to look for another tool to solve my distribution problem since I can't keep Bit working when my collection grows bigger than something around 20 components 😞 would Lerna be a good fit for distributing front end components? Do you have any suggestions of other tools that I can use meanwhile this issue isn't solved?

[davidfirst]

@yurijean , doesn't sound normal. How long does it take for you to export 60 components? If you can add me as a collaborator (david@bit.dev) I can take a look.

[davidfirst]

@yurijean , not sure whether you looked into the linked task: #2844 , but if your node version is >=14.1.0, please try to use an older version.

[blocknomad]

@davidfirst we downgraded Node to 12.0 and it worked! Problem solved! Thank you 😄

[itaymendel]

export performance boost in v15

We'll start rolling out v15 of Bit this week, please reach out to me privately on the public slack community for a sneak peek and getting early access to it.