Plugin does not open parquet files on Mac M1 #83

akelsch · 2023-07-28T15:25:47Z

~~our team has encountered an issue with vscode-parquet-viewer after we have migrated our Apache Spark platform from Azure HDInsight to Azure Databricks.~~ Turns out we cannot open parquet files with your plugin anymore. The tab in VSCode will be loading forever.

I did attach a sample file to this issue. Its contents look like this (converted using a different plugin):

{
  "hello": "abc",
  "world": "123"
}

I would be really grateful if you could fix this issue as your plugin was the go-to choice to work with parquet files in our team.

Thanks!

Edit: Looks like this is unrelated to Databricks, similar to #81 but only happening on Mac M1

part-00000-tid-8369997251915174921-9ae69399-852b-4480-8335-60e65202bcc8-36-1-c000.snappy.parquet.zip

rafaelmarino · 2023-07-28T18:17:10Z

I've been having issues with the extension since yesterday also. The issue is the same for me as in other reports: "the file takes forever to load". I don't know if it eventually loads or not, since I just close it if it doesn't load in a few seconds.

I'm using pandas to check out my files as a workaround, but this extension is very much missed :)

@akelsch, does it make sense to specify how to write your parquet files? I would have thought once they are .parquet, how you wrote them is of no relevance.

I'm developing on:

M1 pro 13.4.1
VSCode 1.80.2

Thanks @dvirtz for your work on this great extension.

akelsch · 2023-07-28T21:48:11Z

Oh, so the issue is not caused by Databricks but rather appeared at the same time as our migration (what a coincidence).

I am also on Mac M1 and downgrading the plugin version to 2.2.2 does actually let me open above parquet file. 2.3.0 seems to be the first breaking version for me with 2.3.2 not working too.

Looks like some form of regression on Mac M1 only though because it is working on my Windows machine.

Edit: Logs from Mac machine:

2023-07-29 00:02:49.433 [error] Activating extension dvirtz.parquet-viewer failed due to an error:
2023-07-29 00:02:49.433 [error] Error: Failed to find binding for arrow-parquet-reader
	at loadBinding (/Users/akelsch/.vscode/extensions/dvirtz.parquet-viewer-2.3.2/node_modules/pkg-prebuilds/bindings.js:95:27)
	at Object.<anonymous> (/Users/akelsch/.vscode/extensions/dvirtz.parquet-viewer-2.3.2/node_modules/parquet-reader/dist/index.js:34:51)
	at u._compile (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:1271)
	...

I am using the default "parquets" backend.

rafaelmarino · 2023-07-28T23:22:50Z

Nice find. Exactly the same behavior I'm experiencing: 2.2.2 is the last working version.

Is this arrow not playing nice with M1 chips perhaps?

PS: Anecdotal but perhaps useful, this week in a python project I couldn't install arrow8.0 in a python3.7 venv (ERROR: Failed building wheel for pyarrow), so it looks like arrow-m1 are not that compatible out of the box.

space7panda · 2023-07-29T21:25:54Z

Also having similar issues:

2023-07-30 00:20:05.383 [info] Extension host with pid 34973 started
2023-07-30 00:20:05.428 [info] ExtensionService#_doActivateExtension vscode.git-base, startup: true, activationEvent: '*', root cause: vscode.git
2023-07-30 00:20:05.432 [info] ExtensionService#_doActivateExtension christian-kohler.path-intellisense, startup: true, activationEvent: '*'
2023-07-30 00:20:05.476 [info] ExtensionService#_doActivateExtension formulahendry.auto-rename-tag, startup: true, activationEvent: '*'
2023-07-30 00:20:05.483 [info] ExtensionService#_doActivateExtension mhutchie.git-graph, startup: true, activationEvent: '*'
2023-07-30 00:20:05.498 [info] ExtensionService#_doActivateExtension nhoizey.gremlins, startup: true, activationEvent: '*'
2023-07-30 00:20:05.596 [info] ExtensionService#_doActivateExtension vscode.git, startup: true, activationEvent: '*'
2023-07-30 00:20:05.608 [info] ExtensionService#_doActivateExtension vscode.github, startup: true, activationEvent: '*'
2023-07-30 00:20:05.618 [info] ExtensionService#_doActivateExtension vscode.github-authentication, startup: false, activationEvent: 'onAuthenticationRequest:github'
2023-07-30 00:20:05.668 [info] ExtensionService#_doActivateExtension vscode.microsoft-authentication, startup: false, activationEvent: 'onAuthenticationRequest:microsoft'
2023-07-30 00:20:05.777 [info] Eager extensions activated
2023-07-30 00:20:05.779 [info] ExtensionService#_doActivateExtension vscode.debug-auto-launch, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:05.780 [info] ExtensionService#_doActivateExtension vscode.merge-conflict, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:05.784 [info] ExtensionService#_doActivateExtension aaron-bond.better-comments, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:05.789 [info] ExtensionService#_doActivateExtension dbaeumer.vscode-eslint, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:05.795 [info] ExtensionService#_doActivateExtension esbenp.prettier-vscode, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:05.817 [info] ExtensionService#_doActivateExtension formulahendry.code-runner, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:05.869 [info] ExtensionService#_doActivateExtension GitHub.copilot, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:06.084 [info] ExtensionService#_doActivateExtension Gruntfuggly.todo-tree, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:06.128 [info] ExtensionService#_doActivateExtension usernamehw.errorlens, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:06.133 [info] ExtensionService#_doActivateExtension vadimcn.vscode-lldb, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:06.139 [info] ExtensionService#_doActivateExtension vscode-icons-team.vscode-icons, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:06.162 [info] ExtensionService#_doActivateExtension withfig.fig, startup: false, activationEvent: 'onStartupFinished'
2023-07-30 00:20:06.618 [info] ExtensionService#_doActivateExtension vscode.json-language-features, startup: false, activationEvent: 'onLanguage:jsonc'
2023-07-30 00:20:06.631 [info] ExtensionService#_doActivateExtension vscode.typescript-language-features, startup: false, activationEvent: 'onLanguage:jsonc'
2023-07-30 00:20:06.644 [info] ExtensionService#_doActivateExtension vscode.emmet, startup: false, activationEvent: 'onLanguage'
2023-07-30 00:20:24.865 [info] ExtensionService#_doActivateExtension dvirtz.parquet-viewer, startup: false, activationEvent: 'onCustomEditor:parquetViewer.parquetViewer'
2023-07-30 00:20:25.037 [error] Activating extension dvirtz.parquet-viewer failed due to an error:
2023-07-30 00:20:25.037 [error] Error: Failed to find binding for arrow-parquet-reader
    at loadBinding (/Users/space7panda/.vscode/extensions/dvirtz.parquet-viewer-2.3.2/node_modules/pkg-prebuilds/bindings.js:95:27)
    at Object.<anonymous> (/Users/space7panda/.vscode/extensions/dvirtz.parquet-viewer-2.3.2/node_modules/parquet-reader/dist/index.js:34:51)
    at Module.u._compile (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:1271)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1243:10)
    at Module.load (node:internal/modules/cjs/loader:1058:32)
    at Module._load (node:internal/modules/cjs/loader:893:12)
    at Function.f._load (node:electron/js2c/asar_bundle:2:13330)
    at Function.l._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:28070)
    at Function.p._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:25404)
    at Function.u._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:91:22191)
    at Module.apply (node:internal/modules/cjs/loader:1082:19)
    at Module.require (/Users/space7panda/.vscode/extensions/github.copilot-1.99.289/node_modules/diagnostic-channel/dist/src/patchRequire.js:16:46)
    at g (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:647)
    at Object.<anonymous> (/Users/space7panda/.vscode/extensions/dvirtz.parquet-viewer-2.3.2/dist/src/arrow-backend.js:27:26)
    at Module.u._compile (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:1271)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1243:10)
    at Module.load (node:internal/modules/cjs/loader:1058:32)
    at Module._load (node:internal/modules/cjs/loader:893:12)
    at Function.f._load (node:electron/js2c/asar_bundle:2:13330)
    at Function.l._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:28070)
    at Function.p._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:25404)
    at Function.u._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:91:22191)
    at Module.apply (node:internal/modules/cjs/loader:1082:19)
    at Module.require (/Users/space7panda/.vscode/extensions/github.copilot-1.99.289/node_modules/diagnostic-channel/dist/src/patchRequire.js:16:46)
    at g (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:647)
    at Object.<anonymous> (/Users/space7panda/.vscode/extensions/dvirtz.parquet-viewer-2.3.2/dist/src/parquet-document.js:45:25)
    at Module.u._compile (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:1271)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1243:10)
    at Module.load (node:internal/modules/cjs/loader:1058:32)
    at Module._load (node:internal/modules/cjs/loader:893:12)
    at Function.f._load (node:electron/js2c/asar_bundle:2:13330)
    at Function.l._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:28070)
    at Function.p._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:25404)
    at Function.u._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:91:22191)
    at Module.apply (node:internal/modules/cjs/loader:1082:19)
    at Module.require (/Users/space7panda/.vscode/extensions/github.copilot-1.99.289/node_modules/diagnostic-channel/dist/src/patchRequire.js:16:46)
    at g (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:647)
    at Object.<anonymous> (/Users/space7panda/.vscode/extensions/dvirtz.parquet-viewer-2.3.2/dist/src/parquet-document-provider.js:40:44)
    at Module.u._compile (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:1271)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1243:10)
    at Module.load (node:internal/modules/cjs/loader:1058:32)
    at Module._load (node:internal/modules/cjs/loader:893:12)
    at Function.f._load (node:electron/js2c/asar_bundle:2:13330)
    at Function.l._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:28070)
    at Function.p._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:25404)
    at Function.u._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:91:22191)
    at Module.apply (node:internal/modules/cjs/loader:1082:19)
    at Module.require (/Users/space7panda/.vscode/extensions/github.copilot-1.99.289/node_modules/diagnostic-channel/dist/src/patchRequire.js:16:46)
    at g (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:647)
    at Object.<anonymous> (/Users/space7panda/.vscode/extensions/dvirtz.parquet-viewer-2.3.2/dist/src/parquet-editor-provider.js:40:37)
    at Module.u._compile (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:1271)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1243:10)
    at Module.load (node:internal/modules/cjs/loader:1058:32)
    at Module._load (node:internal/modules/cjs/loader:893:12)
    at Function.f._load (node:electron/js2c/asar_bundle:2:13330)
    at Function.l._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:28070)
    at Function.p._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:25404)
    at Function.u._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:91:22191)
    at Module.apply (node:internal/modules/cjs/loader:1082:19)
    at Module.require (/Users/space7panda/.vscode/extensions/github.copilot-1.99.289/node_modules/diagnostic-channel/dist/src/patchRequire.js:16:46)
    at g (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:647)
    at Object.<anonymous> (/Users/space7panda/.vscode/extensions/dvirtz.parquet-viewer-2.3.2/dist/src/extension.js:39:35)
    at Module.u._compile (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:4:1271)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1243:10)
    at Module.load (node:internal/modules/cjs/loader:1058:32)
    at Module._load (node:internal/modules/cjs/loader:893:12)
    at Function.f._load (node:electron/js2c/asar_bundle:2:13330)
    at Function.l._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:28070)
    at Function.p._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:25404)
    at Function.u._load (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:91:22191)
    at Module.apply (node:internal/modules/cjs/loader:1082:19)
    at Module.require (/Users/space7panda/.vscode/extensions/github.copilot-1.99.289/node_modules/diagnostic-channel/dist/src/patchRequire.js:16:46)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Function.i [as __$__nodeRequire] (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js:5:98)
    at i.rb (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:127:26583)
    at async Promise.all (index 0)

space7panda · 2023-07-29T21:56:51Z

Also having similar issues:

2023-07-30 00:20:05.383 [info] Extension host with pid 34973 started
2023-07-30 00:20:05.428 [info] ExtensionService#_doActivateExtension vscode.git-base, startup: true, activationEvent: '*', root cause: vscode.git

was playing around with ChatGPT a bit, updated Python, Pip and installed PyArrow, doesn't seem to help with error aswell...

Python 3.11.4
pip 23.2.1
pyarrow.__version__ 12.0.1

space7panda · 2023-07-29T22:17:24Z

mkay i got some slight progress,
i cloned this repo

git clone ...
brew install pipenv
npm i
npm run test

it kind worked for a few sec in tests, I'm gonna dig a bit more tomorrow

error will only be reported when using arrow backend Fixes #83

## [2.3.3](v2.3.2...v2.3.3) (2023-07-30) ### Bug Fixes * import arrow dynamically to avoid failed loading ([2774fe1](2774fe1)), closes [#83](#83) ### General maintenance * don't spellcheck changelog ([198367b](198367b)) ### Refactoring * factor cancellation to base class ([1e5a261](1e5a261))

dvirtz · 2023-07-30T17:26:21Z

🎉 This issue has been resolved in version 2.3.3 🎉

The release is available on:

Your semantic-release bot 📦🚀

akelsch changed the title ~~Plugin does not open parquet files written by Spark on Databricks~~ Plugin does not open parquet files on Mac M1 Jul 28, 2023

dvirtz self-assigned this Jul 30, 2023

dvirtz added a commit that referenced this issue Jul 30, 2023

fix: import arrow dynamically to avoid failed loading

022daaf

error will only be reported when using arrow backend Fixes #83

dvirtz mentioned this issue Jul 30, 2023

fix: import arrow dynamically to avoid failed loading #84

Merged

dvirtz added a commit that referenced this issue Jul 30, 2023

fix: import arrow dynamically to avoid failed loading

a5d301b

error will only be reported when using arrow backend Fixes #83

dvirtz added a commit that referenced this issue Jul 30, 2023

fix: import arrow dynamically to avoid failed loading

af96274

error will only be reported when using arrow backend Fixes #83

dvirtz added a commit that referenced this issue Jul 30, 2023

fix: import arrow dynamically to avoid failed loading

2dc7a2b

error will only be reported when using arrow backend Fixes #83

dvirtz closed this as completed in #84 Jul 30, 2023

dvirtz added a commit that referenced this issue Jul 30, 2023

fix: import arrow dynamically to avoid failed loading

2774fe1

error will only be reported when using arrow backend Fixes #83

dvirtz added the released label Jul 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plugin does not open parquet files on Mac M1 #83

Plugin does not open parquet files on Mac M1 #83

akelsch commented Jul 28, 2023 •

edited

Loading

rafaelmarino commented Jul 28, 2023

akelsch commented Jul 28, 2023 •

edited

Loading

rafaelmarino commented Jul 28, 2023

space7panda commented Jul 29, 2023

space7panda commented Jul 29, 2023

space7panda commented Jul 29, 2023

dvirtz commented Jul 30, 2023

Plugin does not open parquet files on Mac M1 #83

Plugin does not open parquet files on Mac M1 #83

Comments

akelsch commented Jul 28, 2023 • edited Loading

rafaelmarino commented Jul 28, 2023

akelsch commented Jul 28, 2023 • edited Loading

rafaelmarino commented Jul 28, 2023

space7panda commented Jul 29, 2023

space7panda commented Jul 29, 2023

space7panda commented Jul 29, 2023

dvirtz commented Jul 30, 2023

akelsch commented Jul 28, 2023 •

edited

Loading

akelsch commented Jul 28, 2023 •

edited

Loading