Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider alternate (browser friendly) buffer #578

Closed
rolanday opened this issue Feb 6, 2023 · 15 comments · Fixed by #633
Closed

Consider alternate (browser friendly) buffer #578

rolanday opened this issue Feb 6, 2023 · 15 comments · Fixed by #633
Labels
enhancement Add new functionality help wanted

Comments

@rolanday
Copy link

rolanday commented Feb 6, 2023

Hello! :-) I'd love to be able to use this module in my SvelteKit app, but node:buffer dep is not something I've been able to workaround via Vite polyfill (same issue for other frameworks like Nextjs as well).

Have or can you consider alt buffer impl that is browser friendly? Or, any tips on configuring for use in SvelteKit, Vue, Nextjs, etc?

Thanks!

@sindresorhus
Copy link
Owner

Ideally, all code should be using Uint8Array instead of Buffer, but there's just too much Node.js code that depends on Buffer.

If anyone wants to work on this, the first task would be to convert the dependencies of file-type to use Uint8Array instead of Buffer.

@sindresorhus sindresorhus added enhancement Add new functionality help wanted labels Feb 11, 2023
@Borewit
Copy link
Collaborator

Borewit commented Feb 13, 2023

That work has started to some extend Borewit/token-types#246, but yet there is much more work to do.

@volarname
Copy link

+1
this library really needs a browser version without node and webpack deps
something like fromFile(file: File)
because file.mimeType doesnt seems to work properly all times

@Borewit
Copy link
Collaborator

Borewit commented Mar 6, 2023

+1 this library really needs a browser version without node and webpack deps something like fromFile(file: File) because file.mimeType doesnt seems to work properly all times

That function already exist @volarname : fileTypeFromBlob(blob). Where you can pass you file as blob, since File inherits from Blob.

@akkie
Copy link

akkie commented Mar 9, 2023

+1 this library really needs a browser version without node and webpack deps something like fromFile(file: File) because file.mimeType doesnt seems to work properly all times

That function already exist @volarname : fileTypeFromBlob(blob). Where you can pass you file as blob, since File inherits from Blob.

Yes, but it doesn't work in modern frameworks like SvelteKit as example. It uses vite under the hood and shows the warnings:

Module "node:buffer" has been externalized for browser compatibility. Cannot access "node:buffer.Buffer" in client code.

@Borewit
Copy link
Collaborator

Borewit commented Mar 13, 2023

This module has been primary written for Node.js. It does depend on Buffer. There is browser friendly buffer module available, but you depend on your framework how easy that is to fit it in.
There is not much more we can do, as these core dependencies are not shared between browser and Node.js.

@sindresorhus
Copy link
Owner

https://sindresorhus.com/blog/goodbye-nodejs-buffer

@Borewit
Copy link
Collaborator

Borewit commented Oct 25, 2023

https://sindresorhus.com/blog/goodbye-nodejs-buffer

Interesting!, very nice guidance. I will give it try (starting with the dependent modules first).

@Borewit
Copy link
Collaborator

Borewit commented Nov 1, 2023

It is not only node:buffer file-type depends on, also safe-buffer

graph TD;
    FT-->S
    FT(file-type)-->RWNS(readable-web-to-node-stream)
    S(strtok3)-->P(peek-readable)
    S(strtok3)-->TO("@tokenizer/token")
    TY(token-types)-->TO
    TY-->IE("ieee754")
    FT-->TY
    TY-->NB(node:buffer)
    RWNS-->RS(readable-stream)
    RS-->SD(string_decoder)
    SD-->SB(safe-buffer)
    RS-->UD(util-deprecate)
    RS-->I(inherits)
    style NB fill:#F88,stroke:#A44
    style SB fill:#F88,stroke:#A44
    style SD fill:#CCC,stroke:#888
    style IE fill:#CCC,stroke:#888
    style UD fill:#CCC,stroke:#888
    style I fill:#CCC,stroke:#888
Loading

@Borewit
Copy link
Collaborator

Borewit commented Nov 1, 2023

I still don't see a good alternative for the string encoding node:buffer supports: ascii, utf8, utf-8, utf16le, ucs2, ucs-2, base64, base64url, latin1, binary, hex;

e.g.:

Buffer.from(uint8Array).toString(encoding);

@songkeys
Copy link

songkeys commented Nov 4, 2023

I still don't see a good alternative for the string encoding node:buffer supports: ascii, utf8, utf-8, utf16le, ucs2, ucs-2, base64, base64url, latin1, binary, hex;

I think this library can help? https://github.com/sindresorhus/uint8array-extras

I can see there are methods to hex, string, base64. We have to construct every encoding method from sketch though.

@SgtPooki
Copy link

SgtPooki commented Feb 5, 2024

i've got a working browser-consumable fork at https://github.com/sgtpooki/file-type, but stream and file support has been disabled for now (since I don't need it for my use-case). All tests for fileTypeFromBuffer are passing.

@andrewbrennanfr
Copy link

Hi 👋 Facing similar issues after upgrading from a very old fork of this lib.
I feel like I've read a 101 different issues/comments/responses & ultimately keeping ending up in the same situation.

So just to be crystal clear - this lib can't be used without polyfiling node:buffer right? (in my case with something like this)
There's no "out-of-the-box" way to use this on the client-side?

@mgcrea
Copy link

mgcrea commented Apr 22, 2024

Encountering this while playing with Cloudflare workers, would be nice for fileTypeFromStream to also handle ReadableStream, when I try to pass one (from File.stream()) I get Error: Expected an instance of stream.Readable.

EYHN pushed a commit to toeverything/blocksuite that referenced this issue Apr 22, 2024
Fix toeverything/AFFiNE#6603

This pull request switches the [file-type](https://github.com/sindresorhus/file-type) to [@sgtpooki/file-type](https://github.com/sgtpooki/file-type).

It also includes other minor changes and updates.

~~The `file-type` is no longer useful since the toeverything/AFFiNE#3245 resolved in toeverything/AFFiNE#5565

---

More context see toeverything/AFFiNE#3245 #3555 #4818 #5372 sindresorhus/file-type#578

---

## Caveat

The [@sgtpooki/file-type](https://github.com/sgtpooki/file-type) is an unofficial fork package from the community. It may need to be source code audited before updating.

Also, thanks to sgtpooki for their strong contribution to @sgtpooki/file-type
@cbingb666
Copy link

You can use this dependency vite-plugin-node-polyfills to solve the loading problem of node:buffer

Install the package as a dev dependency.

# npm
npm install --save-dev vite-plugin-node-polyfills

# pnpm
pnpm install --save-dev vite-plugin-node-polyfills

# yarn
yarn add --dev vite-plugin-node-polyfills

Add the plugin to your vite.config.ts file.

import { defineConfig } from 'vite'
import { nodePolyfills } from 'vite-plugin-node-polyfills'

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [
    nodePolyfills(),
  ],
})

fourdim pushed a commit to toeverything/blocksuite that referenced this issue Jul 10, 2024
This pull request is for sunsetting the `@sgtpooki/file-type` and using the official file-type v19.1.0.

The `file-type` package has replaced the usage of `node:buffer` with `Uint8Array`. This change allows it to run safely in the browser now.

The file-type package is used in the attachment-block and image-block modules to determine the file type of uploaded files.

Related to #6803 sindresorhus/file-type#578 (comment)
forehalo pushed a commit to toeverything/AFFiNE that referenced this issue Jul 12, 2024
The official `file-type` package has replaced the usage of node:buffer with Uint8Array. This change allows it to run safely in the browser now.

Related to sindresorhus/file-type#578
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Add new functionality help wanted
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants