Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Javascript Solutions Source Code #1408

Closed
Choons opened this issue Dec 18, 2020 · 40 comments
Closed

Javascript Solutions Source Code #1408

Choons opened this issue Dec 18, 2020 · 40 comments
Assignees
Labels
legacy:face mesh Issues related to Face Mesh platform:javascript MediaPipe Javascript issues type:feature Enhancement in the New Functionality or Request for a New Solution

Comments

@Choons
Copy link

Choons commented Dec 18, 2020

I'm happy to see that a javascript solution has been released for the facemesh, but can't seem to find the uncompiled javascript or typescript source code here on GitHub. Am I just missing it, or is it not posted?

@mgyong
Copy link

mgyong commented Dec 21, 2020

We don't plan to release the source code for MP JS Solutions API. Python source is out but not JS for now

@mgyong mgyong self-assigned this Dec 21, 2020
@Choons
Copy link
Author

Choons commented Dec 21, 2020

?? if I'm not mistaken the python code is available? Anyway, without the code, then we need more documentation on the API than the little example reveals. I can't figure out the range of capabilities from that alone.

@djthegr8
Copy link

We don't plan to release the source code for MP JS Solutions API. Python source is out but not JS for now

@mgyong why would that be though? The files can certainly be improved by open sourcing and anyways, people can access the minified code, so no scope for proprietary right?

@djthegr8
Copy link

And regarding documentation, please review my PR #1434 which adds helpful notes about API and Utilities

@afogel
Copy link

afogel commented Jan 20, 2021

any update on this? @mgyong (might have gotten lost b/c most people were probably out of the office around christmas/new years)

@afogel
Copy link

afogel commented Feb 16, 2021

@mgyong @tyrmullen sorry to ping, just curious whether there's any new information about getting better documentation/more information about why MP doesn't plan on releasing the source for the JS Solutions API?

@BSchrift
Copy link

Even if the source code has not been released, has anyone documented the API or created typescript declarations?

@mgyong
Copy link

mgyong commented Mar 29, 2021

The problem is that we require bazel to support the --wasm option and additional work on our end to clear up our internal code in order to release. Currently there are no plans till maybe Q2 2021.

@Choons
Copy link
Author

Choons commented Mar 29, 2021 via email

@mgyong
Copy link

mgyong commented Mar 29, 2021

@Choons It's quite different. tf.js does not use wasm. MP js solutions api does. If it was easy for us to release, we would have done it. Matter of bandwidth and priority

@Choons
Copy link
Author

Choons commented Mar 30, 2021 via email

@chuoling
Copy link
Contributor

One thing we could do quickly is to include the TypeScript declarations (w/ inline comments) in the NPM package. @Choons would that help?

@Choons
Copy link
Author

Choons commented Mar 30, 2021 via email

@chuoling
Copy link
Contributor

Great. @mhays-google owns our JS solutions and will look into it.

@Choons
Copy link
Author

Choons commented Mar 30, 2021

I'll leave this here. https://github.com/spite/FaceMeshFaceGeometry

maybe people will find it useful

@mhays-google
Copy link
Contributor

I am uploading our TypeScript exports file as part of the npm package for each and every solution. For any solution, the easiest way (I've found) to get to this file will be:

https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/control_utils/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/face_detection/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/pose/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/hands/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/holistic/index.d.ts

...et. al.

Even if you are not familiar with TypeScript, this should be pretty decipherable.

While this is not full documentation, per se, it should give you visibility into what can be called within the API. This is a quick fix to address what is obviously a very opaque interface. I will add a README.md to the packages giving these same directions shortly, but I wanted to have this to you by end of day. I'll look into better documentation in general.

@mhays-google
Copy link
Contributor

Regarding the full release of the source code, this is being discussed, I personally support it, but there are a few hurdles.

Hopefully releasing the index file solves a few immediate issues -- thank you for the patience with that, by the way. And we'll look into what it will take to release more of the code (or at least the unobfuscated javascript).

@Choons
Copy link
Author

Choons commented Mar 31, 2021

Much appreciated, Michael! Yeah I love web assembly as a concept, but without documentation of what the wasm modules expose, it's just shooting in the dark trying to guess what's in the API. Also love the concept of using emscripten/llvm on c++ to create javascript/wasm, but in practice I have found the reality is often more difficult to create than just porting the code manually to javascript or Typescript, or Assemblyscript and then to wasm for the performance gains. Maybe the community here can contribute in those areas of the API that are lacking due to the work load of the developers.

@afogel
Copy link

afogel commented Mar 31, 2021

THIS IS AWESOME!! Thanks @mhays-google

@BSchrift
Copy link

BSchrift commented Apr 3, 2021

Thanks @mhays-google! This is so helpful.

I do have one more quick question: is there a reason why the package.json for the npm package doesnt specify a "main" or "typings" field?

It would be great to be able to say, for example, import { Hands } from '@mediapipe/hands' and have typescript know where to get the js code and the associated type declarations. Looking at the distributed package.json I think it would just be a matter of setting:

"main": "hands.js", 
"typings": "index.d.ts",

or the equivalent in each package, unless I'm misunderstanding how the packages are structured.

@lostfictions
Copy link

It's really unfortunate that there's no better documentation available, especially for the helper libraries (@mediapipe/drawing_utils, @mediapipe/camera_utils, @mediapipe/control_utils). Without them, it's remarkably hard to figure out how one would go about implementing drawing code -- for example, drawConnectors is totally opaque since it's shipped as a Closure-minified module, and there's no explanation or documentation elsewhere for how landmark indices are meant to be interpreted. As it stands, MediaPipe JS seems to either be a toy to show that a browser implementation is theoretically possible, or... a solution non-googlers are supposed to reverse-engineer?

@ayoub-root
Copy link

any updates ?

@alexandernst
Copy link

Almost hitting the 2 years mark and still no source code. Google is totally misleading everybody when stating that this is open source. Either release the code or clearly state that it's not open source.

image

@wheelie33
Copy link

lease the source code for MP JS Solutions API. Python source is out but not JS for n

Why?

@wheelie33
Copy link

ap as input if you really needed to (this was discussed in anothe

Can you provide a working example of the HTMLImageElement? I'm trying a VERY VERY small prototype and it has some unhandled exceptions.

@esinanturan
Copy link

Regarding the full release of the source code, this is being discussed, I personally support it, but there are a few hurdles.

Hopefully releasing the index file solves a few immediate issues -- thank you for the patience with that, by the way. And we'll look into what it will take to release more of the code (or at least the unobfuscated javascript).

@mhays-google Any new regarding the release ?

@kuaashish kuaashish added type:feature Enhancement in the New Functionality or Request for a New Solution and removed type:support General questions labels May 29, 2023
@kuaashish
Copy link
Collaborator

@Choons,

JS Source is available for MP Tasks, And Will not be published for other Solutions.

@kuaashish kuaashish assigned kuaashish and unassigned mhays-google Jun 8, 2023
@kuaashish kuaashish added stat:awaiting response Waiting for user response and removed stat:awaiting googler Waiting for Google Engineer's Response labels Jun 8, 2023
@afogel
Copy link

afogel commented Jun 8, 2023

Tasks

Hi @kuaashish, thanks for the update! Based on my read through the tasks documentation, does that meant that we would effectively create out own holistic model using a tasks pipeline that we define for ourselves?

How much less efficient will it be to use the pose, face, and hands tasks in conjuction, as compared to using the holistic model?
Thanks :)

@Choons
Copy link
Author

Choons commented Jun 8, 2023

it's pretty much useless. I moved on to other code a long time ago.

@google-ml-butler google-ml-butler bot removed the stat:awaiting response Waiting for user response label Jun 8, 2023
@alexandernst
Copy link

So long for the "open source" part of this project

@kuaashish
Copy link
Collaborator

@Choons,

Thanks for the confirmation. If this is no longer issue from your end, Can we move ahead and close the this?

@kuaashish kuaashish added the stat:awaiting response Waiting for user response label Jun 12, 2023
@github-actions
Copy link

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale label Jun 20, 2023
@github-actions
Copy link

This issue was closed due to lack of activity after being marked stale for past 7 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
legacy:face mesh Issues related to Face Mesh platform:javascript MediaPipe Javascript issues type:feature Enhancement in the New Functionality or Request for a New Solution
Projects
None yet
Development

No branches or pull requests