Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bounty] nextjs loom replacement pipe #583

Closed
louis030195 opened this issue Oct 25, 2024 · 21 comments
Closed

[bounty] nextjs loom replacement pipe #583

louis030195 opened this issue Oct 25, 2024 · 21 comments
Labels

Comments

@louis030195
Copy link
Collaborator

louis030195 commented Oct 25, 2024

/bounty 100

definition of done:

  • at the end of the day i can go to screenpipe and ask for a loom of a specific time range
  • it should be a nextjs pipe
  • minimum version is taking a time range and stitching together mp4 files (video and audio) and giving me back the video with audio

bonus:

  • highlights, good UX like loom (somehow)

check this nextjs pipe https://github.com/mediar-ai/screenpipe/tree/main/examples/typescript/pipe-simple-nextjs

also be aware nextjs does have some issue with deno sometimes - for example openai streaming failed for me once idk why (we're thinking in switching to bun soon)

related #582

more context:

.route("/experimental/frames/merge", post(merge_frames_handler))

we have endpoint to merge video but might need some changes, e.g. this task might involve a little bit of rust

my use case:

image

PLEASE say the UX you think of and how you're going to implement this before starting the task plz

@louis030195 louis030195 added the enhancement New feature or request label Oct 25, 2024
Copy link

linear bot commented Oct 25, 2024

Copy link

algora-pbc bot commented Oct 25, 2024

## 💎 $100 bounty • Screenpi.pe

### Steps to solve:
1. Start working: Comment /attempt #583 with your implementation plan
2. Submit work: Create a pull request including /claim #583 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to mediar-ai/screenpipe!

Add a bountyShare on socials

Attempt Started (GMT+0) Solution
🟢 @tribhuwan-kumar Oct 25, 2024, 9:43:29 PM #619
🟢 @volfcan Oct 28, 2024, 9:50:57 AM WIP

@tribhuwan-kumar
Copy link
Contributor

tribhuwan-kumar commented Oct 25, 2024

/attempt #583

Algora profile Completed bounties Tech Active attempts Options
@tribhuwan-kumar 4 mediar-ai bounties
Rust, JavaScript,
Lua & more
Cancel attempt

@volfcan
Copy link

volfcan commented Oct 28, 2024

/attempt #583

@tribhuwan-kumar
Copy link
Contributor

@louis030195
i'm thinking to implement something like this, there will a time & date option, user can select a time interval for loom, then it'll merge those specific time ranged videos and audio in one video and send back to the user. there will also an option for quick share, so user can directly share that video on whatsapp, x, etc. i might take some inspiration from loom

(extra feature in this pipe could be a quick summary of that time interval and summary of that video, also it'd be great if user can interact with this pipe)

@volfcan
Copy link

volfcan commented Oct 29, 2024

pretty much the same way with the @tribhuwan-kumar.

there will be a time & date component and when users select the time range for a loom video it'll create the video and send back to the user.

@tribhuwan-kumar
Copy link
Contributor

pretty much the same way with the @tribhuwan-kumar.

there will be a time & date component and when users select the time range for a loom video it'll create the video and send back to the user.

the searching and merging are done, i'm currently working on ux, if you have any suggestion for ux we can collaborate!

@volfcan
Copy link

volfcan commented Oct 29, 2024

pretty much the same way with the @tribhuwan-kumar.
there will be a time & date component and when users select the time range for a loom video it'll create the video and send back to the user.

the searching and merging are done, i'm currently working on ux, if you have any suggestion for ux we can collaborate!

check your twitter dms

@tribhuwan-kumar
Copy link
Contributor

check your twitter dms

there's nothing

@louis030195
Copy link
Collaborator Author

sounds good

@louis030195 i'm thinking to implement something like this, there will a time & date option, user can select a time interval for loom, then it'll merge those specific time ranged videos and audio in one video and send back to the user. there will also an option for quick share, so user can directly share that video on whatsapp, x, etc. i might take some inspiration from loom

(extra feature in this pipe could be a quick summary of that time interval and summary of that video, also it'd be great if user can interact with this pipe)

@tribhuwan-kumar
Copy link
Contributor

@louis030195
is this ui is okay ?
it will open a dialog of merged video and audio

image

@tribhuwan-kumar
Copy link
Contributor

tribhuwan-kumar commented Oct 31, 2024

there was a 500 internal problem with merge frame route, i fixed that. but i noticed some strange behavior while merging videos, imagine this situation, i kill the screenpipe while processing some videos. after killing the screenpipe the video remains uncompleted. and these uncompleted videos are causing errors while merging large amount of videos. maybe i need to add some checks before merging those videos.

i got a corrupt video

PS C:\Users\eirae\Downloads\app\screenpipe> C:\Users\eirae\Downloads\app\screenpipe\screenpipe-app-tauri\src-tauri\ffmpeg\bin\x64\ffmpeg.exe -v error
 -i C:\\Users\\eirae\\.screenpipe\\data\\monitor_65537_2024-10-31_01-32-59.mp4 -f null -
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000022CB0AF1780] moov atom not found
[in#0 @ 0000022CB0ADA540] Error opening input: Invalid data found when processing input
Error opening input file C:\\Users\\eirae\\.screenpipe\\data\\monitor_65537_2024-10-31_01-32-59.mp4.
Error opening input files: Invalid data found when processing input

@louis030195
Copy link
Collaborator Author

@tribhuwan-kumar

nice

video should have audio too at least

i'm actually shipping an endpoint that allows you to stream frames through SSE at high speed for #343

https://github.com/mediar-ai/screenpipe/tree/timeline-ui

i think you might wait i merge this and i'm likely going to do some hacks to stream the audio too so it might solve your problem

@tribhuwan-kumar
Copy link
Contributor

@tribhuwan-kumar

nice

video should have audio too at least

right now, i'm able to merge video to video and audio to audio, (merged video and merged audio are separate)
i tried to merge video to audio but the problem i'm getting is order correction, some audio have only beep sound, some have sound but how can i merge those audio with their corresponding video frame.

i'm actually shipping an endpoint that allows you to stream frames through SSE at high speed for #343

https://github.com/mediar-ai/screenpipe/tree/timeline-ui

i think you might wait i merge this and i'm likely going to do some hacks to stream the audio too so it might solve your problem

for now, i fixed those corrupt video error, by filtering out those video. it prevents corrupt videos getting in input_list.txt, this can be slow bcz it adds an extra check before merging them

@tribhuwan-kumar
Copy link
Contributor

@louis030195
a working beta version:

a.mp4

Copy link

algora-pbc bot commented Nov 1, 2024

💡 @tribhuwan-kumar submitted a pull request that claims the bounty. You can visit your bounty board to reward.

@louis030195
Copy link
Collaborator Author

@louis030195 a working beta version:

a.mp4

thats cool

is there audio in it?

@tribhuwan-kumar
Copy link
Contributor

tribhuwan-kumar commented Nov 1, 2024

@louis030195 a working beta version:

a.mp4

thats cool

is there audio in it?

no, that merged video doesn't have audio, it's just a series of video concatenated by ffmpeg, it doesn't have embedded audio. I tried to embed video with audio but I'm facing issues of order synchronizing :(

@louis030195
Copy link
Collaborator Author

louis030195 commented Nov 1, 2024

@louis030195 a working beta version:

a.mp4

thats cool
is there audio in it?

no, that merged video doesn't have audio, it's just a series of video concatenated by ffmpeg, it doesn't have embedded audio. I tried to embed video with audio but I'm facing issues of order synchronizing :(

are you using database metadata?

you can fetch all the things and order by time

i guess it's possible to have 1-3 seconds of audio desync because we capture timestamp not right sync

also if you try to create a loom over too long range things that i expect to happen:

  • say i closed my lid (screen capture stop but mic keeps going i think) -> will have weird desync with audio

although this should be only for short times

i guess a version 10.0 of this would be to feed this to a voice LLM and fix all the things automatically and comment on the video haha

@tribhuwan-kumar
Copy link
Contributor

are you using database metadata?

you can fetch all the things and order by time

i guess it's possible to have 1-3 seconds of audio desync because we capture timestamp not right sync

also if you try to create a loom over too long range things that i expect to happen:

  • say i closed my lid (screen capture stop but mic keeps going i think) -> will have weird desync with audio

although this should be only for short times

i guess a version 10.0 of this would be to feed this to a voice LLM and fix all the things automatically and comment on the video haha

then i've to create an another route for embedding audio in video bcz this can only concatenate the videos and audios. & thanks for the metadata it would be helpful for embedding :)

Copy link

algora-pbc bot commented Nov 29, 2024

🎉🎈 @tribhuwan-kumar has been awarded $100! 🎈🎊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants