Marker Batch Rename Panel

[IAmVigneswaran]

@milanvarady This is somethings we can look into after the release of 1.0.0.

Maybe during version 1.2.0 or 1.3.0.

Just made a quick mockup.

1. The Rename Panel can be place after Queue Panel.
2. Users could drag and drop FCP Timeline or FCPXML into the dropzone.
3. Marker Data would retrieve the entire list of the Marker names with Notes.
4. Users can select which Marker that would like to rename.
5. They would click the Rename Options tab to set and configure their preference.
6. Once done, they can press Batch Rename Marker Data would write back / save into the FCPXML file.
7. There can be a option to automatically open FCP to import back the FCPXML file into the target Library.
8. The i or gear button would have pre-defined codes/syntax.
9. The Rename Panel is somewhat standalone. The settings are not part of our json configurations.

Some ideas
https://www.publicspace.net/ABetterFinderRename/index.html
https://renamer.com/

@orchetect I am wondering if we could add any special API call for MarkersExtractor or DAWFileKit to achieve this in a optimal fashion?

1. When Marker Data reads the FCPXML to retrieve the entire list of the Marker names, it could use a special MarkersExtractor or DAWFileKit API call to create local json cache file.
2. The json file, contains all the Marker Name and its associated Notes Name.
3. This could be a hidden feature that is not available in our CLI.
4. Marker Data would read the local json cache file to execute the rename tasks for writing.
5. Once done, another API could be used to write back the values in the original FCPXML file or create a new one (making sure all other data are not touched).

Open to suggestions and ideas.

Thank you.

[orchetect]

A few thoughts -

I'm familiar with A Better Finder Rename - awesome tool and I've been using it for probably over 10 years. Worth every penny!

I am wondering if we could add any special API call
[1. ... 4.]

As I've stated in the past, MarkersExtractor is just a data extractor with limited formatting options for pre-determined output profiles. Once you get its output, you are free to do what you like with it. Post-processing or additional formatting of data is primarily a Marker Data feature here. What you decide to do with it is, again, up to you. If you want to write it back to the JSON/CSV/TSV file, that would be your responsibility and discretion.

I suppose there could be an access point in the MarkersExtractor library (not CLI) that would allow manipulation of the data model before exporting the manifest and other files to disk.

5. another API could be used to write back the values in the original FCPXML file or create a new one

This is a new feature all on its own - essentially a "FCPXML Export Profile". It would make more sense in a hypothetical future when more input data sources than just FCPXML may be possible. Currently DAWFileKit has the ability to edit existing FCPXML and/or author a new FCPXML from scratch. We just haven't needed it since we've only been treating FCPXML as a source and not an export format/profile.

Which leads to a bigger question of: what exactly are you wanting to achieve? It feels like batch processing just the Name and Notes fields is an arbitrarily limited application of the necessary engineering that would be needed to build to achieve it. You could do a lot more with it.

A few other thoughts:

• Before and After tabs feels clunky. I would combine them into a table with 4 columns (Name, Notes, New Name, New Notes) so you can directly correlate the data transformation.
• If you want to allow power user features easily, implement RegEx replacement as an option

[IAmVigneswaran]

Thanks for the input, Steffan!

I've been using it for probably over 10 years. Worth every penny!

Same here!

Which leads to a bigger question of: what exactly are you wanting to achieve?

While creating Marker database for VFX/Review/Editorial is the main purpose of Marker Data. The seconday purpose would be the creation of Personal Shot/Frame Library for Reference. Something similar to Shot Deck. It was one of the primary motivation to add colour palette #17.

1. It is extremely easy is to add markers in FCP (using m key).
2. FCP defaults to Marker 1, Marker 2, Marker 3 and so on.
3. But when the users has 20 or more markers, it can be really tedious to rename the markers.
4. Users can have clips of different movies, commercial, tv shows in their timeline.
5. But in order to name them correctly, they would have to rename Marker 1 to Speed Racer - 001, Speed Racer - 002 or even Batman - 001, Batman - 002. Respective to their clips. It can really be tedious.

Marker.Rename.Idea.-.1.mp4

Once they have renamed their markers, they can perform extraction using Name has the Naming Mode with Swatch enabled.

Marker.Rename.Idea.-.2.mp4

User can upload to Notion database.

Marker.Rename.Idea.-.3.mp4

---

If you want to write it back to the JSON/CSV/TSV file, that would be your responsibility and discretion.

I strongly believe, Marker Data manipulating the JSON/CSV/TSV file is not the best or ideal approach. Users would always want to have a backup of their Library with movie reference clips. They would want to make sure that the markers listed in the Timeline is identical to JSON/CSV/TSV file. Which means, manipulation occurs not at the final JSON/CSV/TSV stage, but rather at the FCPXML stage.

This is a new feature all on its own - essentially a "FCPXML Export Profile". It would make more sense in a hypothetical future when more input data sources than just FCPXML may be possible. Currently DAWFileKit has the ability to edit existing FCPXML and/or author a new FCPXML from scratch. We just haven't needed it since we've only been treating FCPXML as a source and not an export format/profile.

1. A new FCPXML Export Profile could be one way.

2. But the above mentioned reason, since DAWFileKit or MarkersExtractor library is already parsing the Markers, it would great if we can have a API call to "Temporarily" read the list of markers. Marker Data could use the cache data to rename the markers. Once done, another API to write back to the FCPXML using the cache data.

3. Users can open/Import the FCPXML into their existing library. FCP would create a new (duplicate) timeline with all the markers renamed respectively.

4. Users can perform the extraction with Marker Data, as per normal.

If you want to allow power user features easily, implement RegEx replacement as an option

Yes, RegEx would be a great feature.

This batch marker rename could also be useful for batch renaming VFX Shot IDs.

[orchetect]

it can be really tedious to rename the markers

Final Cut really doesn't help here, yeah. You can't even drag markers to reposition them. Feels very claustrophobic and archaic. I'm pretty sure it has no batch rename either - you're relegated to editing single markers at a time.

manipulation occurs not at the final JSON/CSV/TSV stage, but rather at the FCPXML stage

Right, and I wouldn't recommend editing the output files any way.

can have a API call to "Temporarily" read the list of markers. Marker Data could use the cache data to rename the markers. Once done, another API to write back to the FCPXML using the cache data

There's kind of 2 different things going on here. One is the concept of mutating the FCPXML data before MarkersExtractor performs its extraction. Another is rewriting a FCPXML file, regardless of whether we actually want to continue with extraction or not. I'd want to clarify the use cases so that the API makes sense.

[IAmVigneswaran]

You can't even drag markers to reposition them. Feels very claustrophobic and archaic.

https://commandpost.io/faq/#why-did-you-remove-moveable-markers

One of the reason why most FCP users would usually place markers on Title. It give a little more customisation.

There's kind of 2 different things going on here. One is the concept of mutating the FCPXML data before MarkersExtractor performs its extraction. Another is rewriting a FCPXML file, regardless of whether we actually want to continue with extraction or not. I'd want to clarify the use cases so that the API makes sense.

Not sure if an intermediate json cache file would make sense. Marker Data would read/write to the json cache file. During the writing process, the API would use the json cache file to write/update the FCPXML.

[orchetect]

intermediate json cache file

That's a lot more work than necessary really. The library could be adapted to just give you its manifest data model without writing any files to disk. Would just require a few changes in the code to make it accessible.

[milanvarady]

This feature would be a small app in itself. We should first see how users actually use Marker Data, and then we can decide if enough users would use this feature. Maybe we can poll to see if this is a feature people would like to see.

[IAmVigneswaran]

This feature would be a small app in itself.

True. But since Marker Data, deals with Markers, it somewhat makes sense to have this feature. That way, Marker Data becomes an all encompassing application for FCP's Makers. As a user of FCP, I kinda hate the idea installing multiple apps to solve and augment different workflows needs.

Maybe we can poll to see if this is a feature people would like to see.

Certainly. But, this batch rename feature can be vital toolkit of Maker Data, because of the above mentioned reasons.

[milanvarady]

@IAmVigneswaran This is how I would implement the rename panel.

Instead of the Before, Rename Options and After panels I would combine everything into a single panel. Also I would simplify the process a bit. This is how I would do it:

• Select .fcpxml(d) file
• We would have 2 options: replace all markers or only replace markers which contain a certain text
• Matching markers would be renamed to a newly defined name follwed by the numbering (001)

We can do the same for notes.

We don't need any intermediate JSON or anything. Because we are only renaming notes and markers we don't need any complicated logic, I can just use an XML parser I think. Maybe @orchetect can help with reading and writing the actual .fcpxml(d) files if I need help.

[orchetect]

You shouldn't be touching XML at all. That's what MarkersExtractor is for.

library could be adapted to just give you its manifest data model without writing any files to disk

Like I said, I can add API to return all marker data that would otherwise be contained in a CSV as an object in memory instead.

Is this process pre- or post- processing with a profile?

That gets a bit more complicated because you would need to modify the data at some point in the middle of the process that MarkersExtractor performs.

[orchetect]

I think this reveals the fact that MarkersExtractor is really doing two jobs. It's parsing out markers from FCPXML, but it's also processing the data in order to conform to output profiles.

Those two processes could be split and it could be possible to invoke them separately.

[milanvarady]

Like I said, I can add API to return all marker data that would otherwise be contained in a CSV as an object in memory instead.

Oh, nice. So if I understand well I need the "raw" data to change the marker names (best would be some Swift object), then a way to save this modified data back into .fcpxml(d).

[orchetect]

way to save this modified data back into .fcpxml(d)

No, you would modify the data then just return the object to MarkersExtractor to complete the process.

[orchetect]

Writing back to FCPXML is non-trivial and not something we can reliably do yet.

[milanvarady]

No, you would modify the data then just return the object to MarkersExtractor to complete the process.

Yes, that's what I meant. So if I'm correct, MarkersExtractor would have two functions. Something along the lines of rawDataFromFCPXML() and rawDataToFCPXML(). But you say the second one is tricky to do?

[orchetect]

I'm not sure how I'm not being clear.

Once we ingest FCPXML, that's it. We can't go back to it at this time. We have to use the parsed data as our own format.

Instead of a single call to MarkersExtractor to convert FCPXML to profile output files, it can be two calls.

First call returns an in-memory data object that can be modified. You modify it, and then call the second call passing the modified data object in and the process completes as usual.

If you want to save the data at that in-between step then that complicates things.

[IAmVigneswaran]

@orchetect

Is this process pre- or post- processing with a profile?

I believe, technically this is a pre-processing? Cause this rename API call/write feature is an additional internal library feature that would not be available in our CLI.

That gets a bit more complicated because you would need to modify the data at some point in the middle of the process that MarkersExtractor performs.

And this feature is separate from our dedicated extract profiles.

@milanvarady

Instead of the Before, Rename Options and After panels I would combine everything into a single panel. Also I would simplify the process a bit.

Yes, I believe we can optimise the workflow better?

Some ideas.

I also have some ideas from a popular free windows tool, Bulk Rename Utility

In Bulk Rename Utility the rename preview is always LIVE. That means, the users will always able preview the new name live.

Taking the same concept, yes, Instead of the Before and After panels, we can just have 2 Tabs Preview and Rename Options.

In the Preview Panel, We can have 4 Columns. Marker (Original), Notes (Original), Marker (New) and Notes (New), Both of the New Columns's text value can be in Green in Colour.

Each time when users make changes in the Rename Options tab, users can click on the Preview tab to preview their rename changes.

---

Instead of a single call to MarkersExtractor to convert FCPXML to profile output files, it can be two calls.

First call returns an in-memory data object that can be modified. You modify it, and then call the second call passing the modified data object in and the process completes as usual.

If you want to save the data at that in-between step then that complicates things.

@orchetect

Ideas:

We can have just 2 Call? One to Read and one to Write and generate a fcpxml(d)?

When users drag and drop their timeline into Marker Data's Workflow Extension, Marker Data would use the Read API Call to list all the Markers and Notes in the Preview Tab.

Once users are satisfied with the new rename of both/or Markers and Notes, Users can press Send to Final Cut Pro button. Marker Data would MarkersExtractor's Write API Call to generate a new FCPXML and launch FCP. And Users can click through the dialogue box as usual.

[IAmVigneswaran]

I am not entirely sure if this workflow makes sense or could work. I just thought of penning down my thoughts and ideas.

It is paramount that the final renamed markers/notes are returned back to Final Cut Pro. This way, users have a complete set of newly renamed markers/notes in their Final Cut Pro library and not just an extracted profile in Notion or Airtable. This allows users to continue extracting their markers in any desired profile and configuration as per usual.

1. We could have a new tab in the WorkflowExtension called Rename. When users drag and drop their timeline into this tab, Marker Data can create a WorkflowExtensionExport.fcpxml file in the Marker Data Cache folder.

2. I wonder if this is possible: When Marker Data uses the Read API Call from the MarkersExtractor, MarkersExtractor would create a JSON File containing all the Marker Name, Notes, Position Data. This JSON file could be stored in the same Marker Data Cache Folder.

3. Marker Data would then utilise (Read and Write) this JSON file for all renaming Tasks and Manipulation and listing in the Preview Tab as shown in the previous reply.

4. Once renaming is complete, Marker Data could use a Write API Call. MarkersExtractor would then use the same JSON File to write and update existing WorkflowExtensionExport.fcpxml file located in the Marker Data Cache folder. In this way, technically, MarkersExtractor wouldn’t need to generate a brand-new FCPXML file? And the user's timeline would be pretty much intact.

5. Once the updating of WorkflowExtensionExport.fcpxml is complete, Marker Data would trigger WorkflowExtensionExport.fcpxml to open in Final Cut Pro. And Users can import the timeline back to their library.

[orchetect]

Definitely possible on the MarkersExtractor side.

1. I can split its operation into two stages as mentioned before.
   1. PARSE: One operation parses FCPXML and flattens it down to a canonical format. This could include some or all of: an in-memory struct, csv, and json. Basically identical to using the CSV or JSON extract profiles.
   2. CONVERT: The second operation can take any of the first operation's outputs as its input (struct, csv, json) and output using any of the Profiles.
2. It's possible for me to add an API to DAWFileKit that can write a barebones FCPXML file with a specified frame rate, start timecode, and array of markers. Whatever structure is required for the Workflow Extension to take.

I think this would serve all of the needs you've described.

[IAmVigneswaran]

Basically identical to using the CSV or JSON extract profiles.

I am also wondering, does it make sense to include the Line Number from FCPXML as a unique identifier in the JSON? Since Line Number is always sequential, it would be always unique? This is to mitigate any identicial Values (Marker and Notes) from the FCPXML in the same timecode.

It's possible for me to add an API to DAWFileKit that can write a barebones FCPXML file with a specified frame rate, start timecode, and array of markers. Whatever structure is required for the Workflow Extension to take.

Instead of creating a barebones FCPXML file, Is it possible to write back to the original FCPXML file, in our case WorkflowExtensionExport.fcpxml? That way, the whole timeline is intact. Users would definitely have super complex timelines containing Multicam clips, Compound clips, effects and titles and etc. It is vital that we have to preserve the original user's FCPXML file.

Using the Line Number as the unique identifier, it would be ideal if we can replace the values of the Markers and Notes.

[orchetect]

does it make sense to include the Line Number from FCPXML

No. Line numbers are brittle and naive. There are better ways to reference markers. We already have a marker ID as an identifier included in the manifest, for one. There could be an additional manifest field that contains an XML path which would be the ideal way to positively identify FCPXML nodes.

Instead of creating a barebones FCPXML file, Is it possible to write back to the original FCPXML file

It's possible if we have XML paths as mentioned above. I could provide API to update existing marker information in FCPXML that way.

Send to Final Cut Pro button

the final renamed markers/notes are returned back to Final Cut Pro

Is this involving Marker Data, a Final Cut extension, or both? I'm a bit confused here. Do you mean Marker Data will be able to rename markers as part of its process, but the user will also want to re-import the markers back to their FCP project? Or is this something entirely contained in the FCP extension with renaming and re-importing back to FCP?

I'm not sure it makes sense to send the entire FCPXML back to Final Cut? What does it do with it? Is there not a scenario where only the markers on a bare timeline should be sent to it?

FCP would create a new (duplicate) timeline with all the markers renamed respectively.

This feels kind of clunky. Is this the only way?

[IAmVigneswaran]

No. Line numbers are brittle and naive. There are better ways to reference markers. We already have a marker ID as an identifier included in the manifest, for one. There could be an additional manifest field that contains an XML path which would be the ideal way to positively identify FCPXML nodes.

Noted! Just wanted to make sure there are no conflicts when the value of the Markers / Notes are identical.

Is this involving Marker Data, a Final Cut extension, or both?

Yes. Both. Users would use the Workflow Extension to Drag and Drop the timeline into the Drop Zone and it would export a WorkflowExtensionExport.fcpxml file into the Marker Data Cache folder (Like how it is currently doing now). With the proposed method to read the RAW Data (Using Read API), Marker Data would also save the RAW JSON file in the same Marker Data Cache folder.

Do you mean Marker Data will be able to rename markers as part of its process.

Yes, that is the idea. Using the RAW JSON Data, Marker Data can all the Markers and Notes. And using the Rename Options, users can BATCH rename Markers and Notes to their heart's content. There can be a Save or Commit button to save and update the JSON file.

Once, Renamed, Marker Data can use Write API. And MarkersExtractor would use the updated JSON file to update and write/append back to the original FCPXML. It can also create a duplicate of the FCPXML file (For backup) called WorkflowExtensionExport-Rename.fcpxml. Marker Data, would the open the FCPXML to import back into FCP.

but the user will also want to re-import the markers back to their FCP project?

Yes. 100% In all professional workflows, users would want to re-import back their timeline.

This is a video some 7 years go. But you can see that user would always want to re-import back their timeline.

https://www.youtube.com/watch?v=A1Diuxkf8E4

Up coming Multicam Flattener app. You can see, that even though the new imported timeline has Multicam Flatten, all the timeline and all other stuff is pretty much intact.

https://www.youtube.com/watch?v=Y2Ezd4F2clQ

Or is this something entirely contained in the FCP extension with renaming and re-importing back to FCP?

The idea is we have new Tab in the Marker Data's Workflow Extension for Drop & Drop the Timelines. Once the data is received, it would automatically launch our Main Marker Data app for all the renaming manipulation.

I'm not sure it makes sense to send the entire FCPXML back to Final Cut? What does it do with it?

It does make sense to send the complete intact timeline. And that is what the users are expecting.

Is there not a scenario where only the markers on a bare timeline should be sent to it?

No. Because in FCP there is no concept of TIMELINE MARKERS. All the makers are anchored to clips. And even if you can COPY and PASTE makers from one timeline to another, it would be all over the place. Users don't do that.

Users can have super complex feature length timeline with 500+ clips. They can retimed, have effects and multi-cam clips and etc. At any given time, users can have 100+ Markers with notes. And users would want batch rename them in one go. And they want to re-import back the new timeline. Once they are happy with the renaming, they can delete their original timeline from their FCP Browser.

This feels kind of clunky. Is this the only way?

Nope. Not at all, re-importing timelines or events is a common practice in the FCP. Check out the above videos.

[orchetect]

Ok, makes sense. This seems doable.