Subproject / subfolders options and structure

[Kimbaras]

Hey!

Not sure if this is in scope / it's not what you meant, so feel free to close this.

Basically I've tried loading my whole library into MMP, and while it's working, picking up the models, seems the "Project" structure is a little messed up.

In my case, the library structure is something similar (these are just examples to give you an idea):

• Main folder
  • Cosplay
    • Halo
    • Warhammer
    • Fantasy
  • Tools
    • Tool 1
    • Tool 2

During the initial setup, MMP created the relative "Projects" based on the subfolder names and for those containing some files.
Given the above example, it created the projects "Halo", "Warhammer" and so on, but not the "macro" or "parent" project "Cosplay".

Now, I understand that my particular folder structure is... well, mine, but I believe a "sub-project" or "sub-folder" option could be interesting (maybe this could be marked as "feature request"?)

Thanks!

[EduardoOliveira]

Hi there,
Thank you for reaching out.
Projects are a flat hierarchy, but, your folder structure is represented in the tag system.
In your example, you will have 2 tags (cosplay and tools) automatically created, and the projects within that folder will be tagged.
Given that, you can filter by tag in the projects screen, showing only the projects with those tags and, in the project page, you can click the tag (shown in the top banner) to see other projects of that tag.

I know it's not a folder system, but maybe it can solve your usecase.

Please explore that feature, and I'm more than happy to discuss further improvements to the platform

[Kimbaras]

Good call.

I'm not really a fan of "tag-heavy" environments and I'd prefer a structure folder, but that is just my personal taste.
I know for sure that I got some folders with go at least 4 subfolders deep, so could be a little confusing relying on the tags, but I'll take a look on that for sure!

Thanks for the feedback!

[EduardoOliveira]

There was another discussion about subfolders on discord, and I would like some feedback on possible future implementation. Please take a look at the general chat around the date of this post.

Let's talk about the expected behavior of having folders within projects.
1- Every folder directly beneath the library folder would be a project if within its tree there is a file of any kind.
2- Folders within a project that don't contain files are ignored.
3- Folders within a project that contain files show up in the UI as folder.
4- Folders within a folder that also contains files are shown at the same level as those files.

• this can allow for deep nesting in the UI, not sure if that's good
• as an alternative, all folders within a project could be shown at the same level in the UI hiding the nesting and flattening the tree to a single level.
  5- Folders within a project that have no files but contain folders with files are not shown in the UI.
• this would simplify the UI
• the name of the folders could still be used as project tags

Further considerations.
This implementation would make moving projects unnecessary?
Would moving assets/folders inbetweeen projects a requirement?
Tags would still be applied at project level, from what information should tags be generated?

Comments and feedback is welcome as always.

[johnwhobbs]

Any thoughts of tags for the actual models, rather than at the project level? I may have projects that have pre-supported and unsupported models, projects that have models of bases, figures, terrain, vehicles, etc, or models with different sizes. I like the idea of automatically assigning tags based on folders but having them flow down to actually tag the model would be much more useful for filtering.

[Borbarad2k1]

I just installed MMP and trying to figure it out. The first project I tried to import was: https://www.kickstarter.com/projects/arenpi/the-desert-guest-house
It is a building with up to three floors and some variations to it(over 36 STL files). In my opinion, the "right" way would be to have a single project folder, followed by a folder for every floor, followed by a folder for every variation of that floor. And only the 'main' project folder is visible in the 'Projects'-view. I click on it and then I see 'First floor', 'Second Floor' ... This way I can see what is necessary for the building.

"1- Every folder directly beneath the library folder would be a project if within its tree there is a file of any kind."
I can just talk for myself, but at the moment I have a folder with the name of the other and maybe the Kickstarter, as a base.
"/Arenpi Dungeon Party/The Desert Guest House/"

"2- Folders within a project that don't contain files are ignored."
I would show all folders. Because, I think there should be a 'New folder' button next to the 'Add asset' button, so one can create a new folder within the UI, but this folder would be empty, and you were unable to add stuff to it. The other reason: Think about two nested empty folders (path: 'project/a/b'). You click on the project, there is a folder 'a'. You click on 'a'. Nothing to see, because 'b' is hidden. Why then show 'a' at all?! This leads to recursively scanning the folders.

"3- Folders within a project that contain files show up in the UI as folder."
Nothing to discuss from my side.

"4- Folders within a folder that also contains files are shown at the same level as those files."
As I pointed out in my use case, I'm all for folders. Flattening the structure, might lead to other issues, like:

• You must recursively search for all files
• What happens with duplicate names
• Sorting
• Grouping

"5- Folders within a project that have no files but contain folders with files are not shown in the UI."
I'm sure, it means something different than '2' but I'm not sure how it differs. So my answer is: see '2'

---

Altough I'm not sure if I would do this with simpler projects, e.g. a person with 3 head options, I would properly just put them in one folder and call the files 'Body', 'Head_A', 'Head_B,' Head_C'

[EduardoOliveira]

I'm currently, slowly, writing a "V2" of the project, where I dropped the concept of project, everything is an asset, being it a file asset (multiple subtypes) , or a folder asset, all assets have tags for easier search, the project concept would appear as a property of a folder asset, that would show it in a project view.
This will take time, but I think it will allow for the kind of flexibility that's being discussed on this thread

[EduardoOliveira]

Hi, I would love for you guys to give a try to the agent:v2 image, it's not close to ready but many of the concepts that were discussed in therms of files system are now implemented, and I really would love some feedback....

One of the changes is that bundle files (3mf, zip, rar, possibly more, let me know) are now treated similarly to a folder, the discovery routine maps its insights, and in the UI you can choose to extract each contained file. That said, it created a new behavior, extracting the files makes them siblings of the bundle in the real filesystem.

• is that ok? (Maniacal eye twitching movement)
• should a folder be created as sibling to the bundle and those assets placed inside of it? Should that folder be .hidden?
• should the extracted files be placed in another "temporary" file structure?

I would truly appreciate some feedback and if possible some sample file structures to replicate and feel locally.

Please check out https://github.com/Maker-Management-Platform/agent/tree/v2

[pejotigrek]

Hi, I would love for you guys to give a try to the agent:v2 image, it's not close to ready but many of the concepts that were discussed in therms of files system are now implemented, and I really would love some feedback....

(...)

Please check out https://github.com/Maker-Management-Platform/agent/tree/v2

hi!

tried to run v2 with just couple of files to test - unfortunately can't start it :(
ui just shows spinning circle, and the agent container log has some database is locked errors:

time=2024-07-29T21:31:06.613Z level=INFO msg="Config loaded" Config="&{Core:{Log:{EnableFile:false Path:log.log} DataFolder:/data} Server:{Port:8000} Library:{Paths:[/library] Blacklist:[.git .svn .hg .bzr .DS_Store .project.stlib .thumb.png .render.png] IgnoreDotFiles:true AssetTypes:map[image:{Name:image Label:Images Extensions:[.png .jpg .jpeg .gif .bmp .webp] Order:1} model:{Name:model Label:Models Extensions:[.stl .3mf] Order:0} slice:{Name:slice Label:Slices Extensions:[.gcode] Order:2} source:{Name:source Label:Sources Extensions:[.stp .step .ste .fbx .f3d .f3z .iam .ipt] Order:99}]} Render:{MaxWorkers:4 ModelColor:#167DF0 BackgroundColor:#FFFFFF} Integrations:{Thingiverse:{Token:}}}"
time=2024-07-29T21:31:07.007Z level=INFO msg="Starting agent"
   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.10.0
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:8000
time=2024-07-29T21:31:07.010Z level=INFO msg="Discovering assets" module=discovery root=/library
time=2024-07-29T21:31:07.854Z level=INFO msg=Rendering asset=norbert/proca/Norbert_PROCA_SLSTTF_V2.stl img=1dc9f2421ddddf96bae90ae3cb0a0d39.r.png asset="&{ID:1dc9f2421ddddf96bae90ae3cb0a0d39 Label:0xc00048aa80 Description:<nil> Path:0xc00048aa70 Root:0xc00048aa60 Extension:0xc00048aa90 Kind:0xc00048aab0 NodeKind:0xc00048aaa0 ParentID:0xc0002735f0 Parent:0xc0002735f0 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc000494100 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:08.777Z level=INFO msg=Rendering asset=peejay/dji/dji_uchwyt-do-telefonu_v3.stl img=d843736a099e0d5b22fde9a548fd5ef9.r.png asset="&{ID:d843736a099e0d5b22fde9a548fd5ef9 Label:0xc00003fbf0 Description:<nil> Path:0xc00003fbe0 Root:0xc00003fbd0 Extension:0xc00003fc00 Kind:0xc00003fc20 NodeKind:0xc00003fc10 ParentID:0xc0001101a0 Parent:0xc0001101a0 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc0000144b0 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:09.151Z level=INFO msg=Rendering asset=peejay/dji/dji_uchwyt-do-telefonu_v4.stl img=25a9ced3464ed4f1d803364fbb053f3b.r.png asset="&{ID:25a9ced3464ed4f1d803364fbb053f3b Label:0xc00048a080 Description:<nil> Path:0xc00048a030 Root:0xc00048a010 Extension:0xc00048a090 Kind:0xc00048a100 NodeKind:0xc00048a0e0 ParentID:0xc0001101a0 Parent:0xc0001101a0 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc000494028 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:09.490Z level=INFO msg=Rendering asset=peejay/dji/dji_uchwyt-do-telefonu_v5.stl img=ebbaf387dcafe7c83c835a69c783a5f8.r.png asset="&{ID:ebbaf387dcafe7c83c835a69c783a5f8 Label:0xc00048a1e0 Description:<nil> Path:0xc00048a190 Root:0xc00048a180 Extension:0xc00048a200 Kind:0xc00048a260 NodeKind:0xc00048a210 ParentID:0xc0001101a0 Parent:0xc0001101a0 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc000494018 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:09.875Z level=INFO msg=Rendering asset=peejay/folia-rolka-przedluzacz_v1.stl img=909825ecf6f0a5781602db141f8b6981.r.png asset="&{ID:909825ecf6f0a5781602db141f8b6981 Label:0xc0003e2110 Description:<nil> Path:0xc0003e2100 Root:0xc0003e2050 Extension:0xc0003e2120 Kind:0xc0003e2140 NodeKind:0xc0003e2130 ParentID:0xc000110d00 Parent:0xc000110d00 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc00037a118 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:10.463Z level=INFO msg=Rendering asset=peejay/garaz-esp32-obudowa_v1.stl img=2d96630091cbbe5c3d2ff2b80848e283.r.png asset="&{ID:2d96630091cbbe5c3d2ff2b80848e283 Label:0xc0002d3c90 Description:<nil> Path:0xc0002d3c80 Root:0xc0002d3c70 Extension:0xc0002d3ca0 Kind:0xc0002d3cc0 NodeKind:0xc0002d3cb0 ParentID:0xc000110d00 Parent:0xc000110d00 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc000494a78 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:10.664Z level=INFO msg=Rendering asset=peejay/garaz-esp32-obudowa_v1_top.stl img=6ce96ae2e182406d6cd751072ebea47f.r.png asset="&{ID:6ce96ae2e182406d6cd751072ebea47f Label:0xc0003e3140 Description:<nil> Path:0xc0003e3130 Root:0xc0003e3120 Extension:0xc0003e3150 Kind:0xc0003e3170 NodeKind:0xc0003e3160 ParentID:0xc000110d00 Parent:0xc000110d00 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc00037a478 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:11.205Z level=INFO msg=Rendering asset=peejay/gopro/_GoPro_2-Finger-knuckle.stl img=4ff218d12c6983348f8cce71cef2face.r.png asset="&{ID:4ff218d12c6983348f8cce71cef2face Label:0xc0000a0080 Description:<nil> Path:0xc0000a0060 Root:0xc0000a0050 Extension:0xc0000a0090 Kind:0xc0000a00b0 NodeKind:0xc0000a00a0 ParentID:0xc0000d2d00 Parent:0xc0000d2d00 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc0004940c0 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:12.901Z level=INFO msg=Rendering asset=peejay/gopro/_GoPro_Lens_Cap.stl img=7e629a3a7f0110c0a45d6160e1195fed.r.png asset="&{ID:7e629a3a7f0110c0a45d6160e1195fed Label:0xc0000a15a0 Description:<nil> Path:0xc0000a1590 Root:0xc0000a1580 Extension:0xc0000a15b0 Kind:0xc0000a15d0 NodeKind:0xc0000a15c0 ParentID:0xc0000d2d00 Parent:0xc0000d2d00 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc000494420 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:13.197Z level=ERROR msg="failed to save extracted asset" module=process asset=Norbert_PROCA_SLSTTF_V2 error="database is locked (5) (SQLITE_BUSY)"
time=2024-07-29T21:31:13.863Z level=INFO msg=Rendering asset=peejay/gopro/_GoPro_sruba.stl img=32bd4027c8c059785e51c0515ef963b3.r.png asset="&{ID:32bd4027c8c059785e51c0515ef963b3 Label:0xc0003e30a0 Description:<nil> Path:0xc0003e3090 Root:0xc0003e3080 Extension:0xc0003e30b0 Kind:0xc0003e30d0 NodeKind:0xc0003e30c0 ParentID:0xc0000d2d00 Parent:0xc0000d2d00 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc000014bf0 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:14.578Z level=INFO msg=Rendering asset=peejay/gopro/_GoPro_sruba_sml.stl img=43f1faf375aa84b55c57b02f3d13fb2e.r.png asset="&{ID:43f1faf375aa84b55c57b02f3d13fb2e Label:0xc0017e5d20 Description:<nil> Path:0xc0017e5d10 Root:0xc0017e5d00 Extension:0xc0017e5d30 Kind:0xc0017e5d50 NodeKind:0xc0017e5d40 ParentID:0xc0000d2d00 Parent:0xc0000d2d00 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc0000152a8 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:15.160Z level=ERROR msg="failed to save render" module=process asset=dji_uchwyt-do-telefonu_v3 error="database is locked (5) (SQLITE_BUSY)"
time=2024-07-29T21:31:15.446Z level=ERROR msg="failed to save render" module=process asset=dji_uchwyt-do-telefonu_v4 error="database is locked (5) (SQLITE_BUSY)"
time=2024-07-29T21:31:15.605Z level=INFO msg=Rendering asset=peejay/gopro/gopro_58mm-filter-mount.stl img=0439ec3cfde1ed1bbb40e6633c825989.r.png asset="&{ID:0439ec3cfde1ed1bbb40e6633c825989 Label:0xc001829380 Description:<nil> Path:0xc001829370 Root:0xc001829360 Extension:0xc001829390 Kind:0xc0018293b0 NodeKind:0xc0018293a0 ParentID:0xc0000d2d00 Parent:0xc0000d2d00 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc000494300 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:16.752Z level=ERROR msg="failed to save render" module=process asset=dji_uchwyt-do-telefonu_v5 error="database is locked (5) (SQLITE_BUSY)"
time=2024-07-29T21:31:17.426Z level=INFO msg=Rendering asset=peejay/gopro/gopro_Lens_Cap_oczko.stl img=c6f325672eb0f0aff2667dae68acfdb4.r.png asset="&{ID:c6f325672eb0f0aff2667dae68acfdb4 Label:0xc000417b80 Description:<nil> Path:0xc000417b70 Root:0xc000417b60 Extension:0xc000417b90 Kind:0xc000417bb0 NodeKind:0xc000417ba0 ParentID:0xc0000d2d00 Parent:0xc0000d2d00 NestedAssets:[] Thumbnail:<nil> SeenOnScan:0xc00037ad38 Properties:map[] Tags:[] CreatedAt:0001-01-01 00:00:00 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC}"
time=2024-07-29T21:31:18.427Z level=ERROR msg="failed to save asset" module=process asset=folia-rolka-przedluzacz_v1 error="database is locked (5) (SQLITE_BUSY)"
time=2024-07-29T21:31:19.352Z level=ERROR msg="failed to save render" module=process asset=_GoPro_sruba error="database is locked (5) (SQLITE_BUSY)"
time=2024-07-29T21:31:20.492Z level=ERROR msg="failed to save asset" module=process asset=dji_uchwyt-do-telefonu_v4 error="database is locked (5) (SQLITE_BUSY)"
2024-07-29T21:34:47.563290649Z 2024-07-29T21:34:47.563290649Z 2024-07-29T21:34:47.563290649Z 2024-07-29T21:34:47.563290649Z id= remote_ip=192.168.3.4 host=192.168.3.4:8098 method=GET uri=/api/system/settings user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 status=404 error=code=404, message=Not Found latency=40728 latency_human=40.728µs bytes_in=0 bytes_out=24

[EduardoOliveira]

I forgot to document, the url should be ip:8000/lib, I just pushed a new version with a nav bar

[pejotigrek]

I forgot to document, the url should be ip:8000/lib, I just pushed a new version with a nav bar

thanks for the hint, I thought I need to use the mmp-ui container to get to the ui ;) ip:8000/lib worked fine :)
while I'm not a fan of the layout/design in v2 ["old one" seems to be much cleaner and not overloaded], I guess it's still a heavy work-in-progress stage, so there's no need to focus my feedback on that :)

regarding the structure - this probably is what I was hoping for when running the "regular" [not v2] version, meaning: the ui mirrors folder structure on disk. so it looks very promising, just maybe it needs to other distinction between files and folders [other than a "dir - dir" comment, but like I said - I guess layout/design is in early stage].

like many others, I've got a structure similar to this:

category/group	project	final files/folders
1. projects group
	project A
		version 1
		version 2
	project B
		version 1
		version 2
2. projects group
	...and so on.

not all projects/groups has all branches deep, but sometimes I need to add more just to know what is what ;)

my testing structure is in the attachement :)
test_library.zip

IMO it's represented in v2 much better [folders and subfolders] than in v1 [tags].

[EduardoOliveira]

Thanks a lot for the feedback, I'll I've that structure a spin soon. The dir - dir is actually, asset kind (dir, model, image, slice, ... Configurable in settings) - node kind ( file, dir, Bundle if is a zip 3mf rar, bundled if it's contained within a bundle and can be extracted) that will become some kind of icon at this point is mostly for tracking

[Kimbaras]

Sorry for the delay, I'll give the new version (v2) a try in the next days and I'll provide feedback from my side as soon as possible!

Thanks!

[Borbarad2k1]

I had similar problems like @pejotigrek . ;) Actually I could copy and paste @pejotigrek comment, every single bit.

The only thing I have noticed, is that you are flattening the content of compressed files. Is this deliberate?
If so: What happens, when there are two folder containing the same file? e.g: ( '/a/file.stl', '/b/file.stl' )

Looks promising. Thank you.

[EduardoOliveira]

Hi @Borbarad2k1, currently, yes its deliberate, it simplefies things. But thats the kind of feedback i'm looking. how would you see that working?
deep navigation of bundle files, keeping the folder structure, and then you decompress it, also keep the same folder structure?

Also I recoment docker pulling the V2 on every day of testing, since I've been doing some chages, for instance the UI is a bit less cluthered @pejotigrek xD

[Borbarad2k1]

Hi, @EduardoOliveira
when I heard about the feature I thought: "Great! After I organize the files in my project I can compress it and save some MBs of storages."

As a user I would expect a bundle file is like a folder and it behaves like this, including the folder structure. The only exception: I would not allow "manipulating" the bundle file, like creating folders, or copying/moving files INTO the bundle file and so on.

I can see that some people would like to have a decomrpess feature either during upload or later. For me personally, it's about saving storage space.

I understand, that sometimes it's necessary to decompress files for internal usage, like creating thumbnails or something similar. This could happen e.g. in a maintained cache folder.

[EduardoOliveira]

Currently I'm only extracting a single image from the bundle to make the thumbnail, and not processing any of the bundled assets, so no thumbnails for the contained assets, would it make sense to render the bundled assets? The generated assets will be a fraction of the bundle content. but still... this thread choses...

[Borbarad2k1]

Hm good question...
I would like to have an image of every asset, just like with the regular folder/assets. The image for a folder(regular or bundle) is either the first asset in the folder or can be chosen in the settings.

But maybe a user does not want MMP to scrape the content of the bundle?!
Use case: User just wants to extract a subset of files and will afterwards delete the bundle.

What about, asking the user when he accesses a bundle?
Generate metadata for this bundle?
Yes/No
[checkbox] Don't ask again for this bundle.

[EduardoOliveira]

There could be a global setting for precessing bundles by default, or a button by bundle to process individually. I'll go with it

[EduardoOliveira]

Update: now the bundles integrate better within the file three, you can choose or not to render assets within the bundle, and chose or not to extract the images to use as thumbnail, all of this is done in memory, so the models are not extracted during discovery, but you'll be able to download/see in the 3d viewer, a temp folder is being used to store this kind of transient data, so you can just clean up without breaking stuff. Also, internally I'm using a file system abstraction that allows me to treat zips and whatnot as normal fs, but, there are also drivers to treat S3, git and others in the same manner, this will not be treated as priority but shouldn't be hard to implement.

When you have the time, feedback is appreciated.

To add multiple libraries just add multiple file system entries in the config file, the /library as default still applies