Continuous Export of Dip & Strike

[chris-traxler]

Dip & strike measurements should be continuously exported into a GeoJSON file. When a point of the measurement is changed or added, PRo3D should update the GeoJSON or XYZ SCY file on disk automatically.

[ThomasOrtner]

Config/Data Management has a checkbox turning on geojson auto export an. Once a geoJSON is exported it gets continously updated with every finished dip and strike measurement.
@AndrewAnnex please comment if you see problems with our assumptions ... only dip and strike, update only on finished annotation.

there will be a release soon where you can try this out

[AndrewAnnex]

Hey @ThomasOrtner, so what I had in mine was the continuous export of the XYZ data, not only the strikes and dips. That is all points in 3d space extracted from the Meshes given the user's trace drawn in Pro3d, essentially the exact input to the regression performed within Pro3d to get the strike and dip. It may be helpful to add a hash of the object to the geojson feature attributes to help external scripts judge faster when things change or not.

Hopefully the file can be a geojson formatted as line delimitated (https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON), such that each trace/line in pro3d only updates that particular line in the file. If I have 10 strike dip measurements, and update only 1 of them, I still expect all 10 to be exported into the file even if there were no updates to the other 9 geojson lines.

If you mean that each trace is exported to individual files (as chosen by the user), and the individual files are continuously updated then I think that is sufficient to satisfy my needs/concerns from the paragraph above.

I'll have to see what you mean about the "update only on finished annotation", my needs are for this file to be updated any time a vertex is added, removed, or moved in the viewer, or when other pertinent settings like the density of the interpolation are changed (how many points of elevation are interpolated between vertices). I do not expect the file to update while I am dragging a vertex around, but only when I stop moving a given vertex or change relevant settings that would trigger the regression to be recomputed within pro3d.

I'll check out the new release when it's available.

Thanks! -Andrew

[haraldsteinlechner]

Hi Andrew,

thanks for the update.

Updating the file after each action is actually no problem (and in fact it actually works this way already). Anyhow, currently we just "overwrite" the file. Actually, i was expecting you to use some sort of FileSystem watcher to trigger updates on your side - but did not check back on it.

That said, i think your delimited json idea seems much better and in fact should not be much work.
To make it clear, actually you only need dip and strike annotations or all annotations in general?

We will discuss the broader setting again next week.

Cheers,
Harald

[haraldsteinlechner]

After discussing this a bit, it boils down to:

• create a new menu item for automatic export (as opposed to overwriting the last export when auto-export is enabled), for both xyz and normal
• add the 'working annotation' - this is the one not yet added to the set of final annotations to the exported set (properly structured beside the already completed annotations)
• compute the dip and strike results
• and append it to the file

what do you think @ThomasOrtner

[AndrewAnnex]

@haraldsteinlechner : Yes the filesystem watcher idea is what I was thinking, either as a single delimited json file or multiple files within a folder. It is much easier if there is just one file, and some precomputed hashes are added to allow clients (ie a python process) to assess quickly which features are actually updated, so overwriting the file is fine.

The more critical thing is the array of 3D points (XYZ) must be included in these export, not just the final strike and dip. Any and All annotations (names, notes, etc) should be included.

[haraldsteinlechner]

@AndrewAnnex we are just collecting what to really put into the file (now, all points are continuously written to the file). I was just wondering - do you need the dip and strike results as such (the plane for example) - i'm asking because we here have some "similar" things in attitude export which could be aligned etc

[haraldsteinlechner]

soo...

the new proposal looks as such:

Disable/Checking is here in the config panel:

Currently we export all visible annotations, their geometry their unique ID and color in the properties as line-seperated json:

{ "geometry": { "coordinates": [ [ [ -2486975.68951201, 2288923.53825004, -275793.212054538 ], [ -2486975.58883715, 2288923.62337435, -275793.278432393 ], [ -2486975.58811196, 2288923.64996945, -275793.440584449 ], [ -2486975.59470039, 2288923.58002145, -275793.079110164 ], [ -2486975.51574832, 2288923.62846411, -275793.069024277 ] ] ], "type": "Polygon" }, "properties": { "color": "[0, 0, 139, 255]", "fullHash": "%ddjp%fc%3ds%d3%08%1a%9d_%c7E%0f%a9%8e", "hash": "%db%bf%1a%04%01B%da%b5T%b6%81%be%aaR%8d%91", "id": "5555dc28-6b7f-45a8-8c63-9fce9348177e" }, "type": "Feature" }
{ "geometry": { "coordinates": [ [ [ -2486975.19125822, 2288923.83576994, -275792.866728204 ], [ -2486974.9391638, 2288924.07311295, -275792.997416629 ], [ -2486974.947619, 2288924.0302327, -275792.786482854 ] ] ], "type": "Polygon" }, "properties": { "color": "[0, 0, 139, 255]", "fullHash": "%84%16%60%7cA%9e%b1%85%e7%e4W%0e6%a8%99%a6", "hash": "%af%0a%a4%c506%2f%7fM1%e9%14%01%ea%26K", "id": "cde34e1f-a043-49ec-a611-47b13ba0255e" }, "type": "Feature" }

Notable, i'm using URL encoding to represent the hash since with Base64 there are backlashes which might be uncool to handle. We can swtich to base64 if you want.

There are two hashes. fullHash represents a precise hash of the internal structure which contains all properties changeable in pro3d. This one could be useful in the future when working with a pro3d rest api or something. hash is just the hash of the annotation json value.

Similarly to color, we can add stuff as you wish. Here is the export code:

PRo3D/src/PRo3D.Base/Annotation/Exporters/GeoJSON.Export.fs

Line 121 in 609a5cf

("id", annotation.key.ToString())

Export is triggered for all interactions except for internal stuff, therefore the current active annotation is virtually finished and exported as usual:

PRo3D/src/PRo3D.Core/Drawing/Drawing-App.fs

Line 528 in 609a5cf

exportGeoJsonStream true bigConfig smallConfig artificiallyFinishedModel path

please let me know if the current design fits your needs and what to include additionally in the exported json.

[AndrewAnnex]

@haraldsteinlechner sorry for the delay, this is exactly what I had in mind regarding the exports. Thanks for working on this so quickly, and I think it is enough for me to put together a proof of concept on my end once I can get a working executable and have a day to tinker.

I think for the hashes I was only thinking about hashing the xyz array to start, the use case being someone like me interactively adjusting the a group of lines in Pro3D one-at-a-time, and the hashes are precomputed by Pro3d to make it easier for the external tool to only recompute statistics on the updated line. I think it would be nicer if they were base64 encoded as well. That said, I think the full hash is likely sufficient if that is computed for the whole object

Reading back to what you said about attitude export, yes including all the strike dip information and related statistics (errors, true thickness, etc) would be great to include with this as well, although is a secondary priority for me at the moment. I'd say adding all of those properties to the feature attributes in the continuous export would be a good addition (minus the x,y,z points which I think would be redundant, unless they are not?).

As an aside, how does Pro3d sample the mesh given a sampling line for the strike dip? If additional sample points are defined between vertices would those xyz points be included in this export? Does the user define the sampling density between vertices?

[haraldsteinlechner]

Hi Andrew,

many thanks for your review of the proposal and the discussion.

ad hashes) i added a property 'geometryHash' which exactly matches your argumentation. You are the only user of this feature so we are fine to tune this. also it is just minor changes to the export code as can be seen here:

PRo3D/src/PRo3D.Base/Annotation/Exporters/GeoJSON.Export.fs

Line 137 in c7b7e51

("fullHash", pickler.ComputeHash(annotation).Hash |> encode |> Json.String)

also we switched to base64 encoding.

ad attitude et al) actually in our discussion we found that completely moving to geojson would be a good approach in the long-term. Also, we are currently discussing a REST api for PRo3D. Do you think this could be useful for your current and future tasks as opposed to using file for communication?

ad sampling) yes afaik the number of sample points can be chosen by the user and a intersection method computes the precise vertices but maybe @ThomasOrtner has more insight here.

ad testing) we are working a new release today.

[AndrewAnnex]

Hi Herald,

geojson is nice because it is human read-able and is well supported, however the geojson that pro3d will be exported will not technically be valid geojson as the coordinates are not WGS84/EPSG4326 (https://datatracker.ietf.org/doc/rfc7946/). Exporting data in the correct coordinates is critical for correct interoperability with other clients so some thought should be put into making it easy for clients to assign the correct projection to the data. I'd say it's not worth it to convert the data to EPSG4326/WGS84 for clients/users like me, so whatever coordinate system pro3d is currently using is likely sufficient/would only need slight tweaks to make the export of the crs information work well. If that exported crs can be a WKT2 CRS string that would be great, maybe as another file next to the continuous export file on the filesystem.

[haraldsteinlechner]

hi thanks for the info. we basically have coordinate transforms at hand. anyhow i'm not master of coordinate systems. i think we will discuss this one given your information.

i suppose you are waiting for the mac version, right?

[ThomasOrtner]

red: "as GeoJSON" produces longitude / latitude / altitude triples on the bases of the IAU Mars ellipsoid. This should be the WGS84 equivalent on Mars, as far as I understand it.

yellow: "as GeoJSON xyz" produces Euclidean coordinates, i.e. position vectors with respect to an origin situated in the center of Mars. Daven Quinn needs these data in Attitude for plane fitting

[AndrewAnnex]

Yes I'm waiting for the macos version but I see that #194 is making progress towards that, I was able to successfully compile/build pro3d using dotnet/build.sh but I am unfamiliar with what to do to run the compiled version, maybe the steps in #194 will make that clear but it would be good to update the readme with those steps. I've programmed in scala before so F# isn't too intimidating a language but I'm just unfamiliar with the workflow with dotnet in order to compile/run from the source code on macos/linux.

@ThomasOrtner interesting, I definitely do not want to fit strike/dips to the longitude/latitude values so xyz is more correct but I may need additional information in order to be able to convert the coordinates into other reference frames/coordinate systems later, but I'll open another issue when I get to that point.

[haraldsteinlechner]

hi andrew. i updated the instruction for running from source to make it clearer- would be cool if you can also run from source since this way we can also create a special branch for you with fast turnaround times aside of the release track.

is your mac apple silicon based (asking because the current pro3d version runs as x64 transpiled but native m1 is almost ready)?

[AndrewAnnex]

My MacBook Pro is a 2017 model, so not a m1 but thanks for asking. I’ll try running from source soon and report back

…

-Andrew Annex

On Apr 23, 2022, at 12:33 AM, Harald Steinlechner ***@***.***> wrote:  hi andrew. i updated the instruction for running from source to make it clearer- would be cool if you can also run from source since this way we can also create a special branch for you with fast turnaround times aside of the release track. is your mac apple silicon based (asking because the current pro3d version runs as x64 transpiled but native m1 is almost ready) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[AndrewAnnex]

hey @haraldsteinlechner I am running into an issue with those commands, looks like a dependency shared library is not being built/included:

 0: initializing aardvark .................................... [0.70x] 3.478 s
 0: [CooTransformation] installing CooTransformationDefault.trl
 0: [CooTransformation] installing ExoMarsRover_VariousCameras.xml
 0: [CooTransformation] installing GraphObject3dDefault.trl
 0: [CooTransformation] installing HiRISE.txt
 0: [CooTransformation] installing InstrumentPlatformsDefault.trl
 0: [CooTransformation] installing InternationalisationRegistryDefault.trl
 0: [CooTransformation] installing pck00010.tpc
 0: [CooTransformation] installing ProgressDefault.trl
 0: [CooTransformation] installing M20_waypoints.json
 0: [CooTransformation] initializing at /Users/andrew/.config/Pro3D/JR/CooTransformationConfig, logging to /Users/andrew/.config/Pro3D/JR/logs
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'CooTransformation.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libCooTransformation.dll, 1): image not found
   at JR.CooTransformation.Init(String configDir, String logDir)
   at PRo3D.Base.CooTransformation.initCooTrafo(String appData) in /Users/andrew/SideProjects/PRo3D/src/PRo3D.Base/CooTransformation.fs:line 86
   at Program.main(String[] argv) in /Users/andrew/SideProjects/PRo3D/src/PRo3D.Viewer/Program.fs:line 191

[haraldsteinlechner]

Hi Andrew, thanks for checking. in the features/autodeploy we fixed this issue.

[AndrewAnnex]

@haraldsteinlechner okay pulling again fixed this (comment edited), thanks!

[haraldsteinlechner]

this is strange since this content should be copied beside the executeable https://github.com/pro3d-space/PRo3D/tree/master/lib/Native/JR.Wrappers/mac regarding instrumentstuff this should not be a problem (at this stage) since this is optional (at least it should start) and loading is under try catch. could you copy it manually? if it still fails we need to look into the dylib... also today we got a CI built mac version working (basically works but has icon/console window issue, will summarize in an issue) thanks, harald Andrew Annex ***@***.***> schrieb am Mo., 25. Apr. 2022, 23:06:

…

@haraldsteinlechner <https://github.com/haraldsteinlechner> I think I am still running into this issue when I pulled a few minutes ago, so I think I am blocked by #196 <#196>. I'll try the latest builds on windows but it would be nice to have parity between the platforms — Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD5KAKYBKSZ2NYXQBFSCYDVG4CO5ANCNFSM5SPR5JYA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AndrewAnnex]

@haraldsteinlechner okay I managed to test things out on windows for the latest beta/alpha/test release. It looks like I am running into some issues:

1. continuous export does not export all vertices sampled between the vertices (viewpoint and sky). This is critical functionality for external projects.

2. I cannot move vertices once placed for a DnS. Ideally I should be able to move the vertices by selecting them once the line is finalized and the sample points (viewpoint or sky) would be recomputed as I drag a vertex around. I may have figured this out before so clearer explanations for how to do that in the docs would be helpful

3. planes need to have configurable opacity

4. planes should also show normal vector arrow, or rather not show arrow in dip direction given plane projection is already illustrating the direction pretty well.

I also ran into an issue in which the click placement of the DnS line significantly differed from the placement of vertices on the mesh. I think this was after I changed the explore center in arcball mode. I closed pro3d and opened a different mesh and the issue went away, so I think it is something related to the view center.

[AndrewAnnex]

@haraldsteinlechner okay after looking at the attitude export there are some duplicated/complimentary features here, I think the continuous export of geojson could be effectively merged with the "attitude" export, where most of the attitude fields (strike, dip, rake, center, hyperbolic_axes, axes, centered_array, etc) would be put into the geojson properties associated to the geojson feature. Alternatively, you could update the export of attitude to include the coordinates in the Mars CRS as another attribute, but I think it makes more sense to export all of the "attitude" attributes as geojson feature properties

[ThomasOrtner]

Unfortunately, (2), (3), and (4) are features that are currently not implemented in PRo3D and should be addressed in two separate issues (we can do (3) and (4) in one 'improved dip and strike visualization' issue)

(2) is more complicated as it involves the 3D manipulation of a vertex along a high-resolution mesh. Editing of annotations has been in our backlogs for several years now but has always been postponed in favor of other features.

[AndrewAnnex]

@ThomasOrtner maybe (2) could be implemented by instead making the individual vertices selectable (like is done in the PickAnnotation action). If the user selects a vertex and then clicks another location on the mesh a new vertex could be created and replace the selected vertex as a simpler fix. Although I could imagine a number of ways that could simplify implementing a more visual/interactive method.

[ThomasOrtner]

True, that would be much simpler. Also adding and removing. At the moment you can just use backspace to remove the last picked point in an annotation you are currently drawing, while Esc cancels it, if that helps at least a bit.

[haraldsteinlechner]

Thanks andrew for your insightful feedback regarding numerous active topics! The simple approach would not be too hard. Speaking for me, i'm currently on longer term vacation but will take another run next week maybe together with @RebeccaNowak and @ThomasOrtner.

[AndrewAnnex]

great, thanks @haraldsteinlechner, I am also working towards understanding the codebase better to begin contributing my own changes, although I have no estimate of how long it will take me.

[ThomasOrtner]

Hey @haraldsteinlechner , what is the status here? You are back from your vacation ;)