Automatically keep track of transformations

[RandomGamingDev]

Increasing access

This would mean that transformations can be fully managed by p5.js, which not only means increased ease of control for the user, but also not having to have 2 sets of data for the same thing, one on p5.js's side and one on the user's side.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

Feature request details

While this is already possible in P2D mode with drawingContext.getTransform(); it doesn't apply to WebGL which doesn't use CanvasRenderingContext2D. Also, the transform returned with drawingContext.getTransform(); is a transformation matrix, which many people won't know how to use.

The way this would be done is through an official p5.js functions that return the transformation matrix, transform or untransform points, or return attributes that are decided as essential (the rest can be taken from the transformation matrix).

[davepagurek]

Related, this PR #7113 is adding support for worldToScreen -- I think in a similar format, screenToWorld would be useful. Direct access to matrices would help for more advanced usage but, as you mention, not every user will be familiar with how to use those. It'd be useful to have a stable API for that for library authors though. For WebGL, it's a little less clear what should be included in the matrix (just model? model + view?) so it might be worth adding some optional parameters like { includeCamera: true } to whichever methods we add.

[RandomGamingDev]

For WebGL I'd say to go with the multiplied result of the model, view, and projection matrices that are typically used for vertex shader transforms.

Also, although the people who use the matrix directly would be more technically inclined, it'd still be best to make things consistent across both the P2D and WebGL renderers by doing the same for P2D, both for the functions and returned matrix except with locked values for things like the z axis which are technically just constant for P2D.

[Garima3110]

I've implemented a screenToWorld() function that converts screen coordinates back to world coordinates for both 2D and WebGL contexts. The function uses inverse transform matrices to revert back to world space, making it versatile across 2D and 3D contexts.

p5.prototype.screenToWorld = function(screenPosition) {
  const renderer = this._renderer;
  if (renderer.drawingContext instanceof CanvasRenderingContext2D) {
    // Handle 2D context
    const inverseTransformMatrix = new DOMMatrix()
      .scale(renderer._pInst.pixelDensity())
      .multiply(renderer.drawingContext.getTransform().inverse());
    const worldCoordinates = inverseTransformMatrix.transformPoint(
      new DOMPoint(screenPosition.x, screenPosition.y)
    );
    return new p5.Vector(worldCoordinates.x, worldCoordinates.y);
  } else {
    // Handle WebGL context (3D)
    const normalizedDeviceCoordinates = new p5.Vector(
      (screenPosition.x / this.width) * 2 - 1,
      (screenPosition.y / this.height) * -2 + 1,
      screenPosition.z || 0
    );

    const inversePMatrix = renderer.states.uPMatrix.invert();
    const inverseMVMatrix = renderer.calculateCombinedMatrix().invert();

    const cameraCoordinates = inversePMatrix.multiplyPoint(normalizedDeviceCoordinates);
    const worldCoordinates = inverseMVMatrix.multiplyPoint(cameraCoordinates);

    return new p5.Vector(worldCoordinates.x, worldCoordinates.y, worldCoordinates.z);
  }
};

The calculateCombinedMatrix() method that I wrote for the worldToScreen() method here :

p5.js/src/webgl/p5.RendererGL.js

Line 938 in acb5ac4

calculateCombinedMatrix() {

could also be enhanced with an optional camera transformation parameter as suggested by @davepagurek .
Let me know if I can proceed with this and submit a PR for the issue, @RandomGamingDev .
Thankyou!

[RandomGamingDev]

Let me know if I can proceed with this and submit a PR for the issue, @RandomGamingDev .
Thankyou!

It looks great! You should proceed with the issue.

After that, all that'll be left is returning attributes individually.

[Garima3110]

It looks great! You should proceed with the issue.

Thanks, I'll submit a PR soon!

[Garima3110]

I apologize for the delay in working on this issue due to some health issue, just in my recovery phase will get back to working on this super soon, till then If anyone wants to work on this, you can, I would try giving my reviews for that too.
Thankyou!