Add torch frontend grid_sample and test function

[Deepdive543443]

Close #22307

[github-actions]

Thanks for contributing to Ivy! 😊👏
Here are some of the important points from our Contributing Guidelines 📝:
1. Feel free to ignore the run_tests (1), run_tests (2), … jobs, and only look at the display_test_results job. 👀 It contains the following two sections:
- Combined Test Results: This shows the results of all the ivy tests that ran on the PR. ✔️
- New Failures Introduced: This lists the tests that are passing on main, but fail on the PR Fork. Please try to make sure that there are no such tests. 💪
2. The lint / Check formatting / check-formatting tests check for the formatting of your code. 📜 If it fails, please check the exact error message in the logs and fix the same. ⚠️🔧
3. Finally, the test-docstrings / run-docstring-tests check for the changes made in docstrings of the functions. This may be skipped, as well. 📚
Happy coding! 🎉👨‍💻

[Deepdive543443]

Hi @Sam-Armstrong The issues was happened because I used an ivy boolean array to mask the grid. It will throw an error when the grid input is a torch Tensor, not ivy array.
I updated the code to transfer the zeros_mask into native array in the last commit. It should fix the issue. But I need to remove the @to_ivy_arrays_and_back decorator, otherwise I cannot transfer the function output to native torch array and feed it into torch.allclose(). Not sure about the reason of this.

[Sam-Armstrong]

Sorry, here's a better example that's working:

import ivy.functional.frontends.torch as torch_frontend
import torch
import numpy as np

if __name__ == '__main__':
    inp = torch.randn((3, 3, 4, 4))
    grid = torch.randint(0, 4, size=(3, 4, 4, 2), dtype=torch.float32)

    mode = 'bilinear'
    padding = 'reflection'

    result_orig = torch.nn.functional.grid_sample(inp.clone(), grid.clone(), mode=mode, padding_mode=padding, align_corners=False)
    print('native fn result shape', result_orig.shape)
    
    result_new = torch_frontend.nn.functional.grid_sample(inp.clone(), grid.clone(), mode=mode, padding_mode=padding, align_corners=False)
    print('frontend fn result shape', result_new.shape)

    print(np.allclose(result_orig, result_new.ivy_array.to_numpy(), atol=1e-3))

I don't understand why grid is a torch.Tensor at that point though, shouldn't it have been converted into an ivy array?

[Deepdive543443]

Hi @Sam-Armstrong So surprise to have you around this early :) Do you think this is a bug on @to_ivy_arrays_and_back decorator? Or should I covert them to ivy array in manually?

[Sam-Armstrong]

We can actually resolve that problem by fixing the minimal example to pass ivy arrays, so we can (and should) remove that ivy.native_array line. I've tried one of the test cases which is failing in pytest, and it seems to be passing fine in the minimal example 🤔 so I'll talk to someone from the testing team about that to see if they know what's going on. By the way, do you know why Exception occured: list index out of range is printed out several times when running this? Thanks!

[Sam-Armstrong]

Suggested change

zeros_mask = ivy.native_array(zeros_mask)

We can remove this now

[Sam-Armstrong]

Suggested change

# @with_unsupported_dtypes({"2.0.1 and below": ("float16", "bfloat16")}, "torch")

[Sam-Armstrong]

Suggested change

# x *= 0

[Sam-Armstrong]

Corrected minimal example:

import ivy.functional.frontends.torch as torch_frontend
import torch
import numpy as np
import ivy

if __name__ == '__main__':
    ivy.set_backend('numpy')
    
    inp = torch.randn((3, 3, 4, 2))
    grid = torch.randint(0, 4, size=(3, 4, 4, 2))

    mode = 'nearest'
    padding = 'border'
    align_corners = True

    result_orig = torch.nn.functional.grid_sample(inp.clone(), grid.clone(), mode=mode, padding_mode=padding, align_corners=align_corners)
    print('native fn result shape', result_orig.shape)
    
    result_new = torch_frontend.nn.functional.grid_sample(ivy.array(inp.clone().detach().numpy()), ivy.array(grid.clone().detach().numpy()), mode=mode, padding_mode=padding, align_corners=align_corners)
    print('frontend fn result shape', result_new.shape)

    print(np.allclose(result_orig, result_new.ivy_array.to_numpy(), atol=1e-4))

[Deepdive543443]

Afternoon @Sam-Armstrong I just added the request change. Would you like share to more details about the test cases that gave you Exception occured: list index out of range? I didn't find how to reproduce this error.
About the pytest part, it looks like it calls a Tensorflow ground truth function with behaviour that doesn't match with PyTorch. We might need to hear from testing team about this issue.
Another thing I want to note is I usually use randn() to generate grid because PyTorch expect most of the grid input in range between -1 and 1(some of them outside this range would be padded). Using randint() should be fine but I found it makes the test fail when doing 3D grid sampling🤔I'm looking into it

Update: Looks like this is because of PyTorch grid_sample() uses upper round, so they seems to have different behaviour when the coordinate is a x.5 values. Applying a ceiling on all x.5 values seems fix this issues.

[Deepdive543443]

Hi @Sam-Armstrong Hope you are doing well this week. Did you get update from test team? From this week, I would need to move most of my effort to graduation. I might response slower than usual, but will try my best on get this PR done ;)

[Sam-Armstrong]

I think @sherry30 is looking into this, not sure if there is any update? No worries at all, best of luck with your graduation work also @Deepdive543443 😁

[sherry30]

Hi @Deepdive543443 @Sam-Armstrong ,
Sorry it took me a while to debug this but I think I've figured out the problem.
Your frontend implementation inplace updates grid array. So in our testing pipeline we first execute the frontend function, that changes the grid array, and then when the torch's native function is called, it produces different results because of the modified grid array.
Your function shouldn't inplace update the argument.
You can try this example out:

import ivy.functional.frontends.torch as torch_frontend
import ivy

if __name__ == '__main__':
    ivy.set_backend('numpy')

    inp = ivy.random_uniform(shape=(2, 2, 1, 6))
    grid = ivy.random_uniform(shape=(2, 2, 1, 6))

    mode = 'nearest'
    padding = 'border'
    align_corners = True
    print(grid)
    result_new = torch_frontend.nn.functional.grid_sample(inp,
                                                          grid, mode=mode,
                                                          padding_mode=padding, align_corners=align_corners)
    print(grid) # different from before

[Deepdive543443]

Hi @sherry30 @Sam-Armstrong After adding copy and type casting I think the function could pass three of the test for now. The remain issues are:

• Tensorflow testing showing a KeyError: 'cpu' error. It seems happened some paddle testing before, but I didn't find how to deal with it in Tensorflow.
• ivy.pad() seems no working when testing for paddle
• Pre-commit's auto-formatting would throw functions everywhere, so I haven't applied it yet

It looks like not just my implementation of grid_sample(), some others test function under test_vision_functions.py are not passing all the testing as well. Is that necessary to make sure all the testing could pass?

[Sam-Armstrong]

Thanks @Deepdive543443, that's great! I don't think these test errors are too much of a big deal - the other backends clearly show that the function is working correctly, so I think we can address these failures later. I've left a couple of comments with things we need to change before we merge, please could you take a look? And thanks for the great work!

[Deepdive543443]

Thank you @Sam-Armstrong and @sherry30 for supervising and guiding. I still not sure about if pre_commit is working correctly, so I placed all functions to the bottom of file after formatting. Hopefully it's good to go :)

[Sam-Armstrong]

@Deepdive543443 I'll just check whether the tf and paddle test failures are ok with Sherry (rather than needing to fix them for this PR), then should be good to merge 😁

[Sam-Armstrong]

Merged! Thanks for the great work you put into this PR @Deepdive543443!

[Deepdive543443]

Thanks @Sam-Armstrong and @sherry30 Finally it is done. Learned a lot from this task :)