[Hold for payment 2024-02-07] [$500] MEDIUM: Fix Google Docs copy/paste of images

[quinthar]

Strategy:
As a SaaS tool, we expect our customers to be using other best-of-breed SaaS products, especially Google Docs. Having a fantastic integration for data exchange with Google Docs is a way to stand out.

Problem:
At one point in the past, we were able to just directly copy/paste charts and image from Google Docs, Google Sheets, Google Slides, etc -- straight into NewDot. That doesn't work anymore.

Solution:
Add the ability to copy objects (not screenshots of objects, just the object itself) from Google Workspace into an Expensify chat, specifically:

• Google Sheet charts
• Google Slide images
• Google Doc images

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~0107c9e5dbbdb903ab
• Upwork Job ID: 1745258481290526720
• Last Price Increase: 2024-01-18
• Automatic offers:
• ikevin127 | Contributor | 28116109

[melvin-bot]

Triggered auto assignment to @conorpendergrast (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~0107c9e5dbbdb903ab

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @Santhosh-Sellavel (External)

[b4s36t4]

Proposal

Please re-state the problem that we are trying to solve in this issue.

MEDIUM: Fix Google Docs copy/paste of images

What is the root cause of that problem?

NA - Feature Request

What changes do you think we should make in order to solve the problem?

When we copy an image from google, google will write the HTML content to the clipboard where it will includes source of the image copied etc.

<meta charset='utf-8'>
<meta charset="utf-8"><b style="font-weight:normal;" id="docs-internal-guid-ef70d940-7fff-9d33-5271-c83bd573c118"><span
        style="font-size:11pt;font-family:'Open Sans',sans-serif;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span
            style="border:none;display:inline-block;overflow:hidden;width:624px;height:451px;"><img
                src="https://lh7-us.googleusercontent.com/vt_1xrWmyFRWKb9zB-uFaougvcrbPSm5hdPaKUMEOtcJuHwtwFAJ_ubht3BAFoaDn59-S5AAJcaEPE8RCnsayNNjcDjhhXZTjWO1_6lgQ-EYbKCnpqiy0Xz5DFod1mOn4CDQHFtElQd8em7G-DqilBk"
                width="624" height="451" style="margin-left:0px;margin-top:0px;" /></span></span></b>

We're handling the pasting of HTML here

App/src/components/Composer/index.tsx

Lines 226 to 241 in 8aec07c

	if (event.clipboardData?.types.includes(TEXT_HTML)) {
	const pastedHTML = event.clipboardData?.getData(TEXT_HTML);
	const domparser = new DOMParser();
	const embeddedImages = domparser.parseFromString(pastedHTML, TEXT_HTML).images;
	// Exclude parsing img tags in the HTML, as fetching the image via fetch triggers a connect-src Content-Security-Policy error.
	if (embeddedImages.length > 0 && embeddedImages[0].src) {
	// If HTML has emoji, then treat this as plain text.
	if (embeddedImages[0].dataset && embeddedImages[0].dataset.stringifyType === 'emoji') {
	handlePastePlainText(event);
	return;
	}
	}
	handlePastedHTML(pastedHTML);
	return;

which will check if the pasted content is an emoji or not.

We should be adding a new condition here which will check for the source of the img tag i.e if url contains googleusercontent that means image is coming form google sources.

Any alternative is welcome, for this point validation is done using URL from JS

Once we have the url of the image, using fetch we get the image and create a new File object and pass it to handlePasteFile callback inside Composer file.

App/src/components/Composer/index.tsx

Lines 233 to 238 in 8aec07c

	if (embeddedImages.length > 0 && embeddedImages[0].src) {
	// If HTML has emoji, then treat this as plain text.
	if (embeddedImages[0].dataset && embeddedImages[0].dataset.stringifyType === 'emoji') {
	handlePastePlainText(event);
	return;
	}

In the above block of code, we should be adding the following code.

try {
    // validate if the image source is a valid URL.
    const url = new URL(embeddedImages[0]?.src);
    // considering images from google, cause google will return the content as html
    if (embeddedImages[0].src && embeddedImages[0].src.includes('googleusercontent')) {
        handlePasteFileFromGoogleDoc(url.href);
        return;
    }
    // eslint-disable-next-line no-empty
} catch {}

const handlePasteFileFromGoogleDoc = useCallback(
    (source: string) => {
        fetch(source, {method: 'GET'}).then((response) => {
            response.blob().then((blob) => {
                const fileType = response.headers.get('content-type') ?? 'image/png';
                const extension = fileType.split('/').pop() ?? 'png';
                // here name is `pasted-file` because the URL we're fetching isn't allowing to fetch the filename due to some CORS related issues..
                // if we want to fetch the file we should be creating a proxy URL where we take the image source and fetch and return via API we're using
                // which will solve issue.
                // Anyway I don't consider we need that cause, we're not caring for the image (or atleast I feel that way.)
                const blobToFile = new File([blob], `pasted-file.${extension}`, {type: fileType});
                onPasteFile(blobToFile);
            });
        });
    },
    [onPasteFile],
);

The above code works for image copied through docs which are image URLs, but to manage the images copied into clipboard as base64 string also need to be handled.

• Read the base64 string.
• Parse the date-url string into a file.
• call the onPasteFile function with the converted file.

Also, since we're onto sheets when I copy the cells from sheets, it would be copied as text but I see whatsapp (atleast for excel) it copy as image.

To create the same effect, google sheets is sending us the HTML which we need to convert the html to into image.

• read the HTML string google-sheets-html-origin tag.
• create the dom, append the created dom to document.
• convert the element to canvas using html2canvas.
• make file using the the toDateURL function from canvas.

html2canvas doesn't work natively in react-native, I think we can leave this feature (sheets) with-in browser itself or we could create a backend where we render the HTML and send as the image.

Code changes.

function dataURItoBlob(dataURI: string): File {
    // convert base64 to raw binary data held in a string
    // doesn't handle URLEncoded DataURIs - see SO answer #6850276 for code that does this
    const byteString = atob(dataURI.split(',')[1]);

    // separate out the mime component
    const mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0];

    const extension = mimeString.split('/')[1];

    // write the bytes of the string to an ArrayBuffer
    const ab = new ArrayBuffer(byteString.length);

    // create a view into the buffer
    const ia = new Uint8Array(ab);

    // set the bytes of the buffer to the correct values
    for (let i = 0; i < byteString.length; i++) {
        ia[i] = byteString.charCodeAt(i);
    }

    // write the ArrayBuffer to a blob, and you're done
    const blob = new Blob([ab], {type: mimeString});

    const file = new File([blob], `pasted-file.${extension}`, {type: mimeString});
    return file;
}

Handle Copied cells from google sheets

const domparser = new DOMParser();
const dom = domparser.parseFromString(pastedHTML, TEXT_HTML);
const embeddedImages = dom.images;

// if pasted HTML is HTML from sheet (i.e sheet-table)
const sheetHTML = dom.getElementsByTagName('google-sheets-html-origin').item(0);

if (sheetHTML) {
    document.body.appendChild(sheetHTML);

    html2canvas(sheetHTML as HTMLElement).then((canvas) => {
        const file = dataURItoBlob(canvas.toDataURL('image/png'));
        onPasteFile(file);
    });
    document.body.removeChild(sheetHTML);
    return;
}

Handle images copied from sheets

// If a HTML has image.
if (embeddedImages[0]?.src) {
    const pastedFile = dataURItoBlob(embeddedImages[0].src);
    onPasteFile(pastedFile);
    return;
}

What alternative solutions did you explore? (Optional)

NA

Demo

Kapture.2024-01-11.at.09.32.38.mp4

[ikevin127]

Proposal

Please re-state the problem that we are trying to solve in this issue

The ability to directly copy/paste charts and images from Google Docs, Google Sheets, Google Slides straight into the chat.

What is the root cause of that problem?

Missing functionality.

What changes do you think we should make in order to solve the problem?

1. For copy & paste of files from Google Drive as attachments (drive.google.com/open?id= URL's):

(Requires Google API key)
Using Google Drive API on the front-end side we can check if a google drive file URL is pasted in the composer and add functionality that will extract the fileID and perform API call to download the file metadata from where we will do the following:

• get the file name and downloadLink
• perform final fetch where the actual file is downloaded
• transform file into blob using the original file name from the metadata
• pass the file to onPasteFile(file) in order to trigger the file management
• if fetch is successful the AttachmentModal will open with file preview and button to Send attachment to chat

2. For copy & paste of images from google docs and slides (googleusercontent.com URL's):

Check if clipboard data contains googleusercontent.com which is part of the image URL and if true:

• use domparser to extract embeddedImages from the parsed clipboard html data
• get the embeddedImage[0.src URL and fetch the file then transform response into blob
• pass the file to onPasteFile(file) in order to trigger the file management
• AttachmentModal will open with file preview and button to Send attachment to chat

3. For copy & paste of google sheet charts / images (base64 source):

Check if clipboard data contains base64 image source and if true perform the folowing actions:

• use domparser to extract embeddedImages from the parsed clipboard html data
• extract base64ImageContent and mimeType then make it into a blob
• pass the file to onPasteFile(file) in order to trigger the file management
• AttachmentModal will open with file preview and button to Send attachment to chat

Videos

MacOS: Chrome / Safari

724d150e-94ed-451c-a5b8-c9daa4fa74ac.mp4

[melvin-bot]

@conorpendergrast, @Santhosh-Sellavel Huh... This is 4 days overdue. Who can take care of this?

[conorpendergrast]

@Santhosh-Sellavel Could you take a look at the two proposals so far and confirm if we can move ahead with either of them? Thanks!

[melvin-bot]

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

[Santhosh-Sellavel]

@conorpendergrast I'll try to get a hold of it by tomorrow.

[quinthar]

@Santhosh-Sellavel tomorrow came and went; can you give a new ETA for when you can handle this?

[Santhosh-Sellavel]

Sorry for the delay, everyone.

Both @b4s36t4 & @ikevin127's proposal looks good here.

But @b4s36t4's proposals work well for Google Docs & Google Slides, but it doesn't work for charts copied from Google Sheets.

@ikevin127 I like your proposal, looks good to me as it addresses charts case from Google Sheets case as well.

[Santhosh-Sellavel]

@ikevin127's proposal looks good to me.
And their proposal Optionally includes solution for google drive also.

C+ Reviewed
🎀 👀 🎀

[melvin-bot]

Triggered auto assignment to @mountiny, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[b4s36t4]

@Santhosh-Sellavel as far as I see when I copy the charts from google sheets, It is coping the string (values) from the graph. Even in the google docs it's not working perfectly.

Having the google docs API to copy data from google docs etc wouldn't good I feel, in terms of UX as well since that require api calling and pricing is also we need to check.

[b4s36t4]

To fix the graph issue, I'm working on a alternate solution which should solve the issue (but might require a package to be installed).

[ikevin127]

@Santhosh-Sellavel PR #34971 is ready for review! 🚀

[b4s36t4]

haha, sorry if I was too intimidating, but the PR is literal copy of my proposal. Hoping this is taken care!!!!.

PS: I'm not expecting any share in bounty, but the efforts should be credited atleast.

[ikevin127]

Thank you for your contribution to this issue!

[mountiny]

Thanks to you both, it will go the other way around in future

[ikevin127]

Some manual testing using the external googleusercontent.com links (Docs / Slides images) did not pass on staging after deploy, I added more details in a PR comment here #34971 (comment).

TLDR: the link violates our Content Security Policy directive: "connect-src" because the https://*.googleusercontent.com is missing from the allowed links list, therefore the fetch fails.

I thought I should let the team handling this issue know about it in case they are not getting the PR notifications anymore.

cc @mountiny @Santhosh-Sellavel

[mountiny]

Created backend change to allow for this content-src https://github.com/Expensify/Cloudflare-Workers/pull/119

[quinthar]

Hi, why did @github-project-automation move to CRITICAL, I'm curious?

[quinthar]

Also, I think there's more work to be done here, though it's off to an awesome start! I am able to copy/paste a chart from Google Sheet directly into NewDot, which is great. However, this didn't work:

1. Open an empty Google Doc
2. Open any image from the internet (such as this cat)
3. Copy and paste that into the Google Doc, so now it's a Google Doc image object
4. Now select and copy that image from Google Docs
5. Correct: You should be able to paste it into a NewDot chat
6. Actual: When you try to paste it, nothing happens.

Reopening this issue to fix this. Thanks for the great work on this so far!

[conorpendergrast]

I tested out Google Slides, and that also works! So it looks like it's just Google Docs that's remaining (I've ticked off Sheets and Slides in the OP)

[ikevin127]

Appreciate the amazing feedback! 🚀

I'm having some trouble understanding how come the Google Docs scenario mentioned here #34306 (comment) doesn't work 🧐

Note: I'm pretty sure it's not a Google Doc permission related issue as the images / objects copied to the clipboard cannot be restricted even if the Google Doc file is Restricted.

We implemented 2 types of clipboard paste for this feature so far:

• base64 images / objects (charts) mostly found in Google Sheets
• googleusercontent.com links mostly found in Google Docs / Slides images

Since the scenario mentioned in the comment above matches the 2nd clipboard type, there shouldn't be any issue❓
Here's how it looks on my side following the steps:

MacOS: Chrome / Safari

Screen.Recording.2024-02-01.at.19.31.27.mov

cc @quinthar @conorpendergrast

[quinthar]

Wow, amazing video!

[quinthar]

I can't reproduce the issue anymore, not sure what I did. Closing again. Great work!!

[conorpendergrast]

Reopening to work out payment. Looks like the automation failed there, so I'll handle once the regression period is over (looks like that's 2024-02-07)

[conorpendergrast]

Payouts due:

• Contributor: $500 @ikevin127 - Contract
  Contributor+: $500 @Santhosh-Sellavel - via Manual request

Upwork job is here.

[Santhosh-Sellavel]

Requested on ND

[JmillsExpensify]

$500 approved for @Santhosh-Sellavel based on summary above.