Fix thumbs for new uploads to private repos #994

Quicksaver · 2018-01-08T17:25:25Z

Closes #993.

- Summary

When working on a private repo, new image uploads don't show the thumbnail until you reload the page, leading the user sometimes to think the upload failed because they see a broken thumbnail icon. The src url of new images don't have the token query parameter that is applied when the page loads.

This fix reloads the file list of the upload directory from the git API (as if it was first loading the media library), and fetches the url of the newly uploaded file from that list. I left the previous built-in-the-method url as a fallback, just in case.

Note: my fix seems the "simplest" in terms of what I could already find in the code, but if you think there's a better way to do it, I'm all up for it. 😄

- Test plan

Tested uploading an image, to a private repository, through the media library. The thumbnail appeared immediately as expected. Also tested in a public repo, all fine as before as well.

- Description for the changelog

(Media Library) Fix broken thumbnail when uploading an image to a private repository

- A picture of a cute animal (not mandatory but encouraged)

verythorough · 2018-01-08T17:29:11Z

Deploy preview for cms-demo ready!

Built with commit 0d14076

https://deploy-preview-994--cms-demo.netlify.com

tech4him1 · 2018-01-08T17:30:11Z

@Quicksaver Do the download_urls automatically have the tokens added to them?

Quicksaver · 2018-01-08T17:33:17Z

@tech4him1 it seems they do, they also come directly from the git API on load at least.

Quicksaver · 2018-01-08T17:35:32Z

PS not really sure why the build is failing, I can't tell locally as running lint on master also fails, but it builds just fine. Please let me know if there's something I can do on my end to fix that.

verythorough · 2018-01-08T17:43:01Z

Deploy preview for netlify-cms-www ready!

Built with commit 0d14076

https://deploy-preview-994--netlify-cms-www.netlify.com

tech4him1 · 2018-01-08T17:43:45Z

@Quicksaver Looks like the build just had a connection issue, should be fixed now.

tech4him1 · 2018-01-09T21:03:37Z

src/backends/github/implementation.js

+      let url = `https://raw.githubusercontent.com/${repo}/${this.branch}${path}`;
+
+      // Assets uploaded to private repos will need valid access tokens.
+      const files = await this.api.listFiles(this.config.get('media_folder'));


I think we should only get the file that we are actually wanting, not the entire media_folder again. This could be accomplished by making a function essentially like listFiles, except for requesting a single file instead of a directory.

Also, I do think double-checking the SHA here is a good idea, as we don't want to run into caching issues. When we were constructing the URL manually before, though, we definitely weren't worrying about caching, so maybe it's not really necessary.

@tech4him1

I think we should only get the file that we are actually wanting

I wanted to do that, but I couldn't find any API in the docs that could be used for that. The Contents API being used now, if pointed at a file rather than at a directory, will also retrieve the contents of that file. So we'd be transferring the whole image file twice, once just to get the metadata containing its download_url, and again later on to show the thumbnail (as it's a different url so it's not cached).

That API also has a 1 Mb limit, so it won't work for larger files, we'd need the Blobs API to retrieve those files, but that doesn't include the download_url.

As far as I can tell, this really seems to be the most direct way to get the tokenized download_url currently used to show the thumbnails. Those alternatives are probably useful for #990 though.

The only other option I can think of is super hacky - parse the token from the download_url of another image. 99.9% sure we just shouldn't.

So that aside, we should:

only take the approach provided here for private repos (check with repos/get)

hurry up and get thumbnail creation in place (Media library needs rendered thumbnails #946), and ensure thumbnails are well below 1mb, so we can stop doing this

erquhart · 2018-02-28T14:51:20Z

@Quicksaver this is better than nothing for private repos, we can still merge it if it's updated to only apply for those.

Quicksaver · 2018-02-28T14:56:39Z

@erquhart I've been wanting to do that but haven't had a chance to work on it at all since back then. 😭

erquhart · 2018-03-02T18:49:26Z

No problem! I'll leave it open for now in case you or someone else finds time to get it in.

robertgonzales · 2018-03-10T00:11:48Z

I'm seeing this issue, but not just for new uploads. No thumbnails load even with the token param.

erquhart · 2018-03-14T18:41:50Z

@robertgonzales that would qualify as a separate issue, can you open one with a repro case?

Quicksaver · 2018-03-16T17:07:25Z

@erquhart @tech4him1 re-fixed for private repositories only (finally!) 😄 I couldn't find anywhere else that was already checking if the repo is private, so I created a new method in the API script just for that, could be useful for other features down the line as well I guess.

tech4him1 · 2018-03-16T17:22:43Z

src/backends/github/implementation.js

+      const isPrivateRepo = await this.api.isPrivateRepo();
+      if (isPrivateRepo) {
+        const files = await this.api.listFiles(this.config.get('media_folder'));
+        for (let file of files) {


I think we could make this code a bit simpler by using find() instead of looping:

const file = files.find(f => (f.sha === mediaFile.sha)); if (file) { url = file.download_url; }

tech4him1 · 2018-03-16T17:24:24Z

src/backends/github/API.js

+  isPrivateRepo() {
+    return this.request(this.repoURL)
+      .then(repo => repo.private)
+      .catch(error => {


request() should catch itself on an API error -- we don't need to catch here unless there is a good reason.

I copied this from the hasWriteAccess() method above. Without knowing more about why that one is written like that, I couldn't tell you if it needs the catch or not, although yeah request() should probably catch it by itself. What do you think, should I keep it or remove it?

Let's go ahead and remove it, Benaiah agreed as well.

Quicksaver · 2018-03-19T15:54:12Z

@tech4him1 removed the catch().

Quicksaver · 2018-03-19T15:55:06Z

@tech4him1 and changed the loop into a .find() as well (almost forgot to mention).

tech4him1 · 2018-03-21T22:59:55Z

src/backends/github/implementation.js

+        }
+      }
+
+      return { id: mediaFile.sha, name: value, size: fileObj.size, url, path: trimStart(path, '/') };
    }


It looks like we're returning the file SHA now, instead of the response (commit) SHA. Maybe I just missed the reason for this, can you explain? 😄

@tech4him1 response.sha is undefined (yep, it was setting an undefined id before), at most it should have been response.object.sha. I set it to the file's sha instead to be more consistent with the other ids from when the media library initializes: https://github.com/netlify/netlify-cms/blob/master/src/backends/github/implementation.js#L94-L104

It's not really part of the original issue, true. But I thought since I was already in that very same line of code, I'd throw that in. 😅

@Quicksaver Sounds good, maybe it should be split into a seperate commit? (same PR is fine)

@tech4him1 makes sense, I split it into a separate commit.

Quicksaver · 2018-03-23T15:04:17Z

@tech4him1 Also, I just stumbled upon a redundant file.filter call in getMedia() in implementation.js (listFiles() in API.js already has an exact same one) so I went ahead and created a commit for it as well. Let me know if I'm filling this PR too much with non-relevant commits. 😁

tech4him1 · 2018-03-23T15:07:09Z

@Quicksaver Thank you! I think that the filter change should be in a seperate PR, since it's not really related (although it think it is pretty obvious 😄). It's up to you on the sha one, it could be lumped with either one, or seperate.

Quicksaver · 2018-03-23T15:28:12Z

@tech4him1 left this PR with only the thumb url fix, moved the other two fixes to #1189.

tech4him1

🎉

erquhart

Thanks!

buzzlightyear182 · 2018-03-31T19:30:03Z

Hello! I'm following the Quick Start Guide from the site so I'm using v 1.0.0 (https://unpkg.com/netlify-cms@^1.0.0) and I have the same issue with uploading images and thumbnails. I saw the PR was merged, but I'm not really sure how that works, is it there already in CDN?

tech4him1 · 2018-03-31T19:48:01Z

@buzzlightyear182 Make sure you have the caret in the link ^1.0.0, so that it automatically upgrades to the latest CMS version (v1.4.0). You can also try ^1.4.0 (still with the caret) to force the version. If you are are still having the problem, you can file a new issue with steps to reproduce.

buzzlightyear182 · 2018-04-01T07:23:34Z

Thanks @tech4him1 I've always had the caret in my file and did the ^1.4.0 too. I opened an issue with a screenshot of the error I got from console: #1220

This was referenced Jan 9, 2018

#67 Port fixes from FES marzeelabs/besugo#68

Merged

Update Netlify CMS marzeelabs/besugo#69

Closed

Merge develop 1.0.1 into master marzeelabs/besugo#70

Merged

tech4him1 reviewed Jan 9, 2018

View reviewed changes

tech4him1 mentioned this pull request Jan 9, 2018

Pull media library images through API #990

Closed

Quicksaver force-pushed the 993_thumbs_private_repo branch from c5ffb18 to d9b6cd1 Compare March 16, 2018 17:01

tech4him1 suggested changes Mar 16, 2018

View reviewed changes

tech4him1 reviewed Mar 16, 2018

View reviewed changes

Quicksaver force-pushed the 993_thumbs_private_repo branch from d9b6cd1 to 3063fc7 Compare March 19, 2018 15:51

tech4him1 requested review from Benaiah, erquhart and tech4him1 and removed request for erquhart March 21, 2018 22:54

tech4him1 reviewed Mar 21, 2018

View reviewed changes

tech4him1 added status: 2-code-review area: extensions/backends labels Mar 23, 2018

Quicksaver force-pushed the 993_thumbs_private_repo branch from 3063fc7 to 0a568e3 Compare March 23, 2018 14:59

Quicksaver force-pushed the 993_thumbs_private_repo branch from 0a568e3 to 6f34b07 Compare March 23, 2018 15:13

tech4him1 approved these changes Mar 23, 2018

View reviewed changes

tech4him1 mentioned this pull request Mar 27, 2018

uploaded filenames not being slugified #1196

Closed

erquhart approved these changes Mar 27, 2018

View reviewed changes

Fix thumbs for new uploads to private repos

0d14076

erquhart force-pushed the 993_thumbs_private_repo branch from 6f34b07 to 0d14076 Compare March 27, 2018 20:45

erquhart merged commit 2475af1 into decaporg:master Mar 27, 2018

Quicksaver deleted the 993_thumbs_private_repo branch March 28, 2018 08:09

erquhart mentioned this pull request Mar 28, 2018

UNPUBLISHED_ENTRY_PERSIST_FAILURE after enabling publish_mode: editorial_workflow #1187

Closed

This was referenced Apr 2, 2018

Regression: Image card with thumbnails not showing after upload #1220

Closed

Fix broken new media uploads in Git Gateway. #1221

Merged

Simple fix for new media uploads in Git Gateway. #1223

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix thumbs for new uploads to private repos #994

Fix thumbs for new uploads to private repos #994

Quicksaver commented Jan 8, 2018

verythorough commented Jan 8, 2018 •

edited

Loading

tech4him1 commented Jan 8, 2018

Quicksaver commented Jan 8, 2018

Quicksaver commented Jan 8, 2018 •

edited

Loading

verythorough commented Jan 8, 2018 •

edited

Loading

tech4him1 commented Jan 8, 2018

tech4him1 Jan 9, 2018

tech4him1 Jan 9, 2018 •

edited

Loading

Quicksaver Jan 10, 2018

erquhart Jan 20, 2018 •

edited

Loading

erquhart commented Feb 28, 2018

Quicksaver commented Feb 28, 2018

erquhart commented Mar 2, 2018

robertgonzales commented Mar 10, 2018

erquhart commented Mar 14, 2018

Quicksaver commented Mar 16, 2018

tech4him1 Mar 16, 2018

tech4him1 Mar 16, 2018 •

edited

Loading

Quicksaver Mar 16, 2018

tech4him1 Mar 16, 2018 •

edited

Loading

Quicksaver commented Mar 19, 2018

Quicksaver commented Mar 19, 2018

tech4him1 Mar 21, 2018

Quicksaver Mar 22, 2018 •

edited

Loading

tech4him1 Mar 23, 2018

Quicksaver Mar 23, 2018 •

edited

Loading

Quicksaver commented Mar 23, 2018

tech4him1 commented Mar 23, 2018 •

edited

Loading

Quicksaver commented Mar 23, 2018

tech4him1 left a comment

erquhart left a comment

buzzlightyear182 commented Mar 31, 2018

tech4him1 commented Mar 31, 2018

buzzlightyear182 commented Apr 1, 2018

Fix thumbs for new uploads to private repos #994

Fix thumbs for new uploads to private repos #994

Conversation

Quicksaver commented Jan 8, 2018

verythorough commented Jan 8, 2018 • edited Loading

tech4him1 commented Jan 8, 2018

Quicksaver commented Jan 8, 2018

Quicksaver commented Jan 8, 2018 • edited Loading

verythorough commented Jan 8, 2018 • edited Loading

tech4him1 commented Jan 8, 2018

tech4him1 Jan 9, 2018

Choose a reason for hiding this comment

tech4him1 Jan 9, 2018 • edited Loading

Choose a reason for hiding this comment

Quicksaver Jan 10, 2018

Choose a reason for hiding this comment

erquhart Jan 20, 2018 • edited Loading

Choose a reason for hiding this comment

erquhart commented Feb 28, 2018

Quicksaver commented Feb 28, 2018

erquhart commented Mar 2, 2018

robertgonzales commented Mar 10, 2018

erquhart commented Mar 14, 2018

Quicksaver commented Mar 16, 2018

tech4him1 Mar 16, 2018

Choose a reason for hiding this comment

tech4him1 Mar 16, 2018 • edited Loading

Choose a reason for hiding this comment

Quicksaver Mar 16, 2018

Choose a reason for hiding this comment

tech4him1 Mar 16, 2018 • edited Loading

Choose a reason for hiding this comment

Quicksaver commented Mar 19, 2018

Quicksaver commented Mar 19, 2018

tech4him1 Mar 21, 2018

Choose a reason for hiding this comment

Quicksaver Mar 22, 2018 • edited Loading

Choose a reason for hiding this comment

tech4him1 Mar 23, 2018

Choose a reason for hiding this comment

Quicksaver Mar 23, 2018 • edited Loading

Choose a reason for hiding this comment

Quicksaver commented Mar 23, 2018

tech4him1 commented Mar 23, 2018 • edited Loading

Quicksaver commented Mar 23, 2018

tech4him1 left a comment

Choose a reason for hiding this comment

erquhart left a comment

Choose a reason for hiding this comment

buzzlightyear182 commented Mar 31, 2018

tech4him1 commented Mar 31, 2018

buzzlightyear182 commented Apr 1, 2018

verythorough commented Jan 8, 2018 •

edited

Loading

Quicksaver commented Jan 8, 2018 •

edited

Loading

verythorough commented Jan 8, 2018 •

edited

Loading

tech4him1 Jan 9, 2018 •

edited

Loading

erquhart Jan 20, 2018 •

edited

Loading

tech4him1 Mar 16, 2018 •

edited

Loading

tech4him1 Mar 16, 2018 •

edited

Loading

Quicksaver Mar 22, 2018 •

edited

Loading

Quicksaver Mar 23, 2018 •

edited

Loading

tech4him1 commented Mar 23, 2018 •

edited

Loading