Get Shopify connector on Cloud #8633

Phlair · 2021-12-08T17:17:26Z

think the issue is that we need to redirect the user to {shop_id}.shopify.com for consent instead of whatever we do now, this was recently added as part of https://github.com/airbytehq/airbyte-internal-issues/issues/322. So there will be a java component here as well and a python/connector component to modify the spec to use the new oauth spec

Based on priorities here, key targets are Google Sheets and Shopify. Other connectors will be benched for now.

sherifnada · 2021-12-14T18:15:40Z

Update Python Spec, see spec reference for help demoing if useful and oauth implementation guide
Add Java oauth flow implementation (🎉 Source Shopify: implement Oauth2.0 for Airbyte-Cloud #9591)
Java part is merged into master
update the version of airbyte-cloud, according to the new OSS Release version.
unblock shopify in UI for cloud: 🐛 Airbyte-webapp: Unblock shopify in connectors list in UI for Cloud #9691

waiting for New OSS Release

deploy/test in dev
Fulfill shopify application requirements, see Shopify OAuth Developer App on lastpass for login/url
acquire permissions to deploy/test on prod
deploy/test in prod

sherifnada · 2022-01-20T15:06:58Z

@bazarnov do you need something from me to update the version available in airbyte-cloud to get the oauth application approved from shopify?

bazarnov · 2022-01-20T15:17:36Z

For now we're good to go without you involved in this.
These steps should be performed:

Release the Shopify to dev-could, so the Shopify team could test the OAuth Flow, as they want (it's required to get the public app, without the public app, the customers will not get it through)
After successful test (fingers crossed), we submitting the "draft" app to obtain the "public" status.
Finally we can release Shopify connector to Cloud version prod.

All the rest: python, java parts are ready and merged into master already.
Currently testing on dev-1, how it goes with sandbox account.

sherifnada · 2022-01-20T16:02:34Z

@bazarnov thanks for the context! Just FYI make sure not to merge the full connector to Cloud master branch or it will become available in prod before it's ready :) so you should deploy from a branch to allow the shopify team to review

bazarnov · 2022-01-21T13:07:34Z

@sherifnada I'm stuck on the testing, waiting for new Release of OSS, current one v0.35.6-aplha does not include the latest commits, such as this: #9691, without it I cannot go forward.

bazarnov · 2022-01-24T13:18:10Z

@sherifnada

There is an error with Airbyte-Cloud deploy on v0.35.7-alpha release.

The error while connection_check is happening (both: OAuth2.0 and API Key) on Airbyte Cloud (dev-1 currently https://dev-1-cloud.airbyte.io/ instance deployed using airbyte-cloud/master it's active, try it).
Check the image attached bellow. Traceback is also available. It happens for all source & destination connectors. Need advice.

OSS Version works good, both OAuth2.0 & API Keys (any connector).

message: "Internal Server Error: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 2 path $"
rootCauseExceptionClassName: "java.lang.Class"
rootCauseExceptionStack: [,…]
0: "com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 2 path $"
1: "\tat com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1562)"
2: "\tat com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1403)"
3: "\tat com.google.gson.stream.JsonReader.nextNonWhitespace(JsonReader.java:1354)"
4: "\tat com.google.gson.stream.JsonReader.doPeek(JsonReader.java:548)"
5: "\tat com.google.gson.stream.JsonReader.peek(JsonReader.java:424)"
6: "\tat com.google.api.client.json.gson.GsonParser.nextToken(GsonParser.java:149)"
7: "\tat com.google.api.client.json.JsonParser.startParsing(JsonParser.java:213)"
8: "\tat com.google.api.client.json.JsonParser.parse(JsonParser.java:358)"
9: "\tat com.google.api.client.json.JsonParser.parse(JsonParser.java:335)"
10: "\tat com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79)"
11: "\tat com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73)"
12: "\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:541)"
13: "\tat com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:520)"
14: "\tat io.airbyte.config.storage.DefaultGcsClientFactory.get(DefaultGcsClientFactory.java:38)"
15: "\tat io.airbyte.config.storage.DefaultGcsClientFactory.get(DefaultGcsClientFactory.java:20)"
16: "\tat io.airbyte.config.helpers.GcsLogs.getOrCreateGcsClient(GcsLogs.java:123)"
17: "\tat io.airbyte.config.helpers.GcsLogs.tailCloudLog(GcsLogs.java:70)"
18: "\tat io.airbyte.config.helpers.LogClientSingleton.getJobLogFile(LogClientSingleton.java:112)"
19: "\tat io.airbyte.server.converters.JobConverter.getLogRead(JobConverter.java:136)"
20: "\tat io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:156)"
21: "\tat io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:143)"
22: "\tat io.airbyte.server.handlers.SchedulerHandler.reportConnectionStatus(SchedulerHandler.java:429)"
23: "\tat io.airbyte.server.handlers.SchedulerHandler.checkSourceConnectionFromSourceCreate(SchedulerHandler.java:168)"
24: "\tat io.airbyte.server.wrapped.ConfigurationApiWrapped.lambda$executeSourceCheckConnection$73(ConfigurationApiWrapped.java:782)"
25: "\tat io.airbyte.cloud.auth.AuthHelper.execute(AuthHelper.java:101)"
26: "\tat io.airbyte.cloud.auth.AuthHelper.callForRole(AuthHelper.java:62)"
27: "\tat io.airbyte.server.wrapped.ConfigurationApiWrapped.executeSourceCheckConnection(ConfigurationApiWrapped.java:780)"
28: "\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)"
29: "\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)"
30: "\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)"
31: "\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)"
32: "\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)"
33: "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)"
34: "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)"
35: "\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)"
36: "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)"
37: "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)"
38: "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)"
39: "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)"
40: "\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)"
41: "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)"
42: "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)"
43: "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)"
44: "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)"
45: "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)"
46: "\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)"
47: "\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)"
48: "\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)"
49: "\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)"
50: "\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)"
51: "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)"
52: "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)"
53: "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)"
54: "\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)"
55: "\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)"
56: "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)"
57: "\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)"
58: "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)"
59: "\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)"
60: "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)"
61: "\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)"
62: "\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)"
63: "\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)"
64: "\tat org.eclipse.jetty.server.Server.handle(Server.java:501)"
65: "\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)"
66: "\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)"
67: "\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)"
68: "\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)"
69: "\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)"
70: "\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)"
71: "\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)"
72: "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)"
73: "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)"
74: "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)"
75: "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)"
76: "\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)"
77: "\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)"
78: "\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)"
79: "\tat java.base/java.lang.Thread.run(Thread.java:833)"

Need advice, where to look at to find the problem?

Possible PR causes the issue:

bazarnov · 2022-01-26T13:15:50Z

Issue is fixed with v0.35.9-alpha. The shopify 0.1.30 is released and deployed on dev-1. We are ready to proceed with Shopify Support Team Test of OAuth flow.

UPDATE:
As far as there is no external access to dev-like resources for external users, like Support Team of Shopify, we have to use the https://dev.airbyte.ga/ which is the same, but using OSS of latest master with dev-UI mod.

sherifnada · 2022-01-26T17:35:56Z

@bazarnov opening this issue since it covers the scope of in progress work as well

bazarnov · 2022-01-26T18:29:12Z

Yes, thanks. We are ready to be tested on https://dev.airbyte.ga/ BTW.

ycherniaiev · 2022-02-07T10:23:59Z

@sherifnada @bazarnov Currently we have some problems on getting our app on marketplace.Here is an answer from Shopify:

_Our records show that on multiple occasions you’ve submitted an app that doesn’t meet Shopify’s public app requirements. As a result, you’ve been temporarily suspended from submitting new apps until March 01, 2022.

You can re-submit your app when the suspension period is over. Please be sure that you have updated it to meet all the requirements for public apps before doing so.

Required changes
Use OAuth to ask for scope permissions immediately after merchants add your app. Check and troubleshoot your app's OAuth by reinstalling your app on your test store using this method. See this screenshot for context on your existing install flow._

sherifnada · 2022-02-07T16:22:01Z

@ycherniaiev does not being in the marketplace prevent us from having a public app? The latter is important, the former is not

sherifnada · 2022-02-17T16:56:16Z

@ycherniaiev what is the status of getting shopify into cloud? This is blocking for Beta certification

sherifnada · 2022-02-22T01:09:47Z

@bazarnov any insight on the above?

ycherniaiev · 2022-02-22T17:45:50Z

@ycherniaiev does not being in the marketplace prevent us from having a public app? The latter is important, the former is not

@sherifnada Marketplace does not prevent from having public app.But asa far as our competitors do not have app on marketplace we decided that in perspective it will be a good point.

sherifnada · 2022-02-23T23:14:25Z

@bazarnov I'm seeing shopify in cloud, does that mean it's fully ready for production use by customers?

sherifnada · 2022-02-23T23:15:40Z

I'm seeing that it's listed as "Draft" -- does that refer to the marketplace? or is it for using the connector itself?

sherifnada · 2022-02-28T03:36:53Z

had offline discussion:

the issue is that we currently allow API key signin via cloud. This caused shopify to reject/suspend our oauth application. Instead, they want us to allow only oauth.

@ycherniaiev, I will reach out once we disable API key signin on cloud.

once we do that, could you let change the URL we use to submit the App to them to use only cloud.airbyte.io ?

ycherniaiev · 2022-02-28T06:11:50Z

@sherifnada Nice! 1 March suspension is over but before that i will prepare all links and descriptions for Shopify

sherifnada · 2022-02-28T16:36:06Z

@ycherniaiev pleaae hold off on submitting for verification as we first need to disable API key auth for Shopify in cloud

bazarnov · 2022-02-28T17:08:38Z

@ycherniaiev pleaae hold off on submitting for verification as we first need to disable API key auth for Shopify in cloud

Do we have to do something with this, like remove this option from the spec?

sherifnada · 2022-02-28T23:01:06Z

no we will do this in a custom way for cloud via this ticket #10381

sherifnada · 2022-03-02T01:52:07Z

FYI disabling shopify in cloud via this PR #10783 for now

lluisgassovillarejo · 2022-08-09T11:59:52Z

Hi! We want to connect to Shopify data, and I see there haven't been many updates for a while on this ticket. Is there a rough estimate of when this will become available in the Clou version?
Thanks!

bazarnov · 2022-10-10T14:34:17Z

Duplicates: #11241
Closing this one.

Phlair added this to the ConnCore Dec 8, 2021 milestone Dec 8, 2021

Phlair self-assigned this Dec 8, 2021

Phlair modified the milestones: ConnCore Dec 8, 2021, ConnCore Dec 15, 2021 Dec 8, 2021

sherifnada modified the milestones: ConnCore Dec 15, 2021, ConnCore Dec 22, 2021 Dec 15, 2021

alexandr-shegeda added this to GL Roadmap Jan 11, 2022

alexandr-shegeda moved this to Backlog in GL Roadmap Jan 11, 2022

antixar moved this from Backlog to Prioritized for scoping in GL Roadmap Jan 11, 2022

bazarnov moved this from Prioritized for scoping to Ready for implementation in GL Roadmap Jan 12, 2022

sherifnada unassigned Phlair Jan 13, 2022

bazarnov moved this from Ready for implementation to Implementation in progress in GL Roadmap Jan 18, 2022

bazarnov self-assigned this Jan 18, 2022

bazarnov linked a pull request Jan 19, 2022 that will close this issue

🎉 Source Shopify: implement Oauth2.0 for Airbyte-Cloud #9591

Merged

12 tasks

bazarnov closed this as completed in #9591 Jan 19, 2022

bazarnov moved this from Implementation in progress to Done in GL Roadmap Jan 19, 2022

sherifnada reopened this Jan 26, 2022

bazarnov moved this from Done to Airbyte review in GL Roadmap Jan 26, 2022

bazarnov moved this from Airbyte review to Implementation in progress in GL Roadmap Jan 31, 2022

igrankova moved this to Implementation in progress in GL Roadmap Feb 2, 2022

bazarnov moved this from Implementation in progress to On Hold in GL Roadmap Mar 1, 2022

sherifnada mentioned this issue Mar 2, 2022

expose oauth as the only available auth option for shopify in cloud #10817

Closed

5 tasks

bleonard added the team/connectors-python label Apr 22, 2022

bazarnov closed this as completed Oct 10, 2022

girarda mentioned this issue Oct 10, 2022

Shopify app approval #11334

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get Shopify connector on Cloud #8633

Get Shopify connector on Cloud #8633

Phlair commented Dec 8, 2021 •

edited

Loading

sherifnada commented Dec 14, 2021 •

edited by bazarnov

Loading

sherifnada commented Jan 20, 2022

bazarnov commented Jan 20, 2022 •

edited

Loading

sherifnada commented Jan 20, 2022

bazarnov commented Jan 21, 2022 •

edited

Loading

bazarnov commented Jan 24, 2022 •

edited

Loading

bazarnov commented Jan 26, 2022 •

edited

Loading

sherifnada commented Jan 26, 2022

bazarnov commented Jan 26, 2022

ycherniaiev commented Feb 7, 2022

sherifnada commented Feb 7, 2022

sherifnada commented Feb 17, 2022

sherifnada commented Feb 22, 2022

ycherniaiev commented Feb 22, 2022

sherifnada commented Feb 23, 2022

sherifnada commented Feb 23, 2022

sherifnada commented Feb 28, 2022

ycherniaiev commented Feb 28, 2022

sherifnada commented Feb 28, 2022

bazarnov commented Feb 28, 2022

sherifnada commented Feb 28, 2022

sherifnada commented Mar 2, 2022 •

edited

Loading

lluisgassovillarejo commented Aug 9, 2022

bazarnov commented Oct 10, 2022

Get Shopify connector on Cloud #8633

Get Shopify connector on Cloud #8633

Comments

Phlair commented Dec 8, 2021 • edited Loading

sherifnada commented Dec 14, 2021 • edited by bazarnov Loading

waiting for New OSS Release

sherifnada commented Jan 20, 2022

bazarnov commented Jan 20, 2022 • edited Loading

sherifnada commented Jan 20, 2022

bazarnov commented Jan 21, 2022 • edited Loading

bazarnov commented Jan 24, 2022 • edited Loading

OSS Version works good, both OAuth2.0 & API Keys (any connector).

bazarnov commented Jan 26, 2022 • edited Loading

sherifnada commented Jan 26, 2022

bazarnov commented Jan 26, 2022

ycherniaiev commented Feb 7, 2022

sherifnada commented Feb 7, 2022

sherifnada commented Feb 17, 2022

sherifnada commented Feb 22, 2022

ycherniaiev commented Feb 22, 2022

sherifnada commented Feb 23, 2022

sherifnada commented Feb 23, 2022

sherifnada commented Feb 28, 2022

ycherniaiev commented Feb 28, 2022

sherifnada commented Feb 28, 2022

bazarnov commented Feb 28, 2022

sherifnada commented Feb 28, 2022

sherifnada commented Mar 2, 2022 • edited Loading

lluisgassovillarejo commented Aug 9, 2022

bazarnov commented Oct 10, 2022

Phlair commented Dec 8, 2021 •

edited

Loading

sherifnada commented Dec 14, 2021 •

edited by bazarnov

Loading

bazarnov commented Jan 20, 2022 •

edited

Loading

bazarnov commented Jan 21, 2022 •

edited

Loading

bazarnov commented Jan 24, 2022 •

edited

Loading

bazarnov commented Jan 26, 2022 •

edited

Loading

sherifnada commented Mar 2, 2022 •

edited

Loading