Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client Datastore API with App Engine Devserver #1328

Closed
mitchhentges opened this issue Oct 21, 2016 · 7 comments
Closed

Client Datastore API with App Engine Devserver #1328

mitchhentges opened this issue Oct 21, 2016 · 7 comments
Assignees

Comments

@mitchhentges
Copy link

I've investigated into #216, but I can't quite get my Client Datastore connection to work with the App Engine Devserver.

I've tried:

# No environment variables, DatastoreOptions.defaultInstance().service()
com.google.datastore.v1.client.DatastoreException: Request had invalid authentication credentials.

# DATASTORE_EMULATOR_HOST=localhost:8080
# DatastoreOptions.defaultInstance().service()
WARNING: No file found for: /v1/projects/navistream2dev:runQuery
SEVERE: com.google.cloud.datastore.DatastoreException: Non-protobuf error: <html><head><title>Error 404</title></head>
<body><h2>Error 404</h2></body>
</html>. HTTP status code was 404.

# export DATASTORE_PROJECT_ID=navistream2dev   
# export DATASTORE_EMULATOR_HOST=localhost:8080
# DatastoreOptions.defaultInstance().service()
WARNING: No file found for: /v1/projects/navistream2dev:runQuery
SEVERE: com.google.cloud.datastore.DatastoreException: Non-protobuf error: <html><head><title>Error 404</title></head>
<body><h2>Error 404</h2></body>
</html>. HTTP status code was 404.

# export DATASTORE_PROJECT_ID=navistream2dev
# DatastoreOptions.defaultInstance().service()
WARNING: failed _ah_AbandonedTransactionDetector: java.lang.RuntimeException: Could not get Cloud Datastore options from environment.
Oct 20, 2016 4:45:00 PM com.google.apphosting.utils.jetty.JettyLogger warn
WARNING: Failed startup of context com.google.appengine.tools.development.DevAppEngineWebAppContext@21ffbe3b{/,/home/mitch/dev/navistream/server/navistream/build/exploded-app}
java.lang.RuntimeException: Could not get Cloud Datastore options from environment.
Caused by: java.io.IOException: Application Default Credentials failed to create the Google App Engine service account credentials class com.google.appengine.repackaged.com.google.api.client.googleapis.extensions.appengine.auth.oauth2.AppIdentityCredential$AppEngineCredentialWrapper. Check that the component 'google-api-client-appengine' is deployed.
Caused by: java.lang.ClassNotFoundException: com.google.appengine.repackaged.com.google.api.client.googleapis.extensions.appengine.auth.oauth2.AppIdentityCredential$AppEngineCredentialWrapper

// Interestingly, this is trying to get `AppIdentityCredential` from "com.google.appengine.repackaged.com.google.api...", rather than from "com.google.api..."

Am I missing an environment variable for configuration to make this work nicely?

@mziccard
Copy link
Contributor

What are you trying to achieve here?

Issue #216 says that you cannot access Devserver's Datastore as it does not expose either REST or gRPC APIs:

currently local dev app services such as Datastore are not accessible via the cloud interface (apiary or gRpc).

So you will not be able to access Devserver's Datastore by changing DatastoreOptions or setting environment variables.

What is your final goal?

@mitchhentges
Copy link
Author

mitchhentges commented Oct 21, 2016

I was really hoping that that part of the first comment would be inaccurate, considering that I'm having unauthenticated errors, and there was a comment about ApplicationDefaultAuthCredentials.

But yeah, what I want to accomplish, is being able to run my application locally. Unfortunately, it's still tied to some appengine APIs (memcache). So, I need to run the application within the appengine devserver, or the memcache-related component will fail. At the same time, I need to talk to a datastore emulator to store my data. If I don't need to run an external datastore emulator, it would make my life 200% easier 😉

@mziccard
Copy link
Contributor

@mitchhentges I believe that comment was about using the actual Datastore service, not the DevServer's emulator, from code running in the DevServer.

I know this is suboptimal but is using the external Datastore emulator such a pain for you? You should be able to start it with:

gcloud beta emulators datastore start

And once you export the emulator host with export DATASTORE_EMULATOR_HOST=localhost:<yourport> you should also be able to access it from the DevServer.

@mitchhentges
Copy link
Author

mitchhentges commented Oct 21, 2016

The reason that starting the external datastore emulator is so annoying is because the port always changes.

With the old, appengine datastore API:

  1. From within IDE, ./gradlew appEngineRun
  2. Application is live

Once we have to deal with an external datastore:

  1. Start datastore emulator
  2. Copy port
  3. Edit IDE run preferences, set environment variable for datastore emulator host/port
  4. ./gradlew appengineRun
  5. Application is live

@mziccard
Copy link
Contributor

You can set the port when you start the emulator:

gcloud beta emulators datastore start --host-port=localhost:<yourpreferredport>

See gcloud beta emulators datastore start --help for more options.

This is still suboptimal but gets rid of points 2 and 3 :)

@mitchhentges
Copy link
Author

Ah, that helps, and it's confirmed that the client datastore api cannot communicate with the appengine devserver datastore. I have clear direction now, thanks!

@josenicomaia
Copy link

josenicomaia commented Mar 20, 2022

@mitchhentges, you can also create an adapter between then. When using Spring Data I can do this:

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @Bean
    public DatastoreService datastoreService() {
        return DatastoreServiceFactory.getDatastoreService();
    }

    @Value("${spring.cloud.gcp.datastore.project-id}")
    public String projectId;

    @Bean
    public Datastore datastore(DatastoreService datastoreService) {
        return new DatastoreAdapter(datastoreService, projectId);
    }
}
public class DatastoreAdapter implements Datastore {
    private final DatastoreService datastoreService;
    private final String projectId;

    public DatastoreAdapter(DatastoreService datastoreService, String projectId) {
        this.datastoreService = datastoreService;
        this.projectId = projectId;
    }

    @Override
    public Transaction newTransaction(TransactionOptions transactionOptions) {
        return null;
    }

    @Override
    public Transaction newTransaction() {
        return null;
    }

    @Override
    public <T> T runInTransaction(TransactionCallable<T> transactionCallable) {
        return null;
    }

    @Override
    public <T> T runInTransaction(TransactionCallable<T> transactionCallable, TransactionOptions transactionOptions) {
        return null;
    }

    @Override
    public Batch newBatch() {
        return null;
    }

    @Override
    public Key allocateId(IncompleteKey incompleteKey) {
        return null;
    }

    @Override
    public List<Key> allocateId(IncompleteKey... incompleteKeys) {
        return null;
    }

    @Override
    public List<Key> reserveIds(Key... keys) {
        return null;
    }

    @Override
    public Entity add(FullEntity<?> fullEntity) {
        return null;
    }

    @Override
    public List<Entity> add(FullEntity<?>... fullEntities) {
        return null;
    }

    @Override
    public void update(Entity... entities) {

    }

    @Override
    public Entity put(FullEntity<?> fullEntity) {
        return null;
    }

    @Override
    public List<Entity> put(FullEntity<?>... fullEntities) {
        List<com.google.appengine.api.datastore.Entity> gaeEntityList = Arrays.stream(fullEntities)
                .map((FullEntity<?> fullEntity) -> {
                    Key incompleteKey = (Key) fullEntity.getKey();
                    com.google.appengine.api.datastore.Key key = convertGCToGAEKey(incompleteKey);
                    com.google.appengine.api.datastore.Entity entity = new com.google.appengine.api.datastore.Entity(key);
                    copyGCToGAEEntityProperties(fullEntity, entity);

                    return entity;
                })
                .collect(Collectors.toList());

        datastoreService.put(gaeEntityList);

        return Arrays.stream(fullEntities)
                .map(fullEntity -> {
                    Key key = (Key) fullEntity.getKey();
                    return Entity.newBuilder(key, fullEntity).build();
                })
                .collect(Collectors.toList());
    }

    @Override
    public void delete(Key... keys) {

    }

    @Override
    public KeyFactory newKeyFactory() {
        return new KeyFactory(projectId, "default");
    }

    @Override
    public Entity get(Key key, ReadOption... readOptions) {
        return null;
    }

    @Override
    public Iterator<Entity> get(Iterable<Key> iterable, ReadOption... readOptions) {
        return null;
    }

    @Override
    public List<Entity> fetch(Iterable<Key> iterable, ReadOption... readOptions) {
        return null;
    }

    @Override
    public <T> QueryResults<T> run(Query<T> query, ReadOption... readOptions) {
        return null;
    }

    @Override
    public DatastoreOptions getOptions() {
        return null;
    }

    @Override
    public Entity get(Key key) {
        try {
            com.google.appengine.api.datastore.Entity entity = datastoreService.get(convertGCToGAEKey(key));
            Entity.Builder builder = Entity.newBuilder(key);
            copyGAEToGCEntityProperties(builder, entity);

            return builder.build();
        } catch (EntityNotFoundException e) {
            throw new RuntimeException(e);
        }
    }

    @Override
    public Iterator<Entity> get(Key... keys) {
        return null;
    }

    @Override
    public List<Entity> fetch(Key... keys) {
        List<com.google.appengine.api.datastore.Key> keyList = Arrays.stream(keys)
                .map(this::convertGCToGAEKey)
                .collect(Collectors.toList());

        Collection<com.google.appengine.api.datastore.Entity> values = datastoreService.get(keyList)
                .values();

        return values
                .stream()
                .map(entity -> {
                    Key key = Key.newBuilder(projectId, entity.getKind(), entity.getKey().getName())
                            .build();

                    Entity.Builder builder = Entity.newBuilder(key);
                    copyGAEToGCEntityProperties(builder, entity);

                    return builder.build();
                }).collect(Collectors.toList());
    }

    @Override
    public <T> QueryResults<T> run(Query<T> query) {
        return null;
    }

    private com.google.appengine.api.datastore.Key convertGCToGAEKey(Key key) {
        return com.google.appengine.api.datastore.KeyFactory.createKey(key.getKind(), key.getName());
    }

    private Key convertGAEToGCKey(Key k) {
        return Key.newBuilder(projectId, k.getKind(), k.getName()).build();
    }

    private void copyGAEToGCEntityProperties(Entity.Builder GCEntityBuilder, com.google.appengine.api.datastore.Entity GAEEntity) {
        GAEEntity.getProperties().forEach((k, v) -> {
            if (v instanceof String) {
                GCEntityBuilder.set(k, (String) v);
            } else if (v instanceof Long) {
                GCEntityBuilder.set(k, (Long) v);
            } else if (v instanceof Double) {
                GCEntityBuilder.set(k, (Double) v);
            } else if (v instanceof Boolean) {
                GCEntityBuilder.set(k, (Boolean) v);
            } else {
                throw new UnsupportedOperationException("Not implemented yet");
            }
        });
    }

    private void copyGCToGAEEntityProperties(FullEntity<?> fullEntity, com.google.appengine.api.datastore.Entity entity) {
        fullEntity.getProperties().forEach((k, v) -> entity.setProperty(k, v.get()));
    }
}

But I don't think this is a good way haha. I guess that just accessing the Google Cloud Datastore API is way easier.

github-actions bot pushed a commit that referenced this issue Sep 15, 2022
… to v1.5.4 (#1328)

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [com.google.cloud:google-cloud-resourcemanager](https://github.com/googleapis/java-resourcemanager) | `1.5.3` -> `1.5.4` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-resourcemanager/1.5.4/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-resourcemanager/1.5.4/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-resourcemanager/1.5.4/compatibility-slim/1.5.3)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-resourcemanager/1.5.4/confidence-slim/1.5.3)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/java-resourcemanager</summary>

### [`v1.5.4`](https://github.com/googleapis/java-resourcemanager/blob/HEAD/CHANGELOG.md#&#8203;154-httpsgithubcomgoogleapisjava-resourcemanagercomparev153v154-2022-08-31)

[Compare Source](https://github.com/googleapis/java-resourcemanager/compare/v1.5.3...v1.5.4)

##### Dependencies

-   update dependency com.google.apis:google-api-services-cloudresourcemanager to v1-rev20220828-2.0.0 ([#&#8203;829](https://github.com/googleapis/java-resourcemanager/issues/829)) ([4821d7a](https://github.com/googleapis/java-resourcemanager/commit/4821d7a6ebeffa5b8ee5ff4bfc3be72309800d00))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-asset).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzMi4xODQuMiIsInVwZGF0ZWRJblZlciI6IjMyLjE4NC4yIn0=-->
github-actions bot pushed a commit that referenced this issue Sep 16, 2022
🤖 I have created a release *beep* *boop*
---


## [3.6.0](googleapis/java-asset@v3.5.0...v3.6.0) (2022-09-15)


### Features

* Add client library support for AssetService v1 BatchGetEffectiveIamPolicies API ([3919a1d](googleapis/java-asset@3919a1d))
* Add client library support for AssetService v1 BatchGetEffectiveIamPolicies API ([#1300](googleapis/java-asset#1300)) ([3919a1d](googleapis/java-asset@3919a1d))
* Release of query system ([3919a1d](googleapis/java-asset@3919a1d))


### Dependencies

* Update dependency com.google.api.grpc:proto-google-cloud-orgpolicy-v1 to v2.3.2 ([#1302](googleapis/java-asset#1302)) ([d01d900](googleapis/java-asset@d01d900))
* Update dependency com.google.api.grpc:proto-google-cloud-orgpolicy-v1 to v2.3.3 ([#1332](googleapis/java-asset#1332)) ([c1511c2](googleapis/java-asset@c1511c2))
* Update dependency com.google.api.grpc:proto-google-cloud-os-config-v1 to v2.5.2 ([#1309](googleapis/java-asset#1309)) ([cf96ee9](googleapis/java-asset@cf96ee9))
* Update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.102.11 ([#1297](googleapis/java-asset#1297)) ([d56eedd](googleapis/java-asset@d56eedd))
* Update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.102.12 ([#1316](googleapis/java-asset#1316)) ([a3713fd](googleapis/java-asset@a3713fd))
* Update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.102.13 ([#1321](googleapis/java-asset#1321)) ([883b7b8](googleapis/java-asset@883b7b8))
* Update dependency com.google.api.grpc:proto-google-cloud-pubsub-v1 to v1.102.14 ([#1334](googleapis/java-asset#1334)) ([56cb4b4](googleapis/java-asset@56cb4b4))
* Update dependency com.google.api.grpc:proto-google-identity-accesscontextmanager-v1 to v1.4.1 ([#1307](googleapis/java-asset#1307)) ([b90baf7](googleapis/java-asset@b90baf7))
* Update dependency com.google.cloud:google-cloud-bigquery to v2.14.4 ([#1301](googleapis/java-asset#1301)) ([64f7ea5](googleapis/java-asset@64f7ea5))
* Update dependency com.google.cloud:google-cloud-bigquery to v2.14.6 ([#1315](googleapis/java-asset#1315)) ([fa179e2](googleapis/java-asset@fa179e2))
* Update dependency com.google.cloud:google-cloud-bigquery to v2.14.7 ([#1320](googleapis/java-asset#1320)) ([06d1a16](googleapis/java-asset@06d1a16))
* Update dependency com.google.cloud:google-cloud-bigquery to v2.15.0 ([#1326](googleapis/java-asset#1326)) ([df36595](googleapis/java-asset@df36595))
* Update dependency com.google.cloud:google-cloud-bigquery to v2.16.0 ([#1336](googleapis/java-asset#1336)) ([870779d](googleapis/java-asset@870779d))
* Update dependency com.google.cloud:google-cloud-core to v2.8.10 ([#1323](googleapis/java-asset#1323)) ([09e03b8](googleapis/java-asset@09e03b8))
* Update dependency com.google.cloud:google-cloud-core to v2.8.11 ([#1331](googleapis/java-asset#1331)) ([a112cec](googleapis/java-asset@a112cec))
* Update dependency com.google.cloud:google-cloud-core to v2.8.9 ([#1314](googleapis/java-asset#1314)) ([8edc2b8](googleapis/java-asset@8edc2b8))
* Update dependency com.google.cloud:google-cloud-pubsub to v1.120.11 ([#1298](googleapis/java-asset#1298)) ([172b34b](googleapis/java-asset@172b34b))
* Update dependency com.google.cloud:google-cloud-pubsub to v1.120.12 ([#1317](googleapis/java-asset#1317)) ([1ea636f](googleapis/java-asset@1ea636f))
* Update dependency com.google.cloud:google-cloud-pubsub to v1.120.13 ([#1322](googleapis/java-asset#1322)) ([b7522b9](googleapis/java-asset@b7522b9))
* Update dependency com.google.cloud:google-cloud-pubsub to v1.120.14 ([#1335](googleapis/java-asset#1335)) ([e9142b4](googleapis/java-asset@e9142b4))
* Update dependency com.google.cloud:google-cloud-resourcemanager to v1.5.3 ([#1318](googleapis/java-asset#1318)) ([aa3a1bb](googleapis/java-asset@aa3a1bb))
* Update dependency com.google.cloud:google-cloud-resourcemanager to v1.5.4 ([#1328](googleapis/java-asset#1328)) ([fca0ce5](googleapis/java-asset@fca0ce5))
* Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.2 ([#1330](googleapis/java-asset#1330)) ([ccb704c](googleapis/java-asset@ccb704c))
* Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.3 ([#1340](googleapis/java-asset#1340)) ([0d87a9d](googleapis/java-asset@0d87a9d))
* Update dependency com.google.cloud:google-cloud-storage to v2.11.3 ([#1299](googleapis/java-asset#1299)) ([d59e6c6](googleapis/java-asset@d59e6c6))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants