BigQuery listTables(DatasetId) does not use project overrided in DatasetId

[tswast]

My Google Application Default Credentials are set to the swast-scratch project, but I wish to list the tables in the publicdata project's samples dataset. This is a public dataset.

$ bq ls publicdata:samples
      tableId       Type
 ----------------- -------
  github_nested     TABLE
  github_timeline   TABLE
  gsod              TABLE
  natality          TABLE
  shakespeare       TABLE
  trigrams          TABLE
  wikipedia         TABLE

Example code that displays this issue:

import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.DatasetId;
import com.google.cloud.bigquery.Table;
import java.io.PrintWriter;
import java.util.Iterator;

public class TableLister {
  public static void listTables(PrintWriter out) {
    BigQuery bigquery = BigQueryOptions.defaultInstance().service();
    Iterator<Table> tables = bigquery.listTables(DatasetId.of("publicdata", "samples")).iterateAll();
    while (tables.hasNext()) {
      out.println(tables.next());
    }
  }

  public static void main(String[] args) {
    listTables(new PrintWriter(System.out));
  }
}

It fails with the error: "message" : "Not found: Dataset swast-scratch:samples", so it appears that it is not using the project that I set in the DatasetId.

http://googlecloudplatform.github.io/gcloud-java/0.2.5/apidocs/com/google/cloud/bigquery/DatasetId.html#of-java.lang.String-java.lang.String-

[mziccard]

Hi Tim!

Actually this behavior is intended. All service operations (create/delete/get/list) are tied to the project id you provided to BigQueryOptions. The reason for which DatasetId and TableId also expose factory methods that take a projectId is that jobs (for instance a copy job) can be created in your bigqueryOptions.projectId() project and operate on tables and datasets of a different project.

Do you have a specific use case for creating BigQueryOptions with a project id and list tables that reside in a dataset of a different project?

[tswast]

Well for public datasets I don't have permission to access the BigQuery API
through them, but I do want to access those tables. It seems weird to say
that to read from a public dataset I need to connect to that project when I
create the service object.

[tswast]

So, it looks like I can work around this issue using BigQueryOptions, but it does feel wrong.

BigQuery bigquery = BigQueryOptions.builder().projectId("publicdata").build().service();
Iterator<Table> tables = bigquery.listTables(DatasetId.of("samples")).iterateAll();
while (tables.hasNext()) {
  out.println(tables.next());
}

I'm not a member of the publicdata project, so it feels strange to connect to it like this.

[tswast]

At the very least, we should raise an exception if the project ID is populated in the DatasetId.

It should not be silently ignored.

[mziccard]

I gave some thoughts on this and I think you are right. I'll try to wrap up a PR, I'll CC you so you can give it a try as soon as it's ready.