Listing redacted BigQuery jobs results in an NPE

[bmenasha]

When run without "bigquery.jobs.listAll" permissions on a project (like "Viewer") that contains other user's jobs this program results in an NPE .

package com.bmenasha;

import com.google.cloud.Page;
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.Job;

public class Main {
public static void main(String[] args){
BigQuery service = BigQueryOptions.builder().projectId(args[0]).build().service();
Page jobPage = service.listJobs(BigQuery.JobListOption.allUsers());
for(Job job: jobPage.values()){
System.out.println("status:" + job.status());
}
}
}

Exception in thread "main" java.lang.NullPointerException
at com.google.cloud.bigquery.JobConfiguration.fromPb(JobConfiguration.java:132)
at com.google.cloud.bigquery.JobInfo$BuilderImpl.(JobInfo.java:166)
at com.google.cloud.bigquery.Job.fromPb(Job.java:264)
at com.google.cloud.bigquery.BigQueryImpl$19.apply(BigQueryImpl.java:498)
at com.google.cloud.bigquery.BigQueryImpl$19.apply(BigQueryImpl.java:495)
at com.google.common.collect.Iterators$8.transform(Iterators.java:817)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at com.bmenasha.Main.main(Main.java:16)

BigQuery is redacting the metadata from these jobs as described here:

https://cloud.google.com/bigquery/docs/managing-jobs

The libraries should be able to parse these redacted job responses and not error when the metadata is missing.
thanks

[bmenasha]

Additionally it should be possible to list BigQuery jobs and be selective about the returned fields... currently we demand that the job's configuration always be returned via JobField.REQUIRED_FIELDS.

When listing many jobs this field can be large. The client library overriding my choice and requesting that this field not be returned.

[sombra-mykola-bakay]

Hello, @bmenasha
Could you please provide branch, or release information?

[bmenasha]

I'm able to recreate with this:

    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>gcloud-java-bigquery</artifactId>
        <version>LATEST</version>
    </dependency>

Which I think is currently 1.46.0

Exception in thread "main" java.lang.NullPointerException
at com.google.cloud.bigquery.JobConfiguration.fromPb(JobConfiguration.java:133)
at com.google.cloud.bigquery.JobInfo$BuilderImpl.(JobInfo.java:183)
at com.google.cloud.bigquery.Job.fromPb(Job.java:485)
at com.google.cloud.bigquery.BigQueryImpl$21.apply(BigQueryImpl.java:626)
at com.google.cloud.bigquery.BigQueryImpl$21.apply(BigQueryImpl.java:623)
at com.google.common.collect.Iterators$7.transform(Iterators.java:750)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
at com.bmenasha.Main.main(Main.java:17)

thanks

[bmenasha]

public static void main(String[] args){
BigQuery service = BigQueryOptions.newBuilder().setProjectId(args[0]).build().getService();
Page jobPage = service.listJobs(BigQuery.JobListOption.stateFilter(JobStatus.State.DONE),
BigQuery.JobListOption.allUsers(),BigQuery.JobListOption.fields(
BigQuery.JobField.STATUS));
for(Object jobo: ((Page) jobPage).getValues()) {
Job job = (Job)jobo;
if (job.getStatus().getError() != null) {
System.out.println("errored job id:" + job.getJobId());
}
}
}

[sombra-mykola-bakay]

@bmenasha,
gcloud-java-bigquery is previous artifact id
Latest version distributed under this artifact is 0.2.8

Current artifact id is google-cloud-bigquery

Latest version dependency:

com.google.cloud
google-cloud-bigquery
1.46.0

Could you please try to reproduce with latest version?

Will try to reproduce too

[bmenasha]

Yes... occurs in

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-bigquery</artifactId>
    <version>1.46.0</version>
</dependency>

thanks

[satishgadepally]

Hi,

In response to : #3739

As mention by bmenasha,

BigQuery is redacting the metadata from these jobs as described on:

https://cloud.google.com/bigquery/docs/managing-jobs

Here it's clearly mentioned,
Call jobs.list and provide the projectId parameter. To list jobs for all users, set the 'allUsers' parameter to true. Setting allUsers to true requires bigquery.jobs.listAll permissions.

Same can be found on https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/list also.

Thanks
Satish.

[lisumac]

Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'com.google.cloud.bigquery.JobConfiguration com.google.cloud.bigquery.JobConfiguration.setProjectId(java.lang.String)' on a null object reference