Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Walkthrough of using private assets #3599

Closed
arsian opened this issue Jan 3, 2024 · 2 comments
Closed

Walkthrough of using private assets #3599

arsian opened this issue Jan 3, 2024 · 2 comments
Assignees
Labels
api: vertex-ai Issues related to the Vertex AI API. priority: p3 Desirable enhancement or fix. May not be included in next release. samples Issues that are directly related to samples. triage me I really want to be triaged. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@arsian
Copy link

arsian commented Jan 3, 2024

How would one go about using assets from their own private bucket (and not the public one in the case of generativeai-downloads/images/scones.jpg)?

Is your feature request related to a problem? Please describe.

I am facing errors (500 internal server) when attempting to access an image in my own GCS and there seems to be an access/permission issue even though I have tried to instantiate a Storage object with my application_default_credentials in order to access the files inside the bucket, and to then use the file to compose a fileUri (i.e. gs://bucket-name/filename.jpeg).

Describe the solution you'd like.

I would like to be able to provide private assets from GCS to VertexAI and get the same result as described here
https://cloud.google.com/vertex-ai/docs/generative-ai/start/quickstarts/quickstart-multimodal#send-request

  1. I created a bucket
  2. Then placed an image inside the bucket
  3. Populated the code below with above info
const { Storage } = require('@google-cloud/storage');
const { VertexAI } = require('@google-cloud/vertexai');

const PROJECT_ID = process.env.PROJECT_ID;
const bucketName = 'arsia-toki-bucket-standard';

async function createNonStreamingMultipartContent(
    projectId = PROJECT_ID,
    location = 'us-central1',
    model = 'gemini-pro-vision',
    mimeType = 'image/jpeg'
) {
    const storage = new Storage({ projectId, keyFilename: './application_default_credentials.json' });

    // Initialize Vertex with your Cloud project and location
    const vertexAI = new VertexAI({ project: projectId, location: location });

    // Instantiate the model
    const generativeVisionModel = vertexAI.preview.getGenerativeModel({
        model: model,
    });

    const [files] = await storage.bucket(bucketName).getFiles();
    if (files.length === 0) {
        throw new Error('No files found in the bucket.');
    }

    const fileUri = `gs://${bucketName}/${files[0].name}`;

    // Create the request
    // For images, the SDK supports both Google Cloud Storage URI and base64 strings
    const filePart = {
        fileData: {
            fileUri: fileUri,
            mimeType: mimeType,
        },
    };
    const textPart = {
        text: 'what is shown in this image?',
    };

    const request = {
        contents: [{ role: 'user', parts: [filePart, textPart] }],
    };

    console.log('Prompt Text:');
    console.log(request.contents[0].parts[1].text);


    console.log('Non-Streaming Response Text:');
    // Create the response stream
    const responseStream = await generativeVisionModel.generateContentStream(request);

    // Wait for the response stream to complete
    const aggregatedResponse = await responseStream.response;

    // Select the text from the response
    const fullTextResponse = aggregatedResponse.candidates[0].content.parts[0].text;

    console.log(fullTextResponse);

    // [END aiplatform_gemini_get_started]
}

createNonStreamingMultipartContent(...process.argv.slice(2)).catch(err => {
    console.error(err.message);
    process.exitCode = 1;
});

Screenshot 2024-01-03 at 2 11 16 PM

Describe alternatives you've considered.

So one solution that I did conjure up was to add to the acl rules before sending the asset to VertexAI and once the analysis is complete, to revoke the public acl rule. But I wonder if you have a more elegant solution.

const { Storage } = require('@google-cloud/storage');
const { VertexAI } = require('@google-cloud/vertexai');

const PROJECT_ID = process.env.PROJECT_ID;
const bucketName = 'arsia-toki-bucket-standard';

async function createNonStreamingMultipartContent(
    projectId = PROJECT_ID,
    location = 'us-central1',
    model = 'gemini-pro-vision',
    mimeType = 'image/jpeg'
) {

    // Initialize Vertex with your Cloud project and location
    const vertexAI = new VertexAI({ project: projectId, location: location });

    // Instantiate the model
    const generativeVisionModel = vertexAI.preview.getGenerativeModel({
        model: model,
    });

    const storage = new Storage({ projectId: PROJECT_ID, keyFilename: './key.json' });

    // Get the file from the storage bucket
    const [files] = await storage.bucket(bucketName).getFiles();
    if (files.length === 0) {
        throw new Error('No files found in the bucket.');
    }

    // the second file is what I want the AI to analyze this time 
    const file = files[1];
    const fileUri = `gs://${bucketName}/${file.name}`;
    console.log('File URI:', fileUri);

    try {
        await file.acl.add({
            entity: 'allUsers',
            role: 'READER',
        });
        console.log('File is temporarily public.');

        // Create the request
        // For images, the SDK supports both Google Cloud Storage URI and base64 strings
        const filePart = {
            fileData: {
                fileUri: fileUri,
                mimeType: mimeType,
            },
        };
        const textPart = {
            text: 'what food items are shown in this image?',
        };
        const request = {
            contents: [{ role: 'user', parts: [filePart, textPart] }],
        };
        console.log('Prompt Text:');
        console.log(request.contents[0].parts[1].text);
        console.log('Non-Streaming Response Text:');
        
// Create the response stream
        const responseStream = await generativeVisionModel.generateContentStream(request);

        // Wait for the response stream to complete
        const aggregatedResponse = await responseStream.response;

        await file.acl.add({
            entity: 'allUsers',
            role: 'READER',
        });

        // Select the text from the response
        const fullTextResponse = aggregatedResponse.candidates[0].content.parts[0].text;

        console.log(fullTextResponse);

    } finally {
        try {
            await file.acl.remove({
                entity: 'allUsers',
                role: 'READER',
            });

            console.log('Public access revoked. File is private again.');
        } catch (error) {
            console.error('Error revoking public access:', error.message);
        }
    }
}

createNonStreamingMultipartContent(...process.argv.slice(2)).catch(err => {
    console.error(err.message);
    process.exitCode = 1;
});

Additional context.

Thanks!

@arsian arsian added priority: p3 Desirable enhancement or fix. May not be included in next release. triage me I really want to be triaged. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. labels Jan 3, 2024
@product-auto-label product-auto-label bot added the samples Issues that are directly related to samples. label Jan 3, 2024
@TokiUrata
Copy link

TokiUrata commented Jan 3, 2024

I'm encountering the same 500 internal server error accessing private GCS assets in VertexAI. Tried the code with proper credentials, but still facing permission issues. Any insights or solutions?

Appreciate any help!

@iennae iennae added the api: vertex-ai Issues related to the Vertex AI API. label Apr 16, 2024
@arbrown arbrown self-assigned this Aug 14, 2024
@arbrown
Copy link
Contributor

arbrown commented Aug 14, 2024

Hi there,

You should be able to access private cloud storage objects as part of a vertex AI API request. However, the service account(s) used at each step might be different than what you are expecting. Check this doc for details on the how service accounts are used in Vertex AI, and what you can do to change them.

In your case, you might have 1, 2, or 3 service accounts in use depending on how you're running the code you provided:

  1. The service account running the code on Google Cloud Platform
  2. The service account in the key file you pass to the storage client
  3. The service agent handling the request to the Vertex AI API

It's possible that these are all the same service account (even the default compute service account) but from the situation you are describing, it sounds like #2 and #3 would be different. It seems like the service account in the key file has access to see the files in cloud storage, but your AI service agent might not have permission to download them.

One possible reason for this discrepancy could be that the storage bucket with your private files is in a different project, and thus the AI service agent for the project specified in process.env.PROJECT_ID does not have access to those files.

If you try running this sample in the same project as the storage bucket, it should work. Also, if you are able to grant access to the compute service account to the bucket with the private files, that should clear up your issues.

@arbrown arbrown closed this as completed Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: vertex-ai Issues related to the Vertex AI API. priority: p3 Desirable enhancement or fix. May not be included in next release. samples Issues that are directly related to samples. triage me I really want to be triaged. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

4 participants