VertexAI - Decoding the Chat history fails with "No text, inline data or function call was found." when a functionResponse exists

[TiVoShane]

Description

In order to save the Chat history, I encode it and save it to a file. When the app starts, I load the Chat history from the file and decode it. This works fine unless the Chat history contains a functionResponse. Decoding the history when it contains a functionResponse fails.

Reproducing the issue

Use the code below to test. Call the await test() function.
The first time, loading the chat history will fail because no file called"testFile" exists. However, saving is successful so the file will exist the next time you call test().
The second time, the file exists but decoding fails.

let getExchangeRate = FunctionDeclaration(
    name: "getExchangeRate",
    description: "Get the exchange rate for currencies between countries",
    parameters: [
        "currencyFrom": Schema(
            type: .string,
            description: "The currency to convert from."
        ),
        "currencyTo": Schema(
            type: .string,
            description: "The currency to convert to."
        ),
    ],
    requiredParameters: ["currencyFrom", "currencyTo"]
)

func makeAPIRequest(currencyFrom: String,
                    currencyTo: String) -> JSONObject {
    // This hypothetical API returns a JSON such as:
    // {"base":"USD","rates":{"SEK": 10.99}}
    return [
        "base": .string(currencyFrom),
        "rates": .object([currencyTo: .number(10.99)]),
    ]
}

func test() async {
    let testFile : String = "testFile"
    // Initialize the Vertex AI service
    let vertex = VertexAI.vertexAI()
    
    // Initialize the generative model
    // Use a model that supports function calling, like a Gemini 1.5 model
    let model = vertex.generativeModel(
        modelName: "gemini-1.5-flash",
        // Specify the function declaration.
        tools: [Tool(functionDeclarations: [getExchangeRate])]
    )
    
    let history = loadChatHistory(from: testFile)
    let chat = model.startChat(history : history)
    
    let prompt = "How much is 50 US dollars worth in Swedish krona?"
    
    // Send the message to the generative model
    Task {
        let response1 = try await chat.sendMessage(prompt)
        
        // Check if the model responded with a function call
        guard let functionCall = response1.functionCalls.first else {
            fatalError("Model did not respond with a function call.")
        }
        // Print an error if the returned function was not declared
        guard functionCall.name == "getExchangeRate" else {
            fatalError("Unexpected function called: \(functionCall.name)")
        }
        // Verify that the names and types of the parameters match the declaration
        guard case let .string(currencyFrom) = functionCall.args["currencyFrom"] else {
            fatalError("Missing argument: currencyFrom")
        }
        guard case let .string(currencyTo) = functionCall.args["currencyTo"] else {
            fatalError("Missing argument: currencyTo")
        }
        
        // Call the hypothetical API
        let apiResponse = makeAPIRequest(currencyFrom: currencyFrom, currencyTo: currencyTo)
        
        // Send the API response back to the model so it can generate a text response that can be
        // displayed to the user.
        let response = try await chat.sendMessage([ModelContent(
            role: "function",
            parts: [.functionResponse(FunctionResponse(
                name: functionCall.name,
                response: apiResponse
            ))]
        )])
        
        // Log the text response.
        guard let modelResponse = response.text else {
            fatalError("Model did not respond with text.")
        }
        saveChatHistory(chat.history, to: testFile)
        print(modelResponse)
    }
}

func saveChatHistory(_ history: [ModelContent], to filename: String) {
    let fileManager = FileManager.default
    let urls = fileManager.urls(for: .documentDirectory, in: .userDomainMask)
    if let documentDirectory = urls.first {
        let fileURL = documentDirectory.appendingPathComponent(filename)
        let encoder = JSONEncoder()
        encoder.outputFormatting = .withoutEscapingSlashes
        do {
            let data = try encoder.encode(history)
            try data.write(to: fileURL)
            print("Chat history saved successfully.")
        } catch {
            print("Failed to save chat history: \(error)")
        }
    }
}

func loadChatHistory(from filename: String) -> [ModelContent] {

    let fileManager = FileManager.default
    let urls = fileManager.urls(for: .documentDirectory, in: .userDomainMask)
    if let documentDirectory = urls.first {
        let fileURL = documentDirectory.appendingPathComponent(filename)
        let decoder = JSONDecoder()
        do {
            let data = try Data(contentsOf: fileURL)
            if let jsonString = String(data: data, encoding: .utf8) {
                print("Loaded JSON string: \(jsonString)")
            }
            let history = try decoder.decode([ModelContent].self, from: data)
            print("Chat history loaded successfully.")
            return history
        } catch {
            print("Failed to load chat history: \(error)")
        }
    }
    return []
}

Firebase SDK Version

11.1

Xcode Version

16.0

Installation Method

Swift Package Manager

Firebase Product(s)

VertexAI

Targeted Platforms

iOS

Relevant Log Output

Failed to load chat history: Error Domain=NSCocoaErrorDomain Code=260 "The file “testFile” couldn’t be opened because there is no such file." UserInfo={NSUnderlyingError=0x600000cd36c0 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}, NSURL=file:///Users/Shane/Library/Developer/CoreSimulator/Devices/C98366BC-B53E-466D-AAEC-2D950385105D/data/Containers/Data/Application/D490FDC4-6CB9-4F46-90B4-AC9D76562B42/Documents/testFile, NSFilePath=/Users/Shane/Library/Developer/CoreSimulator/Devices/C98366BC-B53E-466D-AAEC-2D950385105D/data/Containers/Data/Application/D490FDC4-6CB9-4F46-90B4-AC9D76562B42/Documents/testFile}
nw_connection_copy_connected_local_endpoint_block_invoke [C1] Connection has no local endpoint
nw_connection_copy_connected_local_endpoint_block_invoke [C1] Connection has no local endpoint
Chat history saved successfully.
50 US dollars is worth 549.5 Swedish krona. 

Submit function called
[FirebaseVertexAI] To enable additional logging, add `-FIRDebugEnabled` as a launch argument in Xcode.
[FirebaseVertexAI] Model projects/expense-tracking-59dac/locations/us-central1/publishers/google/models/gemini-1.5-flash initialized.
Loaded JSON string: [{"role":"user","parts":[{"text":"How much is 50 US dollars worth in Swedish krona?"}]},{"parts":[{"functionCall":{"name":"getExchangeRate","args":{"currencyTo":"SEK","currencyFrom":"USD"}}}],"role":"model"},{"role":"function","parts":[{"functionResponse":{"response":{"base":"USD","rates":{"SEK":10.99}},"name":"getExchangeRate"}}]},{"parts":[{"text":"50 US dollars is worth 549.5 Swedish krona. \n"}],"role":"model"}]
Failed to load chat history: dataCorrupted(Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "text", intValue: nil), CodingKeys(stringValue: "inlineData", intValue: nil)], debugDescription: "No text, inline data or function call was found.", underlyingError: nil))
Chat history saved successfully.
50 US dollars is worth 549.5 Swedish krona.

If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

Replace this line with the contents of your Package.resolved.

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

Replace this line with the contents of your Podfile.lock!

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[andrewheard]

Hi @TiVoShane, thank you for providing the the full code snippet. That made it easy to understand your use case for decoding FunctionResponse, even though it's not a type that comes from the backend. I've got a PR out for review to resolve the issue now and will keep you posted.

[andrewheard]

@TiVoShane The fix for this issue is merged into main now and will go out in the Firebase 11.3 (next next) release. As a heads up, if you try this out on main (or the upcoming Firebase 11.2 release) there will be a few breaking changes that might require you to tweak your code slightly if you use those APIs:

firebase-ios-sdk/FirebaseVertexAI/CHANGELOG.md

Lines 7 to 14 in 3e2c79d

	- [changed] **Breaking Change**: The methods for starting streaming requests
	(`generateContentStream` and `sendMessageStream`) are now throwing and
	asynchronous and must be called with `try await`. (#13545, #13573)
	- [changed] **Breaking Change**: Creating a chat instance (`startChat`) is now
	asynchronous and must be called with `await`. (#13545)
	- [changed] **Breaking Change**: The source image in the
	`ImageConversionError.couldNotConvertToJPEG` error case is now an enum value
	instead of the `Any` type. (#13575)

[TiVoShane]

Any idea when 11.3 will be released? Thanks for fixing this! Shane Miller https://twitter.com/TiVoShane https://www.instagram.com/tivoshane <https://www.instagram.com/tivoshane?igsh=OXV3amY0ejAzejgz&utm_source=qr>

…

On Mon, Sep 9, 2024 at 2:42 PM Andrew Heard ***@***.***> wrote: @TiVoShane <https://github.com/TiVoShane> The fix for this issue is merged into main now and will go out in the Firebase 11.3 (next next) release. As a heads up, if you try this out on main (or the upcoming Firebase 11.2 release) there will be a few breaking changes that might require you to tweak your code slightly if you use those APIs: https://github.com/firebase/firebase-ios-sdk/blob/3e2c79dd62289dc0c503eabbdb262ad6118432c8/FirebaseVertexAI/CHANGELOG.md?plain=1#L7-L14 — Reply to this email directly, view it on GitHub <#13593 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5ICKHRXIZRNSOPIVSRCNLZVX2ZBAVCNFSM6AAAAABNWWVKESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZYHEZTQMBVGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[paulb777]

We plan to release 11.3 the week of September 30th