Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream functions and their properties. #337

Merged
merged 4 commits into from
Aug 21, 2023
Merged

Stream functions and their properties. #337

merged 4 commits into from
Aug 21, 2023

Conversation

raulraja
Copy link
Contributor

@raulraja raulraja commented Aug 20, 2023

This PR introduces support for promptStreaming.
When you use promptStreaming, you get back a stream where you can collect each of the individual properties of a type as they are streamed from the LLM. Finally, you get the final result serialized.
This PR also includes some fixes to the @Description annotations so they work properly in Kotlin, which broke after we fixed the Scala and Java ones.

The implementation for streaming functions looks like this:

  1. Get the schema of the CFunction we ask the LLM to reply with.
  2. Create a fake example in JSON format of the object deriving the example from the schema.
  3. Parse the malformed chunked text as it's being streamed in search of the current property.
  4. Once parsing succeeds, we search the property in the example to get its path.
  5. We do this for every property until all are streamed.
  6. We stream the result back.

Example:

package com.xebia.functional.xef.conversation.streaming

import com.xebia.functional.xef.conversation.Description
import com.xebia.functional.xef.conversation.llm.openai.OpenAI
import com.xebia.functional.xef.conversation.llm.openai.promptStreaming
import com.xebia.functional.xef.llm.StreamedFunction
import com.xebia.functional.xef.prompt.Prompt
import kotlinx.serialization.Serializable

@Serializable
data class MissionDetail(
  @Description("The current speed of the spacecraft") val speed: Int,
  @Description("Destination planets") val destinationPlanets: List<String>,
  @Description("Estimated time of arrival at destination") val eta: String,
  @Description("Mission ID") val missionID: Int
)

@Serializable
data class SpaceTool(
  @Description("The name of the tool") val name: String,
  @Description("The type of the tool") val type: String,
  @Description("The weight of the tool") val weight: Int
)

@Serializable
data class InterstellarCraft(
  @Description("The designation name of the spacecraft") val designation: String,
  @Description("The current mission name") val mission: String,
  @Description("Coordinates of the spacecraft") val coordinates: Pair<Int, Int>,
  @Description("Details related to the mission") val missionDetail: MissionDetail,
  @Description("A list of at least 10 tools that should be in the space craft")
  val tools: List<SpaceTool>
)

suspend fun main() {
  OpenAI.conversation {
    promptStreaming<InterstellarCraft>(Prompt("Make a spacecraft with a mission to Mars"))
      .collect { element ->
        when (element) {
          is StreamedFunction.Property -> {
            println("${element.path} = ${element.value}")
          }
          is StreamedFunction.Result -> {
            println(element.value)
          }
        }
      }
  }
}

When we run this program each line below corresponds to an individual property being streamed then the final object is returned.

[designation] = MarsCraft
[mission] = Mars Mission
[coordinates, first] = 0
[coordinates, second] = 0
[missionDetail, speed] = 10000
[missionDetail, destinationPlanets] = Mars
[missionDetail, eta] = 1 year
[missionDetail, missionID] = 123
[tools, name] = Navigation System
[tools, type] = Electronics
[tools, weight] = 100
[tools, name] = Life Support System
[tools, type] = Life Support
[tools, weight] = 200
[tools, name] = Communication System
[tools, weight] = 150
[tools, name] = Solar Panels
[tools, type] = Power Generation
[tools, weight] = 300
[tools, name] = Rocket Engines
[tools, type] = Propulsion
[tools, weight] = 500
[tools, name] = Scientific Instruments
[tools, type] = Research
[tools, weight] = 250
[tools, name] = Robot Arm
[tools, type] = Manipulator
[tools, name] = Sample Collection Tools
[tools, name] = Heat Shield
[tools, type] = Protection
[tools, weight] = 400
[tools, name] = Landing Gear
[tools, type] = Mechanical
InterstellarCraft(designation=MarsCraft, mission=Mars Mission, coordinates=(0, 0), missionDetail=MissionDetail(speed=10000, destinationPlanets=[Mars], eta=1 year, missionID=123), tools=[SpaceTool(name=Navigation System, type=Electronics, weight=100), SpaceTool(name=Life Support System, type=Life Support, weight=200), SpaceTool(name=Communication System, type=Electronics, weight=150), SpaceTool(name=Solar Panels, type=Power Generation, weight=300), SpaceTool(name=Rocket Engines, type=Propulsion, weight=500), SpaceTool(name=Scientific Instruments, type=Research, weight=250), SpaceTool(name=Robot Arm, type=Manipulator, weight=150), SpaceTool(name=Sample Collection Tools, type=Research, weight=100), SpaceTool(name=Heat Shield, type=Protection, weight=400), SpaceTool(name=Landing Gear, type=Mechanical, weight=200)])

Process finished with exit code 0

@javipacheco javipacheco merged commit cc27e13 into main Aug 21, 2023
@javipacheco javipacheco deleted the stream-functions branch August 21, 2023 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants