Skip to content

v0.0.54.1
Compare
Choose a tag to compare
@cmodi-meta cmodi-meta released this 06 Dec 22:33
v0.0.54.1
0a68271
This commit was created on github.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

We are excited to announce a major update to the Llama Stack Kotlin Library which now supports both local and remote inferencing on Android apps. Building on the existing remote inferencing capabilities, this release introduces significant changes to enable seamless local inferencing integration and providing developers more flexibility with their AI workflows. This release focuses on delivering features centered around these capabilities.

Release v0.0.54.1 includes local modules as part of the Kotlin Library dependency in Maven

Local Inference Support

  • Leverage ExecuTorch on-device framework (commit: 0a12e33) for on-device inferencing.
  • Script for downloading ExecuTorch aar file.
  • Allow passing various configurations from Android app: .pte and tokenizer file, sequence length, and temperature.
  • Send stats metrics from ExecuTorch (tok/sec).
  • Handle prompt formatting based on model.
  • Support conversational history.

Remote Inference Support

  • Enabled remote support with Llama Stack server v0.0.54
  • Fix lib compile issues due to Stainless autogen invalid types (link and link)

Supporting Models

  • The models supported in the app vary based on whether you are doing remote or local inferencing.

Remote Model Support

  • For remote usage, the app supports all models that the Llama Stack server backend supports. This includes a range of models from the lightweight 1B Llama models to the extensive 405B models.

Local Model Support

For on-device usage, the following models are supported:

  • Llama 3.2 Quantized 1B/3B
  • Llama 3.2 1B/3B in BF16
  • Llama 3.1 8B Quantized
  • Llama 3 8B Quantized

Getting Started

  • Pointer to an Android demo app. (Note tag: android-0.0.54.1)
  • Quick start instructions on how to add the Kotlin SDK to their Android app
  • Instructions on how a power developer can contribute to the Kotlin SDK, debug or just play with it to learn more.

If you have any questions feel free to raise an issue and we’d be happy to help!

What’s Next?

This is only the beginning with enabling features on Llama Stack to run on Android devices. We will continue to expand the capabilities of Llama Stack, new use cases, and applications! Specifically we look to focus on:

  • Agentic workflow with streaming
  • Image and Speech reasoning
  • Local/on-device agentic components like memory banks
  • Examples with RAG usage

Stay tuned on future releases and updates!

Contributors

@cmodi-meta, @dltn, @Riandy, @WuhanMonkey, @yanxi0830, and big thank you to the ExecuTorch team.

Contributors

Riandy, WuhanMonkey, and 3 other contributors
Assets 2
Loading