Complete No-API Self-Host Option

[BradKML]

Hello from another project! Can this repo be run with a local scraper and reliance on Ollama instead of paywalled systems? langgenius/dify#4565

[mattjoyce]

You can run the scraper locally, but the structured output is designed for openai from what I can see,
I also would like an Ollama option and while I think I could implement it, I'm not sure which model Ollama can run that would be good for OpenAI style json schemas.

I think the problem will often be context size, this project uses 32k.

[BradKML]

Yeah I was kinda concerned if the scraper is really SaaS or not, since for small tasks with full-blog crawling you don't need to get too fancy (maybe a few anti-bot dodge tactics but that is it)
Also for the context size, would using a RAG (embeddings and rerankers) or a large context model be able to help with the situation? How far is this from wget caching or ArchiveBox (similar to HTTRack)? ArchiveBox/ArchiveBox#191
Small reference to browser fingerprint bypassing: https://github.com/niespodd/browser-fingerprinting

[appinteractive]

You could try https://ollama.com/library/nexusraven with function calling. There might be other models that have bigger contexts but I would need to research further.

And ollana is fully openai compatible to my knowledge.

[BradKML]

Ollama is truly the go-to option but they still lagging over rerankers. For context windows Command-R and Gradient's finetuning are probs the best bet... Hopefully more models are supported in the future.

[appinteractive]

I think there are fine tuned versions of mistral with larger context windows. But I don't think they use 0.3 already with function calling.

[mattjoyce]

Mistral 0.3 was released recently, they have included structured output in that version.
However, It think with scraping context size is critical.

I'm going to have a go to include Ollama, and test.

[BradKML]

Could you document how Firecrawl can work with Python and Ollama?

[mattjoyce]

firecrawl has integrated llm calling in the api, it will be fair amount of work to make it natively work with Ollama.
but if you want to use firecrawl with Ollama you can do something like this....

from ollama import Client
import os
from typing import Optional
from firecrawl import FirecrawlApp

os.environ["FIRECRAWL_API_URL"] = "http://localhost:3002"
os.environ["FIRECRAWL_API_KEY"] = "your_api_key"
app=FirecrawlApp()
url = "https://www.humblebundle.com/books/cicd-mastery-for-cloud-infrastructure-packt-books"
params = {
    'pageOptions': {
        'onlyMainContent': True
    }
}

scraped_data = app.scrape_url(url)
prompt1 = f"""TASK -> Summerize this scraped webpage
CONTENT -> {scraped_data['markdown']}
"""


client = Client(host="http://localhost:11434")
response = client.chat(
    model="mistral:latest",
    keep_alive=600,
    options={"temperature": 0},
    messages=[
        {
            "role": "system",
            "content": "You excel at reading and interpreting scraped webpage data",
            "role": "user",
            "content": prompt1,
        },
    ],
)
response1 = response["message"]["content"]
print(scraped_data)
print(response1)

but that is not structured output.

[FingerLiu]

just use new llama 3.1， I tested it it works fine.

[BradKML]

That is good to know, including function calls right?

[FingerLiu]

yep

[BradKML]

Damn this needs to be integrated into systems like OpenDevin then