Skip to content

Commit

Permalink
document temperature and system prompt sensitivity
Browse files Browse the repository at this point in the history 
Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com>
  • Loading branch information
@prabhu
prabhu committed Feb 8, 2025
1 parent 943afde commit 66927d0
Show file tree
Hide file tree
Showing 4 changed files with 245 additions and 9 deletions.
6 changes: 2 additions & 4 deletions contrib/fine-tuning/Modelfile
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
FROM ./cdx1-gguf-q8_0.gguf

PARAMETER num_ctx 16384
PARAMETER num_ctx 16000
PARAMETER temperature 0.05
PARAMETER top_k 10
PARAMETER top_p 0.5

SYSTEM """You are cdxgen, an expert in CycloneDX and xBOM."""
SYSTEM """You are a helpful assistant."""

LICENSE """
apache-2.0
Expand Down
10 changes: 5 additions & 5 deletions contrib/fine-tuning/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,13 @@ bash fine-tune-mlx.sh
cp -rf prabhuat ~/.lmstudio/models/
lms ls
lms server status
lms load CycloneDX/cdx1-mlx --exact --gpu max --identifier cdx1-test --context-length 8192
lms load CycloneDX/cdx1-mlx-8bit --exact --gpu max --identifier cdx1-test --context-length 16000
```

System prompt:

```text
You are cdxgen, an expert in CycloneDX and xBOM.
You are a helpful assistant.
```

### gguf testing with ollama
Expand All @@ -53,18 +53,18 @@ ollama show cdx1-gguf
Model
architecture llama
parameters 14.7B
context length 16384
context length 16000
embedding length 5120
quantization F16
Parameters
num_ctx 16384
num_ctx 16000
temperature 0.05
top_k 10
top_p 0.5
System
You are cdxgen, an expert in CycloneDX and xBOM.
You are a helpful assistant.
License
apache-2.0
Expand Down
2 changes: 2 additions & 0 deletions contrib/xBOMEval/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Introduction

236 changes: 236 additions & 0 deletions contrib/xBOMEval/cdx1-NOTES.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,236 @@
## System Prompt sensitivity

Using the system prompt `You are a helpful assistant.` improves the quality of the generated text. Below we document the responses for the same question with alternative system prompts.

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 --system-prompt "You are cdxgen, an expert in CycloneDX and xBOM."
==========
The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the BOM format, such as "bom-1.4" for CycloneDX 1.4.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 --system-prompt "You are cdxgen, an helpful assistant."
==========
The bomFormat property specifies the format of the BOM. It is a required property and should be set to "application/vnd.cyclonedx.bom+json" for a CycloneDX BOM.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 --system-prompt "You are cdxgen, an expert in CycloneDX specification and cdxgen tool."
==========
The bomFormat property specifies the format of the BOM. It can be set to "application/vnd.cyclonedx.bom+json" for JSON format or "application/vnd.cyclonedx.bom+xml" for XML format.
==========
```

Good results with this system prompt.

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 --system-prompt "You are a helpful assistant."
==========
The `bomFormat` property specifies the format of the BOM. It is a required property and should be set to "CycloneDX".
==========
```

Repeat the test with multiple temperature.

Temperature: 0.1

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.1 --system-prompt "You are a helpful assistant."
==========
The `bomFormat` property specifies the format of the BOM. It is a required property and should be set to "CycloneDX".
==========
```

Above 0.1, the models gets **lazy**.

Temperature: 0.2

```
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.2 --system-prompt "You are a helpful assistant."
==========
The `bomFormat` property specifies the format of the BOM. It is a string value that identifies the format of the BOM.
==========
```

Temperature: 0.3

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.3 --system-prompt "You are a helpful assistant."
==========
The `bomFormat` property specifies the format of the BOM. It is a string value and is required.
==========
```

Temperature: 0.4

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.4 --system-prompt "You are a helpful assistant."
==========
The `bomFormat` property specifies the format of the BOM. It is a string value and can be used to indicate the format of the BOM, such as "cyclonedx", "spdx", or "sbom".
==========
```

Temperature: 0.5

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.5 --system-prompt "You are a helpful assistant."
==========
The `bomFormat` property specifies the format of the BOM.
==========
```

Temperature: 0.6 - 1.0 (Identical responses)

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.6 --system-prompt "You are a helpful assistant."
==========
The `bomFormat` property specifies the format of the BOM.
==========
```

## Temperature sensitivity

We tested temperature sensitivity without a system prompt. Below are the results:

0.05 - 0.1 is the optimal temperature for `cdx1` family of models. Anything above 0.1 reduces the quality of the generated response.

### Test

Question: What is the purpose of the bomFormat property in a CycloneDX BOM?
Answer: The `bomFormat` property specifies the format of the Bill of Materials (BOM). It is used to identify the file as a CycloneDX BOM because BOMs do not inherently have a filename convention, and JSON schema does not support namespaces for this purpose. The value of this property must always be \"CycloneDX\" to clearly indicate that the document is indeed a CycloneDX BOM.

### Responses

For the fused mlx model `bf16`, `0.1` appears to yield a decent response.

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05
==========
The bomFormat property specifies the format of the BOM. It is used to indicate the format of the BOM, such as JSON, XML, or YAML.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.1
==========
The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other relevant information about the BOM format.
==========
```

For the quantized versions, 0.05 and 0.1 yield acceptable response.

Temperature: 0

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0
==========
The bomFormat property specifies the format of the BOM. It is used to indicate the version of the CycloneDX specification that the BOM conforms to.
==========
```

Temperature: 0.01

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.01
==========
The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other formats.
==========
```

Temperature: 0.05

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05
==========
The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other formats.
==========
```

Temperature: 0.1

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.1
==========
The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other formats.
==========
```

Temperature: 0.2

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.2
==========
The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other format identifiers.
==========
```

Temperature: 0.3

Creative answers begin from 0.3 onwards.

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.3
==========
The bomFormat property specifies the format of the BOM. It's always set to "CBOR" for CycloneDX BOMs.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.4
==========
The bomFormat property specifies the format of the BOM. It's always set to "CBOR" for CycloneDX BOMs.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.5
==========
The bomFormat property specifies the format of the BOM, such as JSON or XML.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.6
==========
The bomFormat property specifies the format of the BOM, such as JSON or XML.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.7
==========
The bomFormat property represents the version of the BOM specification that the document adheres to. It's used to ensure that the BOM is compliant with the specified version.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.8
==========
The bomFormat property represents the version of the BOM specification that the document adheres to. It's used to ensure that the schema used to validate the document is correct.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.9
==========
The bomFormat property represents the version of the BOM specification that the document adheres to, allowing for interoperability and understanding of schema format.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 1.0
==========
The bomFormat property represents the version of the BOM specification that the document adheres to, allowing for context and interpretation of the schema format.
==========
```

```text
mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 1.1
==========
The bomFormat property represents the version of the BOM specification that the document adheres to, supported formats include bom-1, bom-10, bom-11, bom-12, and nonconformant for custom implementations.
==========
```

Beyond 1.1, the responses degrade significantly.

0 comments on commit 66927d0

Please sign in to comment.