umer-wasim commited on
Commit
f88630b
·
verified ·
1 Parent(s): ad0651c

v0.5: trained on dataset v0.5

Browse files
README.md CHANGED
@@ -16,13 +16,13 @@ tags:
16
  library_name: transformers
17
  pipeline_tag: text-generation
18
  ---
19
- # Devseis Endpoint Auditor 0.5B (v0.3)
20
 
21
  A small language model fine-tuned by **Devseis** to write endpoint audit findings for **ISO 27001, GDPR and the EU AI Act**
22
  from collected evidence. It runs on the device (WebLLM in the [Devseis Endpoint Auditor](https://huggingface.co/spaces/Devseis/Arx)
23
  desktop app and phone check), so audit evidence never leaves the computer or phone.
24
 
25
- **v0.3:** First full release: balanced examples from dataset v0.3 (Windows, Linux, macOS, iPhone, Android; real CVEs; evidence labels; computed summary scores).
26
 
27
  ## What the model does — and does not do
28
 
@@ -37,14 +37,12 @@ app uses a built-in template instead. Details: [docs/MODEL_LLM.md](https://huggi
37
  ```python
38
  from transformers import AutoModelForCausalLM, AutoTokenizer
39
  repo = "Devseis/endpoint-auditor-0.5b"
40
- tok = AutoTokenizer.from_pretrained(repo, revision="v0.3")
41
- model = AutoModelForCausalLM.from_pretrained(repo, revision="v0.3")
42
  ```
43
 
44
  The LoRA adapter alone is in `lora/` (apply it to `Qwen/Qwen2.5-0.5B-Instruct` with PEFT). Browser builds:
45
- [Devseis/endpoint-auditor-0.5b-q0f16-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q0f16-MLC) (desktop),
46
- [Devseis/endpoint-auditor-0.5b-q4f16_1-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q4f16_1-MLC) (phones) and
47
- [Devseis/endpoint-auditor-0.5b-q4f32_1-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q4f32_1-MLC).
48
 
49
  ## Prompt format
50
 
@@ -56,11 +54,11 @@ prompt in that file. Tasks: `finding`, `summary`, `ai_classification`; the model
56
  | | |
57
  |---|---|
58
  | Base model | [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) (Apache-2.0) |
59
- | Data | [Devseis/endpoint-auditor-synthetic](https://huggingface.co/datasets/Devseis/endpoint-auditor-synthetic) revision `v0.3` (synthetic, CC BY 4.0) |
60
- | Examples | 3000 balanced over task, OS and status (3072 tokens) |
61
- | Method | LoRA r=16, alpha=32 (8.8 M trainable parameters), lr 2e-4, 1 epoch, 375 steps of 8 examples |
62
  | Hardware | CPU only: a 4-core Intel MacBook Pro (PyTorch 2.2, eager attention) |
63
- | Final validation loss | 0.0109 |
64
 
65
  ## Evaluation
66
 
@@ -71,19 +69,21 @@ uses; *fields match* = check id / status / risk (findings), score and top gaps (
71
 
72
  | Task | n | Valid JSON | Grounded | Fields match |
73
  |---|---|---|---|---|
74
- | ai_classification | 9 | 0% | 0% | 0% |
75
- | finding | 43 | 0% | 0% | 0% |
76
- | summary | 8 | 0% | 0% | 0% |
77
- | **all** | 60 | **0%** | **0%** | **0%** |
 
78
 
79
- **This model (v0.3):**
80
 
81
  | Task | n | Valid JSON | Grounded | Fields match |
82
  |---|---|---|---|---|
83
- | ai_classification | 9 | 100% | 100% | 100% |
84
- | finding | 43 | 100% | 100% | 100% |
85
- | summary | 8 | 100% | 100% | 100% |
86
- | **all** | 60 | **100%** | **100%** | **100%** |
 
87
 
88
  ## Limitations
89
 
@@ -101,7 +101,7 @@ Apache-2.0, by **Devseis**. Please credit "Devseis Endpoint Auditor by Devseis".
101
  author = {{Devseis}},
102
  title = {Devseis Endpoint Auditor 0.5B},
103
  year = {2026},
104
- version = {v0.3},
105
  url = {https://huggingface.co/Devseis/endpoint-auditor-0.5b}
106
  }
107
  ```
 
16
  library_name: transformers
17
  pipeline_tag: text-generation
18
  ---
19
+ # Devseis Endpoint Auditor 0.5B (v0.5)
20
 
21
  A small language model fine-tuned by **Devseis** to write endpoint audit findings for **ISO 27001, GDPR and the EU AI Act**
22
  from collected evidence. It runs on the device (WebLLM in the [Devseis Endpoint Auditor](https://huggingface.co/spaces/Devseis/Arx)
23
  desktop app and phone check), so audit evidence never leaves the computer or phone.
24
 
25
+ **v0.5:** Qwen2.5-0.5B fine-tuned for chat and reports in one model (2,400 examples: the v0.3 report tasks plus grounded chat, a third of the chat prompts carrying MY DEVICE, what the browser detected). Small enough for phones and weak graphics chips; powers the Arx chat.
26
 
27
  ## What the model does — and does not do
28
 
 
37
  ```python
38
  from transformers import AutoModelForCausalLM, AutoTokenizer
39
  repo = "Devseis/endpoint-auditor-0.5b"
40
+ tok = AutoTokenizer.from_pretrained(repo, revision="v0.5")
41
+ model = AutoModelForCausalLM.from_pretrained(repo, revision="v0.5")
42
  ```
43
 
44
  The LoRA adapter alone is in `lora/` (apply it to `Qwen/Qwen2.5-0.5B-Instruct` with PEFT). Browser builds:
45
+ [Devseis/endpoint-auditor-0.5b-q0f16-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q0f16-MLC), [Devseis/endpoint-auditor-0.5b-q4f16_1-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q4f16_1-MLC), [Devseis/endpoint-auditor-0.5b-q4f32_1-MLC](https://huggingface.co/Devseis/endpoint-auditor-0.5b-q4f32_1-MLC).
 
 
46
 
47
  ## Prompt format
48
 
 
54
  | | |
55
  |---|---|
56
  | Base model | [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) (Apache-2.0) |
57
+ | Data | [Devseis/endpoint-auditor-synthetic](https://huggingface.co/datasets/Devseis/endpoint-auditor-synthetic) revision `v0.5` (synthetic, CC BY 4.0) |
58
+ | Examples | 2400 balanced over task, OS and status (3072 tokens) |
59
+ | Method | LoRA r=16, alpha=32 (8.8 M trainable parameters), lr 2e-4, 1 epoch, 300 steps of 8 examples |
60
  | Hardware | CPU only: a 4-core Intel MacBook Pro (PyTorch 2.2, eager attention) |
61
+ | Final validation loss | 0.0162 |
62
 
63
  ## Evaluation
64
 
 
69
 
70
  | Task | n | Valid JSON | Grounded | Fields match |
71
  |---|---|---|---|---|
72
+ | ai_classification | 4 | 0% | 0% | 0% |
73
+ | chat | 38 | 100% | 95% | 0% |
74
+ | finding | 15 | 0% | 0% | 0% |
75
+ | summary | 3 | 0% | 0% | 0% |
76
+ | **all** | 60 | **63%** | **60%** | **0%** |
77
 
78
+ **This model (v0.5):**
79
 
80
  | Task | n | Valid JSON | Grounded | Fields match |
81
  |---|---|---|---|---|
82
+ | ai_classification | 4 | 100% | 100% | 100% |
83
+ | chat | 38 | 100% | 97% | 82% |
84
+ | finding | 15 | 100% | 100% | 100% |
85
+ | summary | 3 | 100% | 100% | 100% |
86
+ | **all** | 60 | **100%** | **98%** | **88%** |
87
 
88
  ## Limitations
89
 
 
101
  author = {{Devseis}},
102
  title = {Devseis Endpoint Auditor 0.5B},
103
  year = {2026},
104
+ version = {v0.5},
105
  url = {https://huggingface.co/Devseis/endpoint-auditor-0.5b}
106
  }
107
  ```
config.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "_name_or_path": "/Users/umer/Documents/6 - mix/endpoint-audit/training/models/Qwen2.5-0.5B-Instruct",
3
  "architectures": [
4
  "Qwen2ForCausalLM"
5
  ],
 
1
  {
2
+ "_name_or_path": "training/models/Qwen2.5-0.5B-Instruct",
3
  "architectures": [
4
  "Qwen2ForCausalLM"
5
  ],
eval/base-eval.json CHANGED
@@ -1,31 +1,37 @@
1
  {
2
  "model": "base model",
3
- "data": "training/out-v0.3/test.jsonl",
4
  "count": 60,
5
  "seed": 5,
6
  "tasks": {
7
  "ai_classification": {
8
- "n": 9,
9
  "valid_json": 0.0,
10
  "grounded": 0.0,
11
  "fields_match": 0.0
12
  },
 
 
 
 
 
 
13
  "finding": {
14
- "n": 43,
15
  "valid_json": 0.0,
16
  "grounded": 0.0,
17
  "fields_match": 0.0
18
  },
19
  "summary": {
20
- "n": 8,
21
  "valid_json": 0.0,
22
  "grounded": 0.0,
23
  "fields_match": 0.0
24
  }
25
  },
26
  "overall": {
27
- "valid_json": 0.0,
28
- "grounded": 0.0,
29
  "fields_match": 0.0
30
  }
31
  }
 
1
  {
2
  "model": "base model",
3
+ "data": "training/out-v0.5/test.jsonl",
4
  "count": 60,
5
  "seed": 5,
6
  "tasks": {
7
  "ai_classification": {
8
+ "n": 4,
9
  "valid_json": 0.0,
10
  "grounded": 0.0,
11
  "fields_match": 0.0
12
  },
13
+ "chat": {
14
+ "n": 38,
15
+ "valid_json": 1.0,
16
+ "grounded": 0.9473684210526315,
17
+ "fields_match": 0.0
18
+ },
19
  "finding": {
20
+ "n": 15,
21
  "valid_json": 0.0,
22
  "grounded": 0.0,
23
  "fields_match": 0.0
24
  },
25
  "summary": {
26
+ "n": 3,
27
  "valid_json": 0.0,
28
  "grounded": 0.0,
29
  "fields_match": 0.0
30
  }
31
  },
32
  "overall": {
33
+ "valid_json": 0.6333333333333333,
34
+ "grounded": 0.6,
35
  "fields_match": 0.0
36
  }
37
  }
eval/model-eval.json CHANGED
@@ -1,23 +1,29 @@
1
  {
2
- "model": "adapter training/models/auditor-lora-v0.3/final",
3
- "data": "training/out-v0.3/test.jsonl",
4
  "count": 60,
5
  "seed": 5,
6
  "tasks": {
7
  "ai_classification": {
8
- "n": 9,
9
  "valid_json": 1.0,
10
  "grounded": 1.0,
11
  "fields_match": 1.0
12
  },
 
 
 
 
 
 
13
  "finding": {
14
- "n": 43,
15
  "valid_json": 1.0,
16
  "grounded": 1.0,
17
  "fields_match": 1.0
18
  },
19
  "summary": {
20
- "n": 8,
21
  "valid_json": 1.0,
22
  "grounded": 1.0,
23
  "fields_match": 1.0
@@ -25,7 +31,7 @@
25
  },
26
  "overall": {
27
  "valid_json": 1.0,
28
- "grounded": 1.0,
29
- "fields_match": 1.0
30
  }
31
  }
 
1
  {
2
+ "model": "adapter training/models/auditor-lora-v0.5/final",
3
+ "data": "training/out-v0.5/test.jsonl",
4
  "count": 60,
5
  "seed": 5,
6
  "tasks": {
7
  "ai_classification": {
8
+ "n": 4,
9
  "valid_json": 1.0,
10
  "grounded": 1.0,
11
  "fields_match": 1.0
12
  },
13
+ "chat": {
14
+ "n": 38,
15
+ "valid_json": 1.0,
16
+ "grounded": 0.9736842105263158,
17
+ "fields_match": 0.8157894736842105
18
+ },
19
  "finding": {
20
+ "n": 15,
21
  "valid_json": 1.0,
22
  "grounded": 1.0,
23
  "fields_match": 1.0
24
  },
25
  "summary": {
26
+ "n": 3,
27
  "valid_json": 1.0,
28
  "grounded": 1.0,
29
  "fields_match": 1.0
 
31
  },
32
  "overall": {
33
  "valid_json": 1.0,
34
+ "grounded": 0.9833333333333333,
35
+ "fields_match": 0.8833333333333333
36
  }
37
  }
lora/adapter_config.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "alpha_pattern": {},
3
  "auto_mapping": null,
4
- "base_model_name_or_path": "/Users/umer/Documents/6 - mix/endpoint-audit/training/models/Qwen2.5-0.5B-Instruct",
5
  "bias": "none",
6
  "fan_in_fan_out": false,
7
  "inference_mode": true,
@@ -21,12 +21,12 @@
21
  "revision": null,
22
  "target_modules": [
23
  "up_proj",
24
- "gate_proj",
25
  "down_proj",
 
 
26
  "v_proj",
27
- "o_proj",
28
- "q_proj",
29
- "k_proj"
30
  ],
31
  "task_type": "CAUSAL_LM",
32
  "use_dora": false,
 
1
  {
2
  "alpha_pattern": {},
3
  "auto_mapping": null,
4
+ "base_model_name_or_path": "training/models/Qwen2.5-0.5B-Instruct",
5
  "bias": "none",
6
  "fan_in_fan_out": false,
7
  "inference_mode": true,
 
21
  "revision": null,
22
  "target_modules": [
23
  "up_proj",
24
+ "q_proj",
25
  "down_proj",
26
+ "gate_proj",
27
+ "k_proj",
28
  "v_proj",
29
+ "o_proj"
 
 
30
  ],
31
  "task_type": "CAUSAL_LM",
32
  "use_dora": false,
lora/adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:053e52925b7f66ba1269a0d31717ca432a5bcb9502da462750ca3d6a82170010
3
  size 35237104
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:861d4711656a9ac65192f69dd705577dc84985fd0da17d5e38c5c88cdcc3e3da
3
  size 35237104
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:786b626c904841ff4f4478b9b65df3e6d4a19d6f68296bd00a2175dc35310b54
3
  size 988097536
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef55a602e0e439f543e8102603fecdff3a405075265f024f7e99e4034c257200
3
  size 988097536