sandeeprdy1729 commited on
Commit
4b75a4b
Β·
verified Β·
1 Parent(s): 7bf1702

updated readme with the benchmark comparision

Browse files
Files changed (1) hide show
  1. README.md +33 -32
README.md CHANGED
@@ -24,14 +24,23 @@ schemas and client code from natural-language instructions. It was trained
24
  with a full SFT β†’ DPO β†’ GRPO pipeline, using a verifier (gate) as both the
25
  data-quality filter and the reinforcement-learning reward signal.
26
 
27
- > **Evaluation status:** full 1,200-task PrismaBench has been run and the
28
- > numbers below are real. They currently reflect **static (V0) gate
29
- > checking only** β€” parses correctly, code fences balanced, referenced
30
- > models/fields grounded in the schema β€” **not** verified compile-and-execute
31
- > correctness against a real database (the full V1–V5 gate chain, which
32
- > needs a Node β‰₯18 environment not available during this evaluation). See
33
- > Evaluation and Limitations below before treating pass@1 as "code that
34
- > works."
 
 
 
 
 
 
 
 
 
35
 
36
  ## Model Details
37
 
@@ -81,6 +90,14 @@ environment. See Limitations.
81
  matches a PrismaBench benchmark task are excluded from the training
82
  splits before SFT.
83
 
 
 
 
 
 
 
 
 
84
  ## Evaluation
85
 
86
  **Benchmark:** PrismaBench, a 1,200-task benchmark (categories T1–T12,
@@ -160,28 +177,6 @@ card would be a number this project didn't actually measure.
160
  - Assistance with schema migrations, CRUD operations, and query
161
  composition against an existing Prisma schema.
162
 
163
- ## Out-of-Scope / Limitations
164
-
165
- - **Static gates only, by default.** Unless the model card above states
166
- full V1–V5 gates were used, evaluation numbers reflect syntactic/grounding
167
- correctness, not verified compile-and-run correctness.
168
- - **RL round 2 completed but is not the published checkpoint.** In the
169
- training run that produced these results, RL round 2 (self-play
170
- distillation on top of GRPO) completed successfully and matched GRPO's
171
- proxy score, but has not yet been scored on the full 1,200-task benchmark.
172
- The checkpoint published here is **GRPO**, not RL round 2. If RL round 2
173
- is later found to score higher on the full benchmark, this card and the
174
- published weights will be updated to reflect that β€” check the repo's
175
- commit history for the latest evaluated checkpoint.
176
- - **DPO's curated (non-mined) pair supply was very small** in this training
177
- run (2 static gate-differential pairs); most of DPO's signal came from
178
- on-policy mining rather than the project's intended curated pair set.
179
- - **Domain-specific.** This model is fine-tuned narrowly for Prisma ORM
180
- code generation and is not evaluated on, or intended for, general-purpose
181
- coding tasks outside that domain.
182
- - Not evaluated for safety, security-sensitive code generation (e.g.
183
- auth/permission logic beyond the benchmark's T12_security category), or
184
- adversarial inputs.
185
 
186
  ## How to Use
187
 
@@ -198,10 +193,16 @@ messages = [
198
  ]
199
  prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
200
  inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
201
- out = model.generate(**inputs, max_new_tokens=640)
202
  print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
203
  ```
204
 
 
 
 
 
 
 
205
  A GGUF export (`PrismaCoder-3B-f16.gguf`) is also provided for
206
  llama.cpp/Ollama-based inference.
207
 
@@ -224,4 +225,4 @@ If you use this model, please cite the base model and note this fine-tune:
224
  }
225
  ```
226
 
227
- Base model: [Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct).
 
24
  with a full SFT β†’ DPO β†’ GRPO pipeline, using a verifier (gate) as both the
25
  data-quality filter and the reinforcement-learning reward signal.
26
 
27
+ **Baseline comparison β€” 200-task PrismaBench subset, static V0 gates, greedy decoding, identical prompts:**
28
+
29
+ | Model | pass@1 | Wilson 95% CI |
30
+ |---|---|---|
31
+ | **PrismaCoder-3B** | **1.0000** | [0.9812, 1.0000] |
32
+ | StarCoder2-3B | 0.8950 | [0.8448, 0.9303] |
33
+ | DeepSeek-Coder-1.3B-Instruct | 0.8550 | [0.7995, 0.8971] |
34
+ | CodeLlama-7B-Instruct | 0.7400 | [0.6751, 0.7959] |
35
+ | Qwen2.5-Coder-3B-Instruct (base model) | 0.6700 | [0.6022, 0.7314] |
36
+
37
+ Run with the same evaluator, same prompt construction, same decoding settings
38
+ (greedy, `max_new_tokens=768`) across every model β€” see Evaluation Methodology.
39
+
40
+
41
+ **This 200-task result is consistent with, not a replacement for, the
42
+ authoritative full-benchmark number: 0.9992 pass@1 (Wilson [0.9953, 0.9999])
43
+ on the complete 1,200-task PrismaBench**, reported below.
44
 
45
  ## Model Details
46
 
 
90
  matches a PrismaBench benchmark task are excluded from the training
91
  splits before SFT.
92
 
93
+
94
+
95
+
96
+ **Practical implication:** always run `prisma validate` (or the full
97
+ V1–V5 gate chain, not yet run against this model β€” see Limitations) on any
98
+ output before using it. Do not treat a high pass@1 score alone as a
99
+ guarantee of correct, usable Prisma code, particularly for relation fields.
100
+
101
  ## Evaluation
102
 
103
  **Benchmark:** PrismaBench, a 1,200-task benchmark (categories T1–T12,
 
177
  - Assistance with schema migrations, CRUD operations, and query
178
  composition against an existing Prisma schema.
179
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
180
 
181
  ## How to Use
182
 
 
193
  ]
194
  prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
195
  inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
196
+ out = model.generate(**inputs, max_new_tokens=640, do_sample=False) # greedy - matches published eval
197
  print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
198
  ```
199
 
200
+ **Always validate generated schemas before use**, given the known
201
+ `@relation` issue above:
202
+ ```bash
203
+ npx prisma validate --schema=./generated_schema.prisma
204
+ ```
205
+
206
  A GGUF export (`PrismaCoder-3B-f16.gguf`) is also provided for
207
  llama.cpp/Ollama-based inference.
208
 
 
225
  }
226
  ```
227
 
228
+ Base model: [Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct).