I am running qwen2.5-1.5b Instruct-q4_k_m.gg. on my galaxy s35. I copied and pasted new parameters and an API key of here but I still can’t get it to do what I want. Im new to all this, really dont know what im doing 
but im trying. My question is can I use Heretic to do this or is that not for a phone.
Oh. I think there’s probably a simpler way to do this
:
If you are already running Qwen2.5-1.5B-Instruct-Q4_K_M.gguf locally on your phone, you probably do not need to run Heretic itself on the phone.
There is already a Heretic-processed version of the same Qwen2.5 1.5B model with a Q4_K_M GGUF available here:
saidutta69/Qwen2.5-1.5B-Instruct-heretic
The repo currently provides:
Qwen2.5-1.5B-Instruct-heretic-Q4_K_M.gguf
So, if the Android app you are using can load ordinary GGUF models, the first thing I would try is simply:
your current Qwen2.5-1.5B-Instruct Q4_K_M
↓
same Android app
same prompt
same/default-ish settings
↓
Qwen2.5-1.5B-Instruct-heretic Q4_K_M
In other words: change the model file first, not everything else at the same time.
That gives you a much cleaner test.
The important caveat is that this will mainly help if “I can’t get it to do what I want” means that the normal Qwen model is refusing requests. Heretic modifies the model weights to reduce refusal behavior; it does not turn a 1.5B model into a more capable model.
The Heretic model card makes the same distinction: the modification suppresses refusals, but does not add capability or judgment.
How I would test it
First make sure your current official Qwen model can answer a simple harmless prompt normally.
The official Qwen GGUF is here:
Qwen/Qwen2.5-1.5B-Instruct-GGUF
Then load the Heretic Q4_K_M version in the same app, preferably without changing a lot of generation parameters at the same time.
The result gives a useful first split:
Heretic version now does what you wanted
→ refusal/alignment was probably an important part of the problem
Both models generate normal text, but neither does what you wanted
→ probably not primarily an Android/GGUF problem
→ look at the prompt/system prompt, or the limits of a 1.5B model
Original Qwen works, but the Heretic GGUF will not load
→ check the model download and the Android app/runtime's GGUF support
Both models give broken/empty/repetitive output
→ look at the app/runtime, chat formatting, or backend rather than Heretic itself
The app crashes or gets killed
→ memory/context settings become more interesting
This is why I would avoid changing the model, sampler parameters, context size, API settings, etc. all at once. One change gives you much more information.
The API key is a separate thing
I think this part may be causing some confusion.
A Hugging Face access token and a local GGUF model are different layers.
Very roughly:
Hugging Face token
→ authentication to Hugging Face services/resources
→ for example downloading something that requires authentication,
or calling a hosted service
GGUF file
→ the actual model file your local inference app loads
Heretic
→ a tool that modifies model weights
→ or, in this case, someone has already done that work
and published the resulting GGUF for you
So adding a Hugging Face API/access token does not change the weights or refusal behavior of a GGUF that is already running locally.
Your particular app may still use a Hugging Face token for downloading models or some other Hub feature, so the token is not necessarily useless — it is just separate from the local model’s behavior.
Hugging Face’s token documentation is here:
Also, do not post the actual token publicly. If a real token has already been exposed somewhere, revoke/rotate it.
For this particular Heretic model, the model page currently also says that it is not deployed by an Inference Provider, but that does not prevent you from downloading/running the GGUF locally.
Is GGUF on Android a reasonable approach?
Yes. Running GGUF models locally on Android is a real and fairly well-established path.
llama.cpp itself has current Android documentation covering an Android Studio binding, GGUF loading, automatic prompt/template formatting, Termux, and Android NDK builds:
llama.cpp Android documentation
So the useful distinction here is not really:
Heretic = desktop
Qwen GGUF = phone
It is closer to:
Heretic
→ model-modification step
Heretic-processed GGUF
→ finished model artifact
Android GGUF app / llama.cpp-based runtime
→ inference step
The modification can happen elsewhere. Your phone only needs to be able to run the resulting GGUF.
The Heretic Qwen model page explicitly publishes multiple GGUF quantizations, including Q4_K_M, and shows llama.cpp as a supported local runtime:
If the output itself looks broken
If the model loads but you see things such as:
- empty responses
- strange role markers
- the model continuing your prompt instead of answering
- endless/repetitive generation
- failure to stop properly
then I would not immediately blame Heretic.
Chat/instruct models expect their messages in a particular format. Hugging Face calls this a chat template. Different models can require different control tokens even when the visible conversation looks the same.
Qwen2.5 uses an instruct/chat format, and a good GGUF runner will normally take care of that formatting for you. The Heretic repo also includes the corresponding tokenizer/chat-template material, so there is no particular reason at this point to assume that the derivative model simply “has no template.”
But Android apps can bundle different versions of llama.cpp or implement the chat layer differently. So if text generation itself is malformed, the app name and version become much more useful information.
There have been Android integration issues where a Qwen GGUF loaded and generated, but stopping/EOG handling differed between applications. That does not mean you have the same bug; it just shows why “the GGUF loads” and “the chat integration is correct” are two different tests.
If it is slow or crashes
I would only investigate this branch if you actually have a performance or stability problem.
For memory, start with a modest context size rather than a huge one. The llama.cpp Android documentation specifically suggests something around 4096 as a reasonable starting point because an excessive context can cause a memory spike and get the process killed:
llama.cpp Android documentation
For speed, Android is also a little unintuitive: using the maximum possible CPU thread count does not necessarily give the best performance on heterogeneous mobile CPUs.
So I would first use the app’s defaults or conservative settings, get a known-good baseline, and optimize afterward.
There are also separate CPU/GPU/Vulkan/OpenCL issues depending on the phone and runtime, but I would not start there unless the basic CPU/local path is actually failing.
If you want to narrow it down further
You do not need to collect a huge debug log yet.
The most useful information would simply be:
Android app + version:
Exact Galaxy model:
Harmless example prompt:
Actual response:
What you wanted it to do instead:
That last part is especially useful.
If the actual response is basically:
"I can't help with that..."
then trying the Heretic GGUF is a very direct experiment.
If the response is instead confused, inaccurate, unable to follow a complicated task, or simply not capable enough, then a refusal-removal model may not solve the real problem and a larger/better-suited model or different prompt would be the next branch.
One final small note: the Heretic model above is a community derivative, not an official Qwen release. Its model-card evaluation numbers are the uploader’s evaluation, so I would treat them as useful information rather than an independent guarantee.
It also intentionally removes a lot of refusal behavior. For private local experimentation that may be exactly what you want, but if you ever expose it as a service to other people, remember that the model card explicitly says there is no extra safety-filtering layer.
For your current phone setup, though, I would start with the simple experiment: download the Heretic Q4_K_M GGUF and try it in the same local app before changing anything else.