from Hacker News

Ask HN: How are you using llama.cpp?

by danielEM on 7/12/24, 5:15 PM with 2 comments

For what purposes you use llama.cpp?

What tools do you use with llama.cpp?

Is there anything you recommend to avoid when it comes to llama.cpp?

Want to collect your best practices/experiences and advice around llama.cpp. Eg. if you work with Visual Studio Code - what plugins you recommend, and what not. Etc...

  • by roosgit on 7/12/24, 7:34 PM

    I use it to help me write text.

    Don't use any tools. I run it from the command line:

    ./main -f ~/Desktop/prompts/multishot/llama3-few-shot-prompt-10.txt -m ~/Desktop/models/Meta-Llama-3-8B-Instruct-Q8_0.gguf --temp 0 --color -c 1024 -n -1 --repeat_penalty 1.2 -tb 8 --log-disable 2>/dev/null

    I prefer `main` to the new `llama-cli` because when searching history for "llama" I want to get commands that contain the "llama" models, not "mistral" ones, for example.

  • by caprock on 7/12/24, 5:22 PM

    I've not used it myself, but I see a lot of people referencing Ollama. It uses llama.cpp (and maybe more).