from Hacker News

What's new in Llama 2 and how to run it locally

by andrewon on 8/6/23, 11:40 PM with 82 comments

  • by visarga on 8/7/23, 6:28 AM

    In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models.

    The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of people innovating and debating approaches, lots of projects popping up, so much energy!

  • by kordlessagain on 8/7/23, 1:38 PM

    I've been evaluating running non-quantized models on a Google Cloud instance with various GPUs.

    To run a `vllm` backed Llama 2 7b model[1], start a Debian 11 spot instance, with (1) Nvidia L4 and a g2-standard-8 w/100GB of SSD disk (ignoring the advice to use a Cuda installer image):

      sudo apt-get update -y
  sudo apt-get install build-essential -y
  sudo apt-get install linux-headers-$(uname -r) -y
  wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
  sudo sh cuda_11.8.0_520.61.05_linux.run # ~5 minutes, install defaults, type 'accept'/return
  sudo apt-get install python3-pip -y
  sudo pip install --upgrade huggingface_hub 
  
  # skip using token as git credential
  huggingface-cli login (for Meta model access paste token from HF[2])
  
  sudo pip install vllm # ~8 minutes
    Then, edit the test code for a 7b Llama 2 model (paste into llama.py):

      from vllm import LLM
  llm = LLM(model="meta-llama/Llama-2-7b-hf")
  output = llm.generate("The capital of Brazil is called")
  print(output)
    Spot price for this deployment is ~$225/month. The instance will eventually be terminated by Google, so plan accordingly.

    [1] https://vllm.readthedocs.io/en/latest/models/supported_model... [2] https://huggingface.co/settings/tokens

  • by jurmous on 8/7/23, 7:14 AM

    Did anybody try the Llama 2 model with languages other than English? The paper notes that it works best with English and the amount of training data for other languages is only a fraction. Which likely would make it unusable for me..

    See table 10 (page 22) of the whitepaper for the numbers: https://ai.meta.com/research/publications/llama-2-open-found...

    Are there other downloadable models which can be used in a multilingual environment that people here are aware of?

  • by jmorgan on 8/7/23, 11:15 AM

    If you're looking to run Llama 2 locally via a CLI or REST API (vs the web ui this article highlights), there's an open-source project some folks and I have been working on over the last few weeks: https://github.com/jmorganca/ollama

    More projects in this space:

    - llama.cpp which is a fast, low level runner (with bindings in several languages)

    - llm by Simon Willison which supports different backends and has a really elegant CLI interface

    - The MLC.ai and Apache TVM projects

    Previous discussion on HN that might be helpful from an article by the great folks at replicate: https://news.ycombinator.com/item?id=36865495

  • by simonw on 8/7/23, 6:12 AM

    If you want to try Llama 2 on a Mac and have Homebrew (or Python/pip) you may find my LLM CLI tool interesting: https://simonwillison.net/2023/Aug/1/llama-2-mac/

  • by jawerty on 8/7/23, 2:06 PM

    If you’re someone who wants to fine-tune Llama 2 on Google Colab, I have a couple live coding streams I did this past week where I fine tune Llama on my own dataset

    Here’s the stream - https://www.youtube.com/live/LitybCiLhSc?feature=share

    One is with LoRa and the other QLoRa I also do a breakdown on each fine-tuning method. I wanted to make these since I myself have had issues running LLMs locally and Colab is the cheapest GPU I can find haha.

  • by SOLAR_FIELDS on 8/7/23, 12:58 PM

    So I tried getting Longchat running (a 32k context llama 2 7b model released a few days ago) with FastChat and I was able to successfully get it running. It seems what I was trying to use it for (Langchain SQL agent) it is not good enough out of the box. Part of this is that I think Langchain is kind of biased towards OpenAi’s models and perhaps Llamaindex would perform better. However Llamaindex uses a newer version of sqlalchemy that a bunch of data warehouse clients don’t support yet.

    Unfortunately with all of the hype it seems that unless you have a REALLY beefy machine the better 70B model feels out of reach for most to run locally leaving the 7B and 13B as the only viable options outside of some quantization trickery. Or am I wrong in that?

    I want to focus more on larger context windows since it seems like RAG has a lot of promise so it seems like the 7B with giant context window is the best path to explore rather than focusing on getting the 70B to work locally

  • by carom on 8/7/23, 7:38 AM

    Just set things up locally last night. If you're a developer, llama.cpp was a pleasure to build and run. I wanted to run the weights from Meta and couldn't figure out text generation web ui. It seemed that one was optimized for grabbing something off HuggingFace.

    Running on a 3090. The 13b chat model quantized to fp8 is giving about 42 tok/s.

  • by growt on 8/7/23, 8:10 AM

    Is there an overview somewhere how much RAM is needed for which model? Is it possible at all to run 4bit 70B on CPU and RAM?

  • by ktaube on 8/7/23, 7:28 AM

    What's the cheapest way to run e.g. LLaMa2-13B and have it served as an API?

    I've tried Inference Endpoints and Replicate, but both would cost more than just using the OpenAI offering.

  • by MediumOwl on 8/7/23, 11:39 AM

    There's only mention of Nvidia GPUs on the web site, what about AMD?

  • by brucethemoose2 on 8/7/23, 6:04 AM

    I am partial to Koboldcpp over text gen UI for a number of reasons.

    ...But I am also a bit out of the loop. For instance, I have not kept up with the CFG/negative prompt or grammar implementations in the UIs.

  • by gorenb on 8/7/23, 7:51 AM

    I've only used the 13b model and I'd say it was as good as GPT-3 (not GPT-4). It's amazing, and I only have a laptop to run it locally on so 13b is as good as I can do.

  • by Manidos on 8/7/23, 11:46 AM

    One way to connect llama-2 (cpp) to a node.js app is by using this helper class (stdin) https://gist.github.com/HackyDev/814c6d1c96f259a13dbf5b2dabf...

  • by bigcloud1299 on 8/8/23, 3:24 AM

    Does anyone have info on how to run 70B on windows ? :) would appreciate it.

  • by KaoruAoiShiho on 8/7/23, 2:31 PM

    What prompts did you use for the article's decorative art?