I'm so confused about running these models locally only. When reading about the llm tool, I thought, ok, this helps organize all the pieces on my machine. But then it uses a replicate API key, so clearly it requires a network connection. Is this just to download the models? I feel like we need a new license or packaging model that clearly states whether the computation is happening locally or remotely. It's very important to me and often hard to know until I'm a long way into the installation process.
For those getting started, the easiest one click installer I've used is Nomic.ai's gpt4all: https://gpt4all.io/
This runs with a simple GUI on Windows/Mac/Linux, leverages a fork of llama.cpp on the backend and supports GPU acceleration, and LLaMA, Falcon, MPT, and GPT-J models. It also has API/CLI bindings.
I just saw a slick new tool https://ollama.ai/ that will let you install a llama2-7b with a single `ollama run llama2` command that has a very simple 1-click installer for Apple Silicon Mac (but need to build from source for anything else atm). It looks like it only supports llamas OOTB but it also seems to use llama.cpp (via Go adapter) on the backend - it seemed to be CPU-only on my MBA, but I didn't poke too much and it's brand new, so we'll see.
I've also been porting my own notes into a single location that tracks models, evals, and has guides focused on local models: https://llm-tracker.info/
The gold standard of local-only model inference for LLaMA, alpaca, and friends is LLaMA-cpp, https://github.com/ggerganov/llama.cpp No dependencies, no GPU needed, just point it to a model snapshot that you download separately on bittorrent. Simple CLI tools that are usable (somewhat) from shell scripts.
Which includes the llama.cpp backend, and a lot more.
Unfortunately, despite claiming to be the "Automatic1111" of text generation, it doesn't support any of the prompt engineering capabilities (i.e. negative prompts, prompt weights, prompt blending, etc) available in Automatic1111, despite the fact that it's not difficult to implement - https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
Luckily for Ooga Booga, no one else supports it either. Why this is? I have no explanation except that the NLP community doesn't know jack about prompt engineering, which is Kafkaesque
The 7B and 13B models use the same architecture, so llama.cpp already supports it. (Source: running 13b-chat myself using llama.cpp GPU offload.) It's only 70B that has additional extensions that llama.cpp would need to implement.
Llama2 7B and 13B GGML are up and work with existing llama.cpp no changes needed! The 70B does require a llama.cpp change, but I'm sure it won't take long.
Do you have any thoughts on how I can make this more obvious?
It's covered by the documentation for the individual plugins, but I want to make it as easy as possible for people to understand what's going on when they first start using the tool.
It's laziness on my part. I read just your blog post. Had I clicked through to the tool, it clearly says it at the top. My apologies.
I'm very grateful for all the work and writing you are doing about LLMs.
Regarding your note about JSON mode with llama.cpp, I'm writing a wrapper for it on my katarismo project. It is basically the stdout suggestion from that comment, but it is working really well for me when I use it with pocketbase.
Perhaps something as simple as stating it was first built around OpenAI models and later expanded to local via plugins?
I've been meaning to ask you, have you seen/used MS Guidance[0] 'language' at all? I don't know if it's the right abstraction to interface as a plugin with what you've got in llm cli but there's a lot about Guidance that seems incredibly useful to local inference [token healing and acceleration especially].
Yeah, I looked at Guidance and I have to admit I don't fully get it - my main problem was that I can't look at one of their Handlebars templates and figure out exactly what LLM prompts it's going to fire and in what order they will be sent.
I'm much happier with a very thin wrapper where I can explicitly see exactly what prompts are processed when, and where prompts are assembled using very simple string manipulation.
I'm thinking I may pull the OpenAI stuff out of LLM core and make that a plugin as well - that way it will be VERY obvious when you install the tool that you get to pick which LLMs you're going to work with.
Nothing in particular. Setting up a scientific python+pytorch stack is difficult if you're unfamiliar with the python packaging ecosystem.
If you're not on a "happy path" of "ubuntu, nvidia, and anaconda," then lots of things could go wrong if you're configuring from scratch. If you want to run these models efficiently, hardware acceleration is a must, but managing the intersection between {GPU, operating system, architecture, Python version, Python virtualenv location} is tricky.
That's even before you deal with hardware-specific implementation details!
- Running NVidia? Double-check your CUDA library version, kernel module version, (optional) CuDNN, and pytorch version
- Running AMD? Double-check your ROCm library version, AMD drivers, and make sure that you use Pytorch provided by AMD with ROCm support
- On Apple machines? Double-check that your M1 hardware actually has proper hardware support, then download and install a custom Pytorch distribution linked with M1 support, and make sure that the numpy library version has been properly linked with Accelerate.framework or else your BLAS calls run on the CPU rather than the undocumented AMX coprocessor. If you want to run on the ANE, you'll additionally need a working xcode toolchain and a version of the CoreML model compiler that can read your serialized pytorch model files properly.
I think the pain of getting things working makes it easier to just throw up one's hands and pay someone else to run your model for you.
I mean, caveat emptor; CoreML has been barebones for years, and Apple isn't exactly notorious for huge commitment to third party APIs. The writing was on the wall with how Metal was rolled out and how fast OpenCL got dropped, it honestly doesn't surprise me at all at this point. Even the current Apple Silicon support in llama.cpp is fudged with NEON and Metal Shaders instead of Apple's "Neural Engine".
Then you more or less need a GUI like OpenAI built with ChatGPT so you control the whole environment. Even setting up LLM in Homebrew required me to do the whole install twice because of some arcane error.
I think I can fix that by shipping a bottle release - "brew install simonw/llm/llm" currently attempts to compile Pydantic's Rust extensions, which means it runs incredibly slowly.
In general the default LLM models are too big to run on a consumer GPU without quantization.
That means additional steps or alternative downloads are needed.
Also generative graphics models are about 6 months further advanced than generative text models because Stable Diffusion came out before Llama (Llama being the first "useful" smaller-large-language-model)
We know that GPT-4 is the king at this time, so only researchers, developers, and enthusiasts setup local LLMs. They don't need easier way to setup. For image generation, Local Stable Diffusion is the king for certain big genre of image generation. Many people including PC newbie try it, so there are many guides.
The best place to start is the localllama subreddit.
The install guide describes text generation webui, a generic frontend for multiple ways of running models locally, and llama.cpp, a way of running llama-derived models locally on CPU from the command-line.
The bundled models that the oobabooga offers during install are probably not what you want. To find good models the localllama wiki has a handy models page. GPTQ models are only running on GPU, GGML models are for running on CPU or GPU through llama.cpp (directly or via textgen webui).
Note that just in case you have the hardware necessary to run the biggest available model llama2-70b (for example two RTX 3090 with a total of 48GB of VRAM), there is currently a small bug (with a fix) documented at https://github.com/oobabooga/text-generation-webui/issues/32...
Idk, to me this reads pretty clearly remote (for this model in particularly):
> My LLM tool provides command-line access to a wide variety of language models, both via web APIs and self-hosted on your own machine ... The brand new llm-replicate plugin provides CLI access to models hosted on Replicate, and this morning a16z-infra released a16z-infra/llama13b-v2-chat which provides Replicate API access to the new Llama 2 13B chat model.
I'm also unsure what this would have to do with the license, when you install any other software do you expect the license to tell you whether processing happens locally or in the cloud?
My hunch is that OpenAI is a lot cheaper. I've spent $0.26 on 115 seconds of compute with Llama 2 on Replicate so far, which is only a dozen test prompts.
I am not familiar with Replicate, but based on their website, they charge per GPU type. I didn't see the GPU type set in the example. Is it baked in as part of the "a16z-infra/llama13b-v2-chat" model?
I feel like Simon's been on a tear with these LLM postings. Simon, I really enjoying you swashbuckling through this, and then documenting your travels.
This would be a solid point if he kept putting out nonsense or useless but catchy things, but he keeps putting out pretty concise useful things. Useful tech things are pretty popular here, for good reason.
