On: Ask HN: Local LLM's

[brucethemoose2]

Koboldcpp (a nice frontend for llama.cpp) is The Way.

You really want to run OSX though, as its not very fast without Metal (or Vulkan). Also, you need a relatively high memory M1 model to run the better llama variants.

[Ms-J → brucethemoose2]

I'll take a look into Koboldcpp, a frontend is always nice, thanks! I do have the max specs on this M1.