langchain4j+local-ai小试牛刀

序

本文主要研究一下如何本地运行local-ai并通过langchain4j集成调用。

步骤

curl安装

curl https://localai.io/install.sh | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21509    0 21509    0     0  45959      0 --:--:-- --:--:-- --:--:-- 45861
  Downloading local-ai...
######################################################################## 100.0%
  Installing local-ai to /usr/local/bin...
  Creating local-ai user...
  Adding current user to local-ai group...
  Creating local-ai systemd service...
  The LocalAI API is now available at 127.0.0.1:8080.
  Install complete. Run "local-ai" from the command line.
[WARN]  No NVIDIA/AMD GPU detected. LocalAI will run in CPU-only mode.

运行

local-ai
9:49AM INF env file found, loading environment variables from file envFile=/etc/localai.env
9:49AM INF Setting logging to info
9:49AM INF Starting LocalAI using 16 threads, with models path: /usr/share/local-ai/models
9:49AM INF LocalAI version: v2.26.0 (09941c0bfb9119bb01a04b2a0a16897ecf2cd087)
9:49AM INF Preloading models from /usr/share/local-ai/models
9:49AM INF core/startup process completed!
9:49AM INF LocalAI API is listening! Please connect to the endpoint for API documentation. endpoint=http://0.0.0.0:8080

访问http://localhost:8080

安装model

访问http://localhost:8080/browse 浏览模型并选择安装
或者使用命令行，这里选择一个小模型

local-ai models install llama-3.2-1b-instruct:q4_k_m

https://hf-mirror.com/hugging-quants/Llama-3.2-1B-Instruct-Q4...

pom.xml

        
            dev.langchain4j
            langchain4j-local-ai
            1.0.0-beta1
        

controller

@RequestMapping("/local-ai")
@RestController
@Slf4j
public class LocalAiController {

    ChatLanguageModel model = LocalAiChatModel.builder()
            .baseUrl("http://localhost:8080/v1")
            .modelName("llama-3.2-1b-instruct:q4_k_m")
//            .maxTokens(3)
            .logRequests(true)
            .logResponses(true)
            .build();

    /**
     * curl -i http://localhost:8080/local-ai/chat?prompt=who are you
     * @param prompt
     * @return
     */
    @GetMapping("/chat")
    public String chat(@RequestParam("prompt") String prompt) {
        return model.chat(prompt);
    }

}

请求

http://localhost:8080/local-ai/chat?prompt=who%20are%20you
返回

I'm an AI assistant, which means I'm a computer program designed to understand and respond to human language. I'm here to help with a wide range of topics, answer questions, provide information, and even chat with you in a friendly and helpful way. I'm a large language model, which means I've been trained on a massive database of text from the internet, books, and other sources, which allows me to generate responses that are often surprisingly accurate and insightful. Some common ways I can assist you include: * Answering questions on a particular topic * Providing definitions and explanations * Offering suggestions and ideas * Summarizing long pieces of text * Translating languages (I can translate text from one language to another) * Offering writing and proofreading services * Generating text on a given topic or topic I'm always learning and improving, so please bear with me if I make any mistakes or don't quite understand the question. I'm here to help, so feel free to ask me anything that's on your mind!

小结

LocalAI是OpenAI 的免费开源替代产品，提供了与 OpenAI 兼容的直接替代 REST API，可以在本地直接运行大语言模型LLM、生成图像、音频等。它提供了docker安装及脚本安装。langchain4j提供了langchain4j-local-ai用于集成local-ai.

doc

• langchain4j local-ai
• localai.io