11![ Java 11+] ( https://img.shields.io/badge/Java-11%2B-informational )
2- ![ llama.cpp b1382 ] ( https://img.shields.io/badge/llama.cpp-%23b1382 -informational )
2+ ![ llama.cpp b1645 ] ( https://img.shields.io/badge/llama.cpp-%23b1645 -informational )
33
44# Java Bindings for [ llama.cpp] ( https://github.com/ggerganov/llama.cpp )
55
@@ -20,48 +20,18 @@ Access this library via Maven:
2020</dependency >
2121```
2222
23- Here is a short example :
23+ There are multiple [ examples ] ( src/test/java/examples ) . Make sure to set ` model.home ` and ` model.name ` to run them :
2424
25- ``` java
26- public class Example {
25+ ``` bash
26+ mvn exec:java -Dexec.mainClass=" examples.MainExample" -Dmodel.home=" /path/to/models" -Dmodel.name=" codellama-13b.Q5_K_M.gguf"
27+ ```
2728
28- public static void main (String ... args ) throws IOException {
29- LlamaModel . setLogger((level, message) - > System . out. print(message));
30- ModelParameters modelParams = new ModelParameters ()
31- .setNGpuLayers(43 );
32- InferenceParameters inferParams = new InferenceParameters ()
33- .setTemperature(0.7f )
34- .setPenalizeNl(true )
35- .setMirostat(InferenceParameters . MiroStat . V2 )
36- .setAntiPrompt(" \n " );
29+ You can also run some integration tests, which will automatically download a model to ` model.home ` :
3730
38- String modelPath = " /run/media/konstantin/Seagate/models/llama2/llama-2-13b-chat/ggml-model-q4_0.gguf" ;
39- String system = " This is a conversation between User and Llama, a friendly chatbot.\n " +
40- " Llama is helpful, kind, honest, good at writing, and never fails to answer any " +
41- " requests immediately and with precision.\n " ;
42- BufferedReader reader = new BufferedReader (new InputStreamReader (System . in, StandardCharsets . UTF_8 ));
43- try (LlamaModel model = new LlamaModel (modelPath, modelParams)) {
44- System . out. print(system);
45- String prompt = system;
46- while (true ) {
47- prompt += " \n User: " ;
48- System . out. print(" \n User: " );
49- String input = reader. readLine();
50- prompt += input;
51- System . out. print(" Llama: " );
52- prompt += " \n Llama: " ;
53- for (String output : model. generate(prompt, inferParams)) {
54- System . out. print(output);
55- prompt += output;
56- }
57- }
58- }
59- }
60- }
31+ ``` bash
32+ mvn verify -Dmodel.home=/path/to/models
6133```
6234
63- Also have a look at the [ examples] ( src/test/java/examples ) .
64-
6535### No Setup required
6636
6737We support CPU inference for the following platforms out of the box:
@@ -150,6 +120,50 @@ Look for the shared library in `build`.
150120
151121## Documentation
152122
123+ ### Example
124+
125+ This is a short example on how to use this library:
126+
127+ ``` java
128+ public class Example {
129+
130+ public static void main (String ... args ) throws IOException {
131+ LlamaModel . setLogger((level, message) - > System . out. print(message));
132+ ModelParameters modelParams = new ModelParameters ()
133+ .setNGpuLayers(43 );
134+ InferenceParameters inferParams = new InferenceParameters ()
135+ .setTemperature(0.7f )
136+ .setPenalizeNl(true )
137+ .setMirostat(InferenceParameters . MiroStat . V2 )
138+ .setAntiPrompt(" \n " );
139+
140+ String modelPath = " /run/media/konstantin/Seagate/models/llama2/llama-2-13b-chat/ggml-model-q4_0.gguf" ;
141+ String system = " This is a conversation between User and Llama, a friendly chatbot.\n " +
142+ " Llama is helpful, kind, honest, good at writing, and never fails to answer any " +
143+ " requests immediately and with precision.\n " ;
144+ BufferedReader reader = new BufferedReader (new InputStreamReader (System . in, StandardCharsets . UTF_8 ));
145+ try (LlamaModel model = new LlamaModel (modelPath, modelParams)) {
146+ System . out. print(system);
147+ String prompt = system;
148+ while (true ) {
149+ prompt += " \n User: " ;
150+ System . out. print(" \n User: " );
151+ String input = reader. readLine();
152+ prompt += input;
153+ System . out. print(" Llama: " );
154+ prompt += " \n Llama: " ;
155+ for (String output : model. generate(prompt, inferParams)) {
156+ System . out. print(output);
157+ prompt += output;
158+ }
159+ }
160+ }
161+ }
162+ }
163+ ```
164+
165+ Also have a look at the other [ examples] ( src/test/java/examples ) .
166+
153167### Inference
154168
155169There are multiple inference tasks. In general, ` LlamaModel ` is stateless, i.e., you have to append the output of the
0 commit comments