var RESOURCE_GROUP = "yourResourceGroup";

var modelFilterList = AiParameterArgumentBinding.create()

.key("modelFilterList").value("null");

var modelFilterListType = AiParameterArgumentBinding.create()

.key("modelFilterListType").value("allow");

var configurationData = AiConfigurationBaseData.create()

var configuration = new ConfigurationApi().create(RESOURCE_GROUP, configurationData);

var deploymentCreationRequest =

var deployment = new DeploymentApi().create(RESOURCE_GROUP, deploymentCreationRequest);

[OPTION BEGIN [SAP Cloud SDK for Java]]

In this step, we will consume an LLM through the orchestration service with the created deployment, using the core and orchestration module of the SAP Cloud SDK for Java.

To begin the consumption process for the orchestration you’ve deployed, follow the steps below:

**Prepare the CV File**

• Download the [cv.txt](img/cv.txt) file, which contains the CV used this tutorial. Add it to your project.

• Read the CV file from the correct path using the following code:

var filePath = "path/to/cv.txt";

String cvContent;

try {

cvContent = new String(Files.readAllBytes(Paths.get(filePath)));

} catch (IOException e) {

throw new RuntimeException(e);

}

System.out.println(cvContent);

The next step involves creating the prompt for the LLM including both `SystemMessage` and `UserMessage` components.

• `SystemMessage`: Defines the AI assistant's role and instructions.

• `UserMessage`: Represents the user's input (i.e., the CV content) to be processed by the LLM.

var systemMessage = new SystemMessage(

"""

);

var userMessage = new UserMessage("Candidate Resume: \n" + cvContent);

var prompt = new OrchestrationPrompt(systemMessage, userMessage);

We can define multiple models for the use case. Only use those models that are already deployed in your instances. For this example, we have selected the following three models:

var models = List.of("gpt-4o", "mistralai--mistral-large-instruct", "anthropic--claude-3.5-sonnet");

With the following function we create an `OrchestrationModuleConfig` containing information about the `LLMModule`. This can be extended to contain information regarding templating, masking, filtering and grounding, if desired to use these functionality of orchestration.

OrchestrationModuleConfig createModuleConfig(String modelName) {

var config = LLMModuleConfig.create()

.modelName(modelName)

.modelParams(Map.of( // add model parameters as needed

"max_tokens", 1000,

"temperature", 0.6

));

return new OrchestrationModuleConfig().withLlmConfig(config);

}

The following function writes the responses from different models, stored in a list, to a file:

void createFileFromResponses (ArrayList<Map> responses) {

// Format model responses

var formattedResponses = responses.stream().

map(response -> "Response from model " + response.get("model") +

": \n\n" + response.get("response") + "\n" + "-".repeat(120));

// Write model responses to provided file path

try {

Files.writeString(Path.of("src/main/resources/static/responses.txt"),

String.join("\n", formattedResponses.toList()));

} catch (IOException e) {

throw new RuntimeException(e);

}

}

**Generate Responses for Multiple Models**

This step outlines the process of generating responses for a set of queries using different models. We iterate through the list of models created earlier and query the model with the created prompt using an `OrchestrationClient`.

var client = new OrchestrationClient(new AiCoreService()

.getInferenceDestination(RESOURCE_GROUP).forScenario("orchestration"));

var responses = new ArrayList<Map>();

for (var model: models) {

System.out.println("\n=== Responses for model: %s ===\n".formatted(model));

// Create orchestration module configuration for current model

var moduleConfig = createModuleConfig(model);

// Prompt model with orchestration module configuration

var response = client.chatCompletion(prompt, moduleConfig);

// Add response to list of all model responses

responses.add(Map.of("model", model, "response", response.getContent()));

System.out.println(response.getContent());

}

createFileFromResponses(responses);

• If not done automaticaly by your IDE, add the following imports:

import com.sap.ai.sdk.core.AiCoreService;

import com.sap.ai.sdk.orchestration.OrchestrationClient;

import com.sap.ai.sdk.orchestration.OrchestrationModuleConfig;

import com.sap.ai.sdk.orchestration.OrchestrationPrompt;

import com.sap.ai.sdk.orchestration.SystemMessage;

import com.sap.ai.sdk.orchestration.UserMessage;

import com.sap.ai.sdk.orchestration.model.LLMModuleConfig;

**Important Note**

Ensure at least one orchestration deployment is ready to be consumed during this process.

**Optional Advanced Modules**

Together with document grounding and templating, data masking and content filtering are available to enhance data privacy and safety. Data masking hides sensitive information like phone numbers or organization names, while content filtering can screen for categories such as hate self-harm, sexual content, and violence. In this tutorial, the response generated by the LLM models may carry sensitive information, such as names and phone numbers. For further enhancement, refer to the next tutorial on implementing these modules.

[OPTION END]