Skip to content

integration-testing

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

integration-testing

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
configs
configs
 
 
docs
docs
 
 
jbang-lib
jbang-lib
 
 
logs
logs
 
 
scripts
scripts
 
 
RunIntegrationTest.java
RunIntegrationTest.java
 
 

README.md

ACP Java Tutorial - Integration Testing

Automated integration testing using JBang + AI validation.

Prerequisites

  1. JBang - Install from https://www.jbang.dev/

    curl -Ls https://sh.jbang.dev | bash -s - app setup

  2. Claude Code SDK - Must be installed in local Maven repo

    cd ~/path/to/claude-code-sdk-java
./mvnw install -DskipTests

  3. Claude CLI - Required for AI validation

    npm install -g @anthropic-ai/claude-code

  4. Grok CLI (optional) - Required for modules 01-08 and 21, which launch it as grok agent stdio

    curl -fsSL https://x.ai/cli/install.sh | bash
grok login

Quick Start

cd integration-testing

# Run a single test
jbang RunIntegrationTest.java module-12-echo-agent

# Run all local tests (no API key needed)
./scripts/run-integration-tests.sh --local

# Run all tests (the client modules need the grok CLI)
./scripts/run-integration-tests.sh

# List available modules
jbang RunIntegrationTest.java --list

Module Categories

Local Agent Modules (No API Key Required)

Module Description
module-12-echo-agent Minimal echo agent
module-14-sending-updates Agent sends all update types
module-15-agent-requests Agent requests files/permissions
module-16-in-memory-testing In-memory transport testing

Grok Client Modules (Require the grok CLI)

Module Description
module-01-first-contact Basic connection and prompt
module-02-protocol-basics Initialize handshake
module-03-sessions Session lifecycle
module-04-prompts Prompt/response and stop reasons
module-05-streaming-updates Real-time updates
module-06-update-types All SessionUpdate types
module-07-agent-requests Client file handlers
module-08-permissions Permission handling
module-21-async-client Async (reactive) client

How It Works

  1. Load Config - Read JSON config from configs/<module>.json
  2. Verify Environment - Check required environment variables
  3. Build Module - Run mvn compile or mvn package
  4. Run Module - Execute via mvn exec:java
  5. AI Validation - Use Claude to validate output against expected behavior

Configuration Schema

{
  "moduleId": "module-XX-name",
  "displayName": "Module XX: Title",
  "timeoutSec": 120,
  "requiredEnv": [],
  "requiredCommands": ["grok"],
  "requiresPackage": false,
  "stdin": "1\n",
  "expectedBehavior": "Description of what the module should demonstrate..."
}
  • moduleId - Maven module name
  • displayName - Human-readable name
  • timeoutSec - Max execution time
  • requiredEnv - Required environment variables
  • requiredCommands - Executables that must be on the PATH (optional)
  • requiresPackage - If true, runs mvn package instead of mvn compile
  • stdin - Text fed to the module's standard input, for interactive modules (optional)
  • expectedBehavior - Description for AI validation

AI Validation

Uses Claude (via claude-code-sdk) to semantically validate test output:

  • No brittle regex patterns
  • Understands expected behavior descriptions
  • Returns structured result with confidence score
  • Cost: ~$0.001 per validation using Claude Haiku

Logs

Test output is saved to logs/<module>-<timestamp>.log

Troubleshooting

"Config not found"

Make sure you're in the integration-testing directory.

"Required command not found on PATH: grok"

Install the Grok CLI and run grok login, or use the --local flag to skip the Grok client tests.

Module 08 shows no permission request

Grok reads Claude Code's settings. If ~/.claude/settings.json sets "defaultMode" to "auto" or "acceptEdits", Grok writes hello.txt without asking, and the test fails on the missing PERMISSION REQUEST from agent line.

"Build failed"

Run ./mvnw compile -pl module-XX-name from repo root to see build errors.

"AI validation failed"

Check that Claude CLI is installed and working: claude --version