Skip to main content

​
Overview

Use the rippletide_client package to evaluate your Rippletide agents from Python: create an agent, generate test questions, and score responses programmatically.

​
Installation

First, clone the starter repository which contains the rippletide_client directory: 
git clone https://github.com/rippletideco/starter.git
cd starter
Then install the required dependencies: 
cd rippletide_client
pip install -r requirements.txt

​
Initialize the client

Create a Python script in the starter directory (parent of rippletide_client), then import and initialize the client: 
import os
import sys
from pathlib import Path

# Add the starter directory to Python path so we can import rippletide_client
# This assumes your script is in the starter directory
try:
    starter_dir = Path(__file__).parent
except NameError:
    # If __file__ is not available (e.g., in REPL), use current working directory
    starter_dir = Path.cwd()

sys.path.insert(0, str(starter_dir))

from rippletide_client import RippletideClient

# Initialize with your API key
api_key = os.getenv("RIPPLETIDE_API_KEY", "your-api-key")
if api_key == "your-api-key":
    raise ValueError("Please set RIPPLETIDE_API_KEY environment variable or provide a valid API key")

# You can override the backend URL if needed (staging/local)
base_url = os.getenv("RIPPLETIDE_BASE_URL")  # optional
client = RippletideClient(api_key=api_key, base_url=base_url)
For production, store the API key in an environment variable (for example, RIPPLETIDE_API_KEY) and read it instead of hard-coding.

​
Create an agent for evaluation

Create an agent specifically configured for evaluation. The agent needs a public URL where it can receive chat requests: 
# Create an agent with configuration
agent = client.create_agent(
    name="My Eval Agent",
    public_url="https://your-agent-url.com/chat",  # Your agent's public chat endpoint
    num_nodes=100,  # Optional: number of nodes (default: 100)
    seed=None  # Optional: random seed (auto-generated if not provided)
)

agent_id = agent["id"]
print(f"Created evaluation agent: {agent_id}")
The public_url should point to your agent’s chat endpoint where it can receive messages. This is required for the evaluation system to interact with your agent.

​
Extract questions from a PDF

result = client.extract_questions_from_pdf(
    agent_id=agent_id,
    pdf_path="path/to/document.pdf",
)
print(f"Extracted {len(result.get('qaPairs', []))} Q&A pairs")

​
Fetch generated test prompts

test_prompts = client.get_test_prompts(agent_id)
for prompt in test_prompts:
    print(f"Question: {prompt['prompt']}")
    print(f"Expected Answer: {prompt.get('expectedAnswer', 'N/A')}")

​
Evaluate an agent response

report = client.evaluate(
    agent_id=agent_id,
    question="What is this document about?",
    expected_answer="Optional expected answer",
)

print(f"Label: {report['label']}")
print(f"Justification: {report['justification']}")
for fact in report["facts"]:
    print(f"- {fact['fact']}: {fact['label']}")

​
Full workflow example

Here’s a complete example that creates an agent, uploads knowledge, and evaluates it: 
import os
import sys
from pathlib import Path

# Add the starter directory to Python path
starter_dir = Path(__file__).parent
sys.path.insert(0, str(starter_dir))

from rippletide_client import RippletideClient

# 1. Initialize the client
api_key = os.getenv("RIPPLETIDE_API_KEY")
client = RippletideClient(api_key=api_key)

# 2. Create an agent configured for evaluation
agent = client.create_agent(
    name="Evaluation Agent",
    public_url="https://your-agent-url.com/chat"  # Your agent's chat endpoint
)
agent_id = agent["id"]
print(f"Created agent: {agent_id}")

# 3. Upload a PDF to extract questions and expected answers
client.extract_questions_from_pdf(
    agent_id=agent_id,
    pdf_path="knowledge.pdf"
)

# 4. Get all generated test prompts
test_prompts = client.get_test_prompts(agent_id)
print(f"Found {len(test_prompts)} test prompts")

# 5. Evaluate each prompt
for prompt in test_prompts:
    report = client.evaluate(
        agent_id=agent_id,
        question=prompt["prompt"],
        expected_answer=prompt.get("expectedAnswer"),
    )
    print(f"Question: {prompt['prompt'][:50]}...")
    print(f"Result: {report['label']}")
    print(f"Justification: {report['justification']}\n")