CLI Commands Reference

Note

Complete reference for HoneyHive CLI commands

This document provides detailed specifications for all available command-line interface commands in the HoneyHive SDK.

The HoneyHive CLI provides powerful command-line tools for managing projects, analyzing traces, running evaluations, and integrating with CI/CD pipelines.

Installation and Setup

Installation:

# Install with CLI support
pip install honeyhive[cli]

# Or install with all OpenInference integrations
pip install honeyhive[all-openinference]

Authentication:

# Set API key via environment variable
export HH_API_KEY="your-api-key"

# Or use CLI login command
honeyhive auth login --api-key your-api-key

# Verify authentication
honeyhive auth whoami

Global Options

All commands support these global options:

--api-key <key>

HoneyHive API key for authentication.

Environment Variable: HH_API_KEY Example: --api-key hh_abc123...

--base-url <url>

Base URL for HoneyHive API.

Default: https://api.honeyhive.ai Environment Variable: HH_BASE_URL Example: --base-url https://api-staging.honeyhive.ai

--output <format>

Output format for results.

Values: json, yaml, table, csv Default: table Example: --output json

--verbose, -v

Enable verbose output.

Example: -v or --verbose

--quiet, -q

Suppress non-essential output.

Example: -q or --quiet

--help, -h: Show help information.

Authentication Commands

honeyhive auth

Manage authentication credentials.

login

honeyhive auth login

Authenticate with HoneyHive.

--api-key <key>

API key for authentication.

Required: Yes Example: honeyhive auth login --api-key hh_abc123...

--save

Save credentials to local config file.

Default: true Example: honeyhive auth login --api-key key --save

Examples:

# Basic login
honeyhive auth login --api-key hh_abc123def456...

# Login without saving
honeyhive auth login --api-key hh_abc123... --no-save

logout

honeyhive auth logout

Remove stored authentication credentials.

--all

Remove all stored credentials.

Default: false

Examples:

# Logout current user
honeyhive auth logout

# Remove all credentials
honeyhive auth logout --all

whoami

honeyhive auth whoami

Show current authenticated user information.

Examples:

# Show current user
honeyhive auth whoami

# Output as JSON
honeyhive auth whoami --output json

Project Commands

honeyhive project

Manage HoneyHive projects.

list

honeyhive project list

List all accessible projects.

--limit <number>

Maximum number of projects to return.

Default: 50 Example: --limit 100

--offset <number>

Number of projects to skip.

Default: 0 Example: --offset 20

Examples:

# List all projects
honeyhive project list

# List with pagination
honeyhive project list --limit 10 --offset 20

# Output as JSON
honeyhive project list --output json

create

honeyhive project create

Create a new project.

--name <name>

Project name.

Required: Yes Example: --name my-new-project

--description <text>

Project description.

Example: --description "My LLM application project"

--settings <json>

Project settings as JSON.

Example: --settings '{"retention_days": 90}'

Examples:

# Create basic project
honeyhive project create --name my-project

# Create with description
honeyhive project create \
  --name my-project \
  --description "Production LLM app"

get

honeyhive project get

Get project details.

<project-name>

Name of the project to retrieve.

Required: Yes Example: honeyhive project get my-project

Examples:

# Get project details
honeyhive project get my-project

# Output as JSON
honeyhive project get my-project --output json

update

honeyhive project update

Update project settings.

<project-name>

Name of the project to update.

Required: Yes

--description <text>: Updated description.

--settings <json>: Updated settings as JSON.

Examples:

# Update description
honeyhive project update my-project \
  --description "Updated description"

# Update settings
honeyhive project update my-project \
  --settings '{"retention_days": 120}'

delete

honeyhive project delete

Delete a project.

<project-name>

Name of the project to delete.

Required: Yes

--confirm

Skip confirmation prompt.

Default: false

Examples:

# Delete with confirmation
honeyhive project delete old-project

# Delete without prompt
honeyhive project delete old-project --confirm

Session Commands

honeyhive session

Manage tracing sessions.

list

honeyhive session list

List sessions in a project.

Project name.: Required: Yes

--limit <number>

Maximum number of sessions to return.

Default: 50

--start-date <date>

Start date filter (ISO format).

Example: --start-date 2024-01-01

--end-date <date>

End date filter (ISO format).

Example: --end-date 2024-01-31

Examples:

# List recent sessions
honeyhive session list # List sessions in date range
honeyhive session list \
  \
  --start-date 2024-01-01 \
  --end-date 2024-01-31

get

honeyhive session get

Get session details.

<session-id>

Session ID to retrieve.

Required: Yes

--include-events

Include events in the session.

Default: false

Examples:

# Get session overview
honeyhive session get session_abc123

# Get session with events
honeyhive session get session_abc123 --include-events

delete

honeyhive session delete

Delete a session.

<session-id>

Session ID to delete.

Required: Yes

--confirm: Skip confirmation prompt.

Examples:

# Delete session
honeyhive session delete session_abc123 --confirm

Event Commands

honeyhive event

Manage and analyze events.

list

honeyhive event list

List events in a session or project.

Project name.

--session-id <id>: Session ID to filter by.

--event-type <type>

Filter by event type.

Values: llm, tool, chain, evaluation, etc.

--limit <number>

Maximum number of events to return.

Default: 100

--start-time <timestamp>: Start time filter (ISO format).

--end-time <timestamp>: End time filter (ISO format).

Examples:

# List recent events
honeyhive event list # List LLM events in session
honeyhive event list \
  --session-id session_abc123 \
  --event-type llm

# List events in time range
honeyhive event list \
  \
  --start-time 2024-01-15T10:00:00Z \
  --end-time 2024-01-15T11:00:00Z

get

honeyhive event get

Get event details.

<event-id>

Event ID to retrieve.

Required: Yes

--include-context

Include parent/child context.

Default: false

Examples:

# Get event details
honeyhive event get evt_abc123

# Get event with context
honeyhive event get evt_abc123 --include-context

search

honeyhive event search

Search events by criteria.

--query <text>

Search query (supports various filters).

Example: --query "model:gpt-4 AND status:error"

Project to search in.

--limit <number>

Maximum results to return.

Default: 50

Examples:

# Search for errors
honeyhive event search \
  \
  --query "status:error"

# Search for specific model
honeyhive event search \
  \
  --query "model:gpt-4 AND event_type:model"

export

honeyhive event export

Export events to file.

Project to export from.: Required: Yes

--output-file <path>

Output file path.

Required: Yes

--format <format>

Export format.

Values: json, jsonl, csv, parquet Default: jsonl

--start-date <date>: Start date for export.

--end-date <date>: End date for export.

--event-types <types>: Comma-separated event types to include.

Examples:

# Export all events
honeyhive event export \
  \
  --output-file events.jsonl

# Export LLM events as CSV
honeyhive event export \
  \
  --output-file llm_events.csv \
  --format csv \
  --event-types llm

# Export date range
honeyhive event export \
  \
  --output-file january_events.jsonl \
  --start-date 2024-01-01 \
  --end-date 2024-01-31

Evaluation Commands

honeyhive eval

Run and manage evaluations.

run

honeyhive eval run

Run evaluations on events.

--evaluators <list>

Comma-separated list of evaluators.

Required: Yes Example: --evaluators factual_accuracy,relevance,quality

--target-events <query>

Query to select target events.

Example: --target-events "event_type:model AND model:gpt-4"

Project containing target events.

--config-file <path>: Path to evaluation configuration file.

--parallel

Run evaluators in parallel.

Default: true

--dry-run: Show what would be evaluated without running.

Examples:

# Run evaluations on recent LLM events
honeyhive eval run \
  \
  --evaluators factual_accuracy,quality \
  --target-events "event_type:model AND start_time:>2024-01-15"

# Dry run to see what would be evaluated
honeyhive eval run \
  \
  --evaluators quality \
  --target-events "session_id:session_abc123" \
  --dry-run

# Run with config file
honeyhive eval run --config-file evaluation_config.yaml

list

honeyhive eval list

List evaluation results.

Project to list evaluations from.

--target-event-id <id>: Filter by target event ID.

--evaluator <name>: Filter by evaluator name.

--start-date <date>: Start date filter.

--end-date <date>: End date filter.

Examples:

# List recent evaluations
honeyhive eval list # List evaluations for specific event
honeyhive eval list \
  \
  --target-event-id evt_abc123

# List quality evaluations
honeyhive eval list \
  \
  --evaluator quality

get

honeyhive eval get

Get evaluation details.

<evaluation-id>

Evaluation ID to retrieve.

Required: Yes

Examples:

# Get evaluation details
honeyhive eval get eval_abc123

compare

honeyhive eval compare

Compare evaluation results.

--evaluations <ids>

Comma-separated evaluation IDs to compare.

Required: Yes

--baseline <id>: Baseline evaluation ID for comparison.

Examples:

# Compare evaluations
honeyhive eval compare \
  --evaluations eval_123,eval_456,eval_789

# Compare against baseline
honeyhive eval compare \
  --evaluations eval_456,eval_789 \
  --baseline eval_123

export

honeyhive eval export

Export evaluation results.

Project to export from.

--output-file <path>: Output file path.

--format <format>

Export format.

Values: json, csv, excel

--evaluator <name>: Filter by evaluator name.

Examples:

# Export all evaluations
honeyhive eval export \
  \
  --output-file evaluations.csv \
  --format csv

# Export specific evaluator results
honeyhive eval export \
  \
  --output-file quality_evals.json \
  --evaluator quality

Trace Analysis Commands

honeyhive trace

Analyze traces and spans.

analyze

honeyhive trace analyze

Analyze trace patterns and performance.

Project to analyze.

--time-window <window>

Time window for analysis.

Values: 1h, 24h, 7d, 30d Default: 24h

--output-file <path>: Save analysis results to file.

--include-metrics: Include detailed metrics in analysis.

Examples:

# Analyze recent traces
honeyhive trace analyze # Analyze last week with metrics
honeyhive trace analyze \
  \
  --time-window 7d \
  --include-metrics \
  --output-file trace_analysis.json

performance

honeyhive trace performance

Analyze trace performance metrics.

Project to analyze.

--groupby <field>

Group results by field.

Values: model, event_type, user_id, session_id

--percentiles <list>

Comma-separated percentiles to calculate.

Default: 50,90,95,99

Examples:

# Performance analysis by model
honeyhive trace performance \
  \
  --groupby model

# Custom percentiles
honeyhive trace performance \
  \
  --percentiles 50,75,90,95,99

errors

honeyhive trace errors

Analyze error patterns in traces.

Project to analyze.

--time-window <window>: Time window for analysis.

--groupby <field>: Group errors by field.

Examples:

# Analyze recent errors
honeyhive trace errors # Group errors by model
honeyhive trace errors \
  \
  --groupby model

Configuration Commands

honeyhive config

Manage CLI configuration.

get

honeyhive config get

Get configuration value.

<key>

Configuration key to retrieve.

Example: honeyhive config get api_key

set

honeyhive config set

Set configuration value.

<key> <value>

Configuration key and value.

Example: honeyhive config set default_project my-project

list

honeyhive config list

List all configuration values.

Examples:

# List all config
honeyhive config list

# List as JSON
honeyhive config list --output json

reset

honeyhive config reset

Reset configuration to defaults.

--confirm: Skip confirmation prompt.

Examples:

# Reset config
honeyhive config reset --confirm

Utility Commands

honeyhive validate

Validate data and configurations.

--config-file <path>: Configuration file to validate.

--data-file <path>: Data file to validate.

--schema <type>

Schema type for validation.

Values: event, evaluation, config

Examples:

# Validate config file
honeyhive validate --config-file config.yaml

# Validate event data
honeyhive validate --data-file events.jsonl --schema event

honeyhive version

Show version information.

Examples:

# Show version
honeyhive version

# Detailed version info
honeyhive version --verbose

honeyhive help

Show help information.

<command>: Show help for specific command.

Examples:

# General help
honeyhive help

# Command-specific help
honeyhive help eval run

Configuration File Format

YAML Configuration:

# honeyhive.yaml
api_key: "hh_your_api_key"
base_url: "https://api.honeyhive.ai"
default_project: "my-project"

output:
  format: "table"
  verbose: false

evaluation:
  parallel: true
  timeout_ms: 30000
  default_evaluators:
    - "quality"
    - "relevance"

trace:
  default_time_window: "24h"
  performance_percentiles: [50, 90, 95, 99]

JSON Configuration:

{
  "api_key": "hh_your_api_key",
  "base_url": "https://api.honeyhive.ai",
  "default_project": "my-project",
  "output": {
    "format": "table",
    "verbose": false
  },
  "evaluation": {
    "parallel": true,
    "timeout_ms": 30000,
    "default_evaluators": ["quality", "relevance"]
  }
}

Environment Variables

The CLI respects these environment variables:

HH_API_KEY: HoneyHive API key for authentication.

HH_BASE_URL

Base URL for HoneyHive API.

Default: https://api.honeyhive.ai

HH_PROJECT: Default project name for operations. Required field that must match your HoneyHive project.

HH_CONFIG_FILE

Path to configuration file.

Default: ~/.honeyhive/config.yaml

HH_OUTPUT_FORMAT

Default output format.

Values: json, yaml, table, csv Default: table

Exit Codes

The CLI uses these exit codes:

0: Success
1: General error
2: Invalid command usage
3: Authentication error
4: Network/API error
5: Data validation error
6: Permission error

Examples and Use Cases

Daily Monitoring:

#!/bin/bash
# Daily monitoring script

PROJECT="production-llm-app"
DATE=$(date -d "yesterday" +%Y-%m-%d)

# Check for errors
honeyhive trace errors \
  \
  --time-window 24h \
  --output json > daily_errors.json

# Performance analysis
honeyhive trace performance \
  \
  --time-window 24h \
  --groupby model > daily_performance.txt

# Run evaluations on recent events
honeyhive eval run \
  \
  --evaluators quality,factual_accuracy \
  --target-events "start_time:>$DATE"

CI/CD Integration:

#!/bin/bash
# CI/CD evaluation script

# Export test session events
honeyhive event export \
  \
  --session-id $TEST_SESSION_ID \
  --output-file test_events.jsonl

# Run evaluations
honeyhive eval run \
  --evaluators quality,accuracy \
  --target-events "session_id:$TEST_SESSION_ID" \
  --output json > evaluation_results.json

# Check if evaluations pass threshold
python check_evaluation_thresholds.py evaluation_results.json

Data Export for Analysis:

#!/bin/bash
# Export data for ML analysis

PROJECT="ml-training-data"
START_DATE="2024-01-01"
END_DATE="2024-01-31"

# Export events
honeyhive event export \
  \
  --start-date $START_DATE \
  --end-date $END_DATE \
  --format parquet \
  --output-file events_jan2024.parquet

# Export evaluations
honeyhive eval export \
  \
  --format csv \
  --output-file evaluations_jan2024.csv

CLI Commands Reference

Installation and Setup

Global Options

Authentication Commands

Project Commands

Session Commands

Event Commands

Evaluation Commands

Trace Analysis Commands

Configuration Commands

Utility Commands

Configuration File Format

Environment Variables

Exit Codes

Examples and Use Cases

See Also