TestIntel — Frequently Asked Questions

General

What is TestIntel?

TestIntel is an AI-powered QA assistant that helps software teams plan, generate, manage, and track tests. It integrates with Jira, Xray, Zephyr Scale, and GitHub to provide a complete test management workflow.

Who is TestIntel for?

QA engineers, test automation engineers, software developers, and QA managers. Anyone involved in software testing can benefit from AI-assisted test generation and management.

Is TestIntel a hosted service or self-hosted?

Self-hosted. You deploy TestIntel on your own infrastructure (server, VM, Docker). Your data never leaves your network. OctoBlue Technologies does not host customer instances or store customer data.

What AI providers are supported?

• Anthropic Claude (default, recommended)
• OpenAI (GPT-4o)
• AWS Bedrock (Claude via your own AWS account)

You bring your own API key (BYOK) and pay the AI provider directly.

Setup

What environment variables do I need?

At minimum: an AI provider key (e.g. ANTHROPIC_API_KEY). If no AI key is configured, TestIntel shows a warning banner after login directing admins to Settings → API Keys & Tokens. See the README for the full list of optional integrations.

How do I connect to Jira/Xray/Zephyr?

Set the relevant environment variables, add the configuration to config.yaml, then run testintel discover to auto-detect your instance's schema.

Can multiple team members use the same instance?

Yes. Deploy one instance on a shared server. Team members access it via the Web UI in their browser. Use TESTINTEL_ADMIN_KEY for admins and TESTINTEL_MEMBER_KEY for team members.

Features

Why aren't some tests being added to staging?

If you previously rejected tests from staging, TestIntel remembers their signatures (name + steps) and automatically skips them on future "Add to Staging" actions. This prevents the same tests from reappearing after rejection.

The "Add to Staging" button will show how many were skipped (e.g. "3 tests staged — 2 previously rejected skipped").

To allow a previously rejected test to be re-staged, either:

• Ask the AI to regenerate it with a different name or steps
• Delete the staging_rejected.json file from your project's storage directory to clear the rejection history

See the Staging Workflow page for full details.

What can TestIntel generate?

• Test strategies
• Test cases (table format or Gherkin .feature files)
• Test automation code (Python, JavaScript, TypeScript, Java, C#, Ruby)
• Impact analysis from code changes / PRs
• CI/CD configuration (GitHub Actions, Jenkins)
• Performance test scripts (k6, JMeter)
• BDD scenarios (Gherkin)

How does coverage gap analysis work?

TestIntel scans your codebase for API endpoints, public functions, and UI components, then cross-references against your test inventory to identify what's tested and what's not. If your repo has an OpenAPI/Swagger spec, it uses that for more precise endpoint detection.

How do I analyze a PR or commit?

Two ways:

1. Paste the URL in chat — Just include the GitHub PR or commit URL in your message:

• "Analyze https://github.com/myorg/myrepo/pull/50"
• "What tests are affected by https://github.com/myorg/myrepo/commit/abc1234"

TestIntel auto-detects the URL, fetches the diff, and includes it in the AI context.

1. Use the PR/Ref field — Fill in the repo and PR # in the Chat settings fields. This is useful when you want all messages in a session to reference the same PR (e.g., first get an impact analysis, then generate tests for it).

Both approaches require a GitHub token configured in Settings → API Keys & Tokens (needed for private repos).

What is the Test Maintenance Score?

A 0-100 health score computed from six weighted components: pass rate, automation rate, freshness (tests run recently), review backlog, flakiness, and coverage. It gives you a single number to track test suite health over time.

How does flaky test detection work?

TestIntel tracks each test's Pass/Fail status across multiple runs. If a test flips between Pass and Fail frequently, it's flagged as flaky with a score and a suggested root cause (timing issue, shared state, environment dependency, etc.).

Can TestIntel read my OpenAPI/Swagger spec?

Yes. If your repo contains openapi.json, openapi.yaml, swagger.json, or swagger.yaml, TestIntel auto-detects it and uses the structured API definition for more precise test generation and coverage analysis.

What are the AI intelligence features?

TestIntel includes several features that make AI output more trustworthy and aligned with your team:

Assumption Transparency — Before generating test cases or strategies, the AI lists the key assumptions it's making (e.g., "Authentication uses JWT tokens", "API follows REST conventions"). You can correct these before proceeding.

Clarification Questions — For large outputs, the AI asks 2-3 targeted questions if it identifies gaps that would significantly change the output (e.g., "Should failed login attempts lock the account?"). It only asks when the answer isn't already in your code context or Knowledge Base.

Testing Profile — Upload a testing profile document to your Knowledge Base to define your team's philosophy (priorities, format preferences, risk tolerance). The AI aligns all output with your stated preferences. See docs/templates/testing-profile.md for a template.

Inventory-Aware Reasoning — When "Include inventory context" is checked, the AI actively reasons about your existing tests: avoids duplicates, identifies gaps, prioritizes underserved areas, and flags potentially obsolete tests.

Confidence Notes — When the AI's confidence in its output is Medium or Low (due to missing context), it adds a brief note at the end explaining why and what additional context would improve the output. High-confidence outputs have no note.

What does a confidence note mean?

A confidence note appears at the end of AI responses when context is incomplete:

• Medium — Some context is available but gaps exist (e.g., no API spec, no acceptance criteria). The output is useful but may need refinement for edge cases.
• Low — Minimal context available. The output is based on common patterns and may not match your specific system. Provide more context (code, KB docs, requirements) for better results.
• No note — High confidence. The AI had sufficient context (code + KB + specific request) to produce targeted output.

Security

Does TestIntel send my code to the cloud?

Your code is sent to the AI provider (Anthropic, OpenAI, or Bedrock) for processing. Anthropic and OpenAI do not train on API data. For maximum privacy, use AWS Bedrock — your code stays in your own AWS account.

Does OctoBlue see my data?

No. TestIntel is self-hosted. The only communication with OctoBlue is license key validation, which sends only the license key — no code, tests, or project data.

Can TestIntel push code to my GitHub repo?

No. The GitHub integration is strictly read-only. TestIntel only reads file trees, file contents, and PR diffs. It never creates commits, branches, or pull requests.

How are API keys stored?

Keys can be set via environment variables (recommended for Docker/CI) or saved via the Settings UI to a local file on the server. Keys are never sent to OctoBlue.

Do I need to restart after changing API keys or tokens?

It depends on how you change them:

Note: If you update a token via the Settings UI and it still seems to use the old value, check that you don't also have the same key set as an environment variable — env vars are loaded at startup and may override the saved value depending on configuration precedence.

Pricing

How much does TestIntel cost?

All tiers are BYOK — you provide your own AI API key.

Is there a free tier?

Not currently. Running without a license key gives you unlicensed solo mode for evaluation.

Do I pay for AI usage separately?

Yes. TestIntel charges for the tool. AI token costs are paid directly to your AI provider (Anthropic, OpenAI, or AWS).

Integrations

Which Jira versions are supported?

Jira Cloud. Jira Server/Data Center support is planned.

Can I use both Xray and Zephyr Scale?

Yes, but typically a project uses one or the other. Both can be configured simultaneously — the sync buttons in the UI show whichever is configured.

What test result formats are supported?

JUnit XML (covers pytest, JUnit, TestNG, NUnit, Playwright, Cypress, behave), Cucumber JSON (behave, Cucumber, SpecFlow), and Allure JSON (cross-framework with environment metadata). Additional formats planned.

What does the Take Snapshot button do?

The Take Snapshot button (on the Inventory tab) captures the current state of your test inventory — total tests, pass/fail counts, automation %, by type, by priority, pending review, etc. This data powers the Dashboard report. Snapshots are also taken automatically whenever test results are imported, so you typically only need the manual button after bulk changes like approving tests, syncing from Jira/Xray/Zephyr, or reorganizing your inventory.

What is the Dashboard?

The Dashboard (opened from the Reporting tab) is a comprehensive HTML report combining inventory state data (from snapshots) with test execution trends. It shows current metrics, automation progress, pass rate, breakdowns by type and priority, and historical trends. You can filter by project, date range, and source.

Can I import requirements from tools other than Jira?

Yes. Upload CSV, JSON, XML, TXT, or MD files from any source (Azure DevOps, Rally, spreadsheets, plain text).

S3 Knowledge Base Setup

How do I configure kb_s3_path for a project?

Set the kb_s3_path field on a project via the Project settings UI or the Projects API:

# Via API
curl -X PUT http://localhost:8000/projects/my-project \
  -H "x-api-key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"kb_s3_path": "my-bucket/docs/project-kb"}'

In the Web UI, navigate to Projects → Edit and fill in the "S3 KB Path" field.

What is the expected format for kb_s3_path?

The format is bucket-name/prefix/path (no s3:// scheme prefix). For example:

my-company-docs/testintel/knowledge-base

TestIntel splits on the first / to determine the bucket name (my-company-docs) and the object prefix (testintel/knowledge-base).

What file types are loaded from S3?

TestIntel loads files matching the KB_FILE_EXTENSIONS list:

.md, .txt, .csv, .json, .yaml, .yml, .xml, .rst, .html, .toml

Files with other extensions are ignored during S3 knowledge base loading.

What dependencies are required?

The boto3 library must be installed for S3 operations. Install it with:

pip install testintel[all]
# or individually:
pip install boto3

AWS credentials are resolved via the standard boto3 chain (environment variables AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, IAM role, ~/.aws/credentials, etc.). The region defaults to AWS_REGION env var or us-east-1.

Results & Project Association

How are test results associated with a project?

Results are associated with a project based on the ?project= parameter in the webhook or upload URL:

# Results go to the "my-app" project
curl -X POST "http://testintel:8000/results/webhook?project=my-app" \
  -H "x-api-key: YOUR_KEY" -H "Content-Type: application/xml" \
  --data-binary @results.xml

# Results go to root scope (No project)
curl -X POST "http://testintel:8000/results/webhook" \
  -H "x-api-key: YOUR_KEY" -H "Content-Type: application/xml" \
  --data-binary @results.xml

What happens if I specify a project that doesn't exist?

TestIntel rejects the request with a 400 error: *"Project 'my-app' not found. Create it in TestIntel first, or omit the project parameter to store results without project association."*

Create the project in TestIntel first (via the UI or API), then your CI pipeline will work. This ensures results are always intentionally associated with the correct project.

What if I don't specify a project?

Results are stored at root scope and labeled "No project" in the Reporting tab. You can view these by selecting "No Project" from the project filter dropdown.

To fix this for future results, add ?project=your-project-name to your CI webhook URL.

Can I reassign results to a different project later?

Not currently. Results stay in the scope they were originally stored in. This is planned as a future enhancement. For now, the best approach is to update your CI webhook URL to include the correct ?project= parameter.

My CI webhook returns {"detail":"Not Found"}

Your TESTINTEL_URL likely has a trailing slash. The URL must not end with /:

• ✅ http://your-server:8000
• ❌ http://your-server:8000/

A trailing slash causes the request path to become //results/webhook which doesn't match any route. Remove the trailing slash from your CI secret/variable and retry.

How do I view results in the Reporting tab?

The project filter dropdown offers:

• Active Project — shows results for the currently selected project
• All Projects — shows results from all projects plus unassociated results
• No Project — shows only unassociated results (no ?project= specified)
• {Project name} — shows results for a specific project

Status Definitions

TestIntel uses several status fields across different parts of the system. This section explains what each status means and where it applies.

Staging Review Statuses

These statuses apply to AI-generated tests in the staging queue, before they are promoted to inventory.

Inventory Test Statuses (test_status — lifecycle)

These statuses track the lifecycle of a test case in the inventory.

Inventory Test Results (test_result — execution)

These statuses reflect the outcome of the most recent test execution.

Automation Status

Indicates whether a test is automated and how it is executed.

Settings Reference

Admin settings are available under Settings in the Web UI. This section explains each configurable option.

Security Settings

Session Settings

Health Score Settings

Code Context Settings

Knowledge Base Settings

Knowledge Base Storage & Backup

Where are uploaded KB documents stored?

By default, KB documents are stored on the local filesystem:

testintel_data/knowledge_base/           # Shared KB (all projects)
testintel_data/knowledge_base/{scope}/   # Project-specific KB

In Docker deployments, this maps to the host via the volume mount in docker-compose.yml:

volumes:
  - ./testintel_data:/app/testintel_data

How do I back up KB data?

Back up the entire testintel_data/ directory. This contains:

• Knowledge Base documents
• Test inventory
• Results history
• User accounts and secrets
• Project configuration

# Simple backup
cp -r ./testintel_data ./testintel_data_backup_$(date +%Y%m%d)

# Or use your existing backup solution (rsync, S3 sync, etc.)
aws s3 sync ./testintel_data s3://my-backups/testintel/

Can I use S3 for KB storage instead?

Yes. Set the KB storage provider to S3 in config.yaml:

knowledge_base:
  storage_provider: s3
  s3:
    bucket: my-kb-bucket
    region: us-east-1

This stores all KB documents in S3 instead of the local filesystem. Benefits:

• Shared across multiple TestIntel instances
• Durable (S3 99.999999999% durability)
• No local backup needed for KB data

Requires boto3 installed (pip install testintel[all]) and AWS credentials configured.

Can I pre-load KB documents before starting TestIntel?

Yes. Drop files directly into the testintel_data/knowledge_base/ directory:

# Shared KB (available to all projects)
cp my-standards.md ./testintel_data/knowledge_base/

# Project-specific KB
mkdir -p ./testintel_data/knowledge_base/my-org/my-project/
cp project-context.md ./testintel_data/knowledge_base/my-org/my-project/

Supported file types: .md, .txt, .csv, .json, .yaml, .yml, .xml, .rst, .html, .toml

TestIntel will pick up these files on the next chat request — no restart needed.

What happens if I lose my server?

If you're using local storage (default) and lose the server without a backup, all data is lost. To prevent this:

1. Regular backups — schedule automated backups of testintel_data/

1. Use S3 storage — for KB and/or all data (most durable option)

1. Docker volumes — ensure your volume mount points to a backed-up location

How do Shared KB and Project KB interact?

Both are included in every chat request. The AI sees all applicable documents:

1. Project-specific KB — loaded first (files under the project's scope)

1. Shared KB — loaded second (root-level files available to all projects)

There is no automatic override — both are present in the AI's context simultaneously.

Best practice:

• Shared KB — general standards that apply across all projects (BDD format, naming conventions, test strategy, automation patterns)
• Project KB — project-specific context (architecture, endpoints, business rules, unique requirements)

If there's a contradiction: The AI tends to favor the more specific (project) instruction over the general (shared) one, but this isn't guaranteed. To make overrides explicit, add a note in the project KB:

"For this project, use Selenium instead of Playwright for browser automation."

The AI will follow the more specific instruction when it sees both.

Tip: Keep shared KB docs focused on *how* to write tests (format, style, conventions). Keep project KB focused on *what* to test (features, architecture, business logic).

Contact & Support

For questions, bug reports, or feature requests:

Email: support@octobluetech.com

We typically respond within 1 business day.