API Evaluation - Tyk Documentation

Availability

Version: Available since v0.2

Overview

API Evaluation enables you to validate API specifications against governance policies before deployment, without requiring the API to be published or stored in your API Repository. This feature helps you catch compliance issues early in the development lifecycle, reducing rework and accelerating the delivery of high-quality APIs.

Key Benefits

Shift-Left Governance: Catch compliance issues during design and development, not after deployment
Reduce Development Cycles: Identify and fix issues before they reach code review or testing phases
Seamless Integration: Easily incorporate governance checks into CI/CD pipelines and development workflows
Detailed Feedback: Receive precise information about violations with line numbers and remediation guidance
No Storage Required: Validate API specifications without storing them in your API Repository

Dependencies

Requires Tyk Governance v0.2 or higher
Requires at least one governance ruleset to be defined

Quick Start

In this tutorial, we’ll validate an API specification against a governance ruleset before deployment.

Prerequisites

Access to Tyk Governance Hub
A governance ruleset ID
An API specification to validate (in OpenAPI format)

Step-by-Step

Identify Your Ruleset Navigate to the Rulesets section in your Tyk Governance dashboard and note the ID of the ruleset you want to use for validation.
Prepare Your API Specification Ensure your API specification is in a valid OpenAPI format (JSON or YAML).

Make an API Request Use the API Evaluation endpoint to validate your specification:

curl -X POST https://your-governance-instance.tyk.io/api/rulesets/evaluate-spec \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_API_KEY" \
-d '{
    "rulesetId": "$RULESET_ID",
    "apiSpec": {
      "name": "My Test API",
      "content": {
        "openapi": "3.0.0",
        "info": {
          "title": "Test API",
          "version": "1.0.0"
        },
        "paths": {
          "/example": {
            "get": {
              "responses": {
                "200": {
                  "description": "OK"
                }
              }
            }
          }
        }
      }
    }
}'

Review the Results The response will include any violations found, with details about each issue:

{
    "status": "Success",
    "message": "Rule violation found",
    "errors": [
        {
            "code": "info-contact",
            "path": ["info"],
            "message": "API must have contact information",
            "severity": "error",
            "range": {
                "start": { "line": 3, "character": 2 },
                "end": { "line": 6, "character": 3 }
            },
            "howToFix": "Add contact information to the info section"
        }
    ]
}

If there are no violations found:

{
  "status" : "Success",
  "message" : "No rule violation found",
  "errors":[]
}

Validation

A successful request with no violations will return an empty errors array
If violations are found, each will include:
- The rule code that was violated
- The path in the API specification where the violation occurred
- A message explaining the issue
- The severity level (error, warning, info, hint)
- The exact location in the file (line and character)
- Guidance on how to fix the issue (if available)

How It Works

API Evaluation works by sending your API specification to the Tyk Governance Hub, where it’s validated against a specified ruleset without being stored in your API Repository. The system applies each rule in the ruleset to your specification and returns detailed results.

Integration into Development Workflow

Integrating with CI/CD Pipelines

API Evaluation can be integrated into your CI/CD pipeline to validate API specifications before they’re deployed automatically. This ensures that only compliant APIs make it to production.

# Example GitHub Actions workflow
name: API Governance Check

on:
  pull_request:
    paths:
      - 'api-specs/**'

jobs:
  validate-api:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Validate API Specification
        run: |
          SPEC_CONTENT=$(cat api-specs/my-api.yaml | awk '{printf "%s\\n", $0}')
          curl -X POST https://your-governance-instance.tyk.io/api/rulesets/evaluate-spec \
            -H "Content-Type: application/json" \
            -H "X-API-Key: ${{ secrets.GOVERNANCE_API_KEY }}" \
            -d "{
              \"rulesetId\": \"your-ruleset-id\",
              \"apiSpec\": {
                \"name\": \"My API\",
                \"content\": \"$SPEC_CONTENT\"
              }
            }" > validation-results.json
            
          # Fail if any errors are found
          ERROR_COUNT=$(jq '.errors | length' validation-results.json)
          if [ $ERROR_COUNT -gt 0 ]; then
            echo "API validation failed with $ERROR_COUNT issues:"
            jq '.errors' validation-results.json
            exit 1
          fi

Pre-commit Validation

Developers can validate their API specifications before committing changes, ensuring they meet governance standards from the start.

#!/bin/bash
# pre-commit hook for API validation

# Get the API specification file
SPEC_FILE=$(git diff --cached --name-only | grep -E '\.json$|\.yaml$|\.yml$' | head -1)

if [ -n "$SPEC_FILE" ]; then
  echo "Validating API specification: $SPEC_FILE"
  
  # Convert the file content to JSON
  if [[ $SPEC_FILE == *.yaml || $SPEC_FILE == *.yml ]]; then
    SPEC_CONTENT=$(yq eval -o=json $SPEC_FILE)
  else
    SPEC_CONTENT=$(cat $SPEC_FILE)
  fi
  
  # Validate the specification
  RESPONSE=$(curl -s -X POST https://your-governance-instance.tyk.io/api/rulesets/evaluate-spec \
    -H "Content-Type: application/json" \
    -H "X-API-Key: YOUR_API_KEY" \
    -d "{
      \"rulesetId\": \"your-ruleset-id\",
      \"apiSpec\": {
        \"name\": \"$(basename $SPEC_FILE)\",
        \"content\": $SPEC_CONTENT
      }
    }")
  
  # Check for errors
  ERROR_COUNT=$(echo $RESPONSE | jq '.errors | length')
  if [ $ERROR_COUNT -gt 0 ]; then
    echo "API validation failed with $ERROR_COUNT issues:"
    echo $RESPONSE | jq '.errors'
    exit 1
  fi
  
  echo "API specification is valid!"
fi

exit 0

Use Cases

Validating APIs During Design Phase

Integrate API Evaluation with design tools to validate specifications during the design phase, before any code is written. Benefits:

Catch issues at the earliest possible stage
Reduce rework and development cycles
Ensure designs align with governance standards from the start

Implementation:

Design an API in your preferred tool
Export the OpenAPI specification
Validate using the API Evaluation endpoint
Review and address any issues
Repeat until the specification passes validation

Automated Testing in Development Workflows

Incorporate API Evaluation into automated testing workflows to ensure continuous compliance during development. Benefits:

Maintain compliance throughout the development process
Prevent regression of governance standards
Provide immediate feedback to developers

Implementation:

Add API validation as a step in your testing pipeline
Run validation after any changes to the API specification
Fail the build if critical violations are found
Generate reports of issues for developers to address

Pre-release Validation Gate

Use API Evaluation as a final check before releasing APIs to production or external consumers. Benefits:

Ensure only compliant APIs are released
Maintain consistent quality standards
Reduce security and compliance risks

Implementation:

Add a validation step in your release pipeline
Block releases with critical violations
Generate compliance reports for audit purposes
Track compliance metrics over time

Best Practices and Recommendations

Integrate early in development: Validate specifications before coding begins to avoid costly rework
Use appropriate rulesets: Select rulesets that match the API’s purpose and criticality
Automate validation: Incorporate validation into CI/CD pipelines and development workflows
Review results carefully: Understand the context of each violation before fixing
Prioritize by severity: Address errors first, then warnings, then informational issues
Track compliance trends: Monitor how compliance improves over time
Update specifications incrementally: Fix critical issues first, then address less severe ones
Document exceptions: When a rule violation is intentional, document the reason
Provide feedback on rules: Help improve governance rules that generate false positives
Use with other governance tools: Combine with API Repository and Ruleset Management for comprehensive governance

FAQs

Can I validate multiple API specifications at once?

What API specification formats are supported?

Is there a size limit for API specifications?

Are evaluated specifications stored in Tyk Governance?

Troubleshooting

Receiving "Invalid request format" error

Evaluation taking too long or timing out

Results show unexpected violations

Authentication failures

General

​Availability

​Overview

​Key Benefits

​Dependencies

​Quick Start

​Prerequisites

​Step-by-Step

​Validation

​How It Works

​Integration into Development Workflow

​Integrating with CI/CD Pipelines

​Pre-commit Validation

​Use Cases

​Validating APIs During Design Phase

​Automated Testing in Development Workflows

​Pre-release Validation Gate

​Best Practices and Recommendations

​FAQs

​Troubleshooting

Availability

Overview

Key Benefits

Dependencies

Quick Start

Prerequisites

Step-by-Step

Validation

How It Works

Integration into Development Workflow

Integrating with CI/CD Pipelines

Pre-commit Validation

Use Cases

Validating APIs During Design Phase

Automated Testing in Development Workflows

Pre-release Validation Gate

Best Practices and Recommendations

FAQs

Troubleshooting