Use this service when you need this capability: parses robots.txt groups and applies deterministic longest-match allow and disallow semantics.
0.005 USDC per call
x402 V2 exact payment on Base mainnet (eip155:8453) using USDC. The HTTP 402 challenge confirms the exact requirements before an authorized payment.
POST https://signalharness.ai/api/agent/services/robots_policy_parse/invokeParses robots.txt groups and applies deterministic longest-match allow and disallow semantics.
Documented public-source retrieval is bounded by the URL, byte, redirect, and timeout limits below. Source availability can affect execution; review the input schema and returned source metadata.
| Field | Type | Required | Limits |
|---|---|---|---|
| url | string | Optional | Pattern: ^https://; Minimum length 9; Maximum length 2048 |
| text | string | Optional | Minimum length 0; Maximum length 65536 |
| userAgent | string | Optional | Minimum length 1; Maximum length 256 |
| targetPath | string | Optional | Minimum length 1; Maximum length 2048 |
{
"text": "User-agent: *\nAllow: /"
}{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"additionalProperties": false,
"minProperties": 1,
"maxProperties": 3,
"required": [],
"properties": {
"url": {
"type": "string",
"pattern": "^https://",
"minLength": 9,
"maxLength": 2048
},
"text": {
"type": "string",
"minLength": 0,
"maxLength": 65536
},
"userAgent": {
"type": "string",
"minLength": 1,
"maxLength": 256
},
"targetPath": {
"type": "string",
"minLength": 1,
"maxLength": 2048
}
}
}Illustrative output from the canonical adapter fixture and schema. It is not a completed paid execution or a current live observation.
{
"groups": [
{
"userAgents": [
"example"
],
"rules": [
{
"directive": "allow",
"path": "example"
}
],
"crawlDelay": 0
}
],
"matchedGroup": [
null
],
"matchedRule": {},
"crawlAllowed": false,
"sitemaps": [
"example"
],
"crawlDelay": 0,
"warnings": [
"Verify the caller-supplied data before relying on this result."
]
}{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"additionalProperties": false,
"minProperties": 7,
"maxProperties": 7,
"required": [
"groups",
"matchedGroup",
"matchedRule",
"crawlAllowed",
"sitemaps",
"crawlDelay",
"warnings"
],
"properties": {
"groups": {
"type": "array",
"minItems": 0,
"maxItems": 256,
"items": {
"type": "object",
"additionalProperties": false,
"minProperties": 3,
"maxProperties": 3,
"required": [
"userAgents",
"rules",
"crawlDelay"
],
"properties": {
"userAgents": {
"type": "array",
"minItems": 1,
"maxItems": 32,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 256
}
},
"rules": {
"type": "array",
"minItems": 0,
"maxItems": 256,
"items": {
"type": "object",
"additionalProperties": false,
"minProperties": 2,
"maxProperties": 2,
"required": [
"directive",
"path"
],
"properties": {
"directive": {
"type": "string",
"enum": [
"allow",
"disallow"
],
"minLength": 5,
"maxLength": 8
},
"path": {
"type": "string",
"minLength": 0,
"maxLength": 2048
}
}
}
},
"crawlDelay": {
"type": [
"number",
"null"
]
}
}
}
},
"matchedGroup": {
"type": [
"array",
"null"
]
},
"matchedRule": {
"type": [
"object",
"null"
]
},
"crawlAllowed": {
"type": [
"boolean",
"null"
]
},
"sitemaps": {
"type": "array",
"minItems": 0,
"maxItems": 256,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 2048
}
},
"crawlDelay": {
"type": [
"number",
"null"
]
},
"warnings": {
"type": "array",
"minItems": 0,
"maxItems": 32,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 1024
}
}
}
}Timeout: 15000 ms. Full schemas are in OpenAPI.
This schema-valid request returns the canonical HTTP 402 challenge without executing the service. A payment-capable client validates the challenge, signs only with prior authority and budget, and retries the identical body.
curl -i -X POST https://signalharness.ai/api/agent/services/robots_policy_parse/invoke \
-H "Content-Type: application/json" \
--data '{"text":"User-agent: *\nAllow: /"}'