AnalysisDeterministic3 credits

analyze_content

Statistical content analysis: language detection, keyword frequency, weighted-lexicon sentiment, and Flesch reading ease. Point it at a URL and it fetches and strips the page for you, or pass text directly. No model call is involved, so the same input always produces the same output.

Use Cases

Content Classification

Route articles by detected language and dominant keywords before they reach a more expensive pipeline stage.

Sentiment Monitoring

Track the tone of reviews and feedback. Every score ships with the exact words that produced it, so a surprising result can be audited.

Readability Auditing

Flesch reading ease over a set of pages, to find documentation that has drifted denser than the rest.

Cheap Pre-filtering

Screen scraped pages on length, language and keywords before spending credits on summarize_content or extract_with_llm.

Multilingual Routing

Detect the language of a page and send it to the right downstream handler or translator.

Corpus Statistics

Word, sentence and average-sentence-length counts across a crawl, for reporting or quality gates.

Endpoint

POST/api/v1/tools/analyze_content

Auth Required

1 req/s on Free plan

3 credits

Parameters

Name	Type	Required	Default	Description
url	string	Optional	-	Page to fetch and analyze. Scripts, styles, nav, header and footer are stripped and the remaining body text is analyzed. Provide either `url` or `content`. Example: https://example.com/blog/post
content	string	Optional	-	Text to analyze directly, skipping the fetch. Takes effect only when `url` is omitted — if both are sent, the fetched page wins. Example: The dashboard is a great upgrade...
analyze_sentiment	boolean	Optional	false	Score sentiment against a weighted English lexicon. Off by default; the `sentiment` object is absent unless you ask for it. On text in another language where no lexicon word appears, `label` is `not_applicable` with `notApplicable: "sentiment-lexicon-is-english-only"`, rather than a `neutral` that was never measured. Example: true
extract_keywords	boolean	Optional	true	Return the ten most frequent words longer than three characters, with their counts. English function words such as `that`, `with` and `from` are never counted, and German ones such as `nicht`, `eine` and `dass` are left out too when the text is detected as German. Example: true
detect_language	boolean	Optional	true	Detect the language and return an ISO 639-1 code, or `und` when the text is too short or the language is not recognised. Example: true
analyze_readability	boolean	Optional	false	Compute Flesch reading ease and a coarse grade level. Off by default; the `readability` object is absent unless you ask for it. Example: true
timeout	number	Optional	10000	Fetch timeout in milliseconds, between 1000 and 30000. Only applies when `url` is used. Example: 10000
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Request Examples

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/analyze_content \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/dashboard-review",
    "analyze_sentiment": true,
    "analyze_readability": true
  }'

Response Example

200 OK310ms

{
  "success": true,
  "data": {
    "content_length": 407,
    "word_count": 75,
    "sentence_count": 6,
    "avg_sentence_length": 13,
    "language": "en",
    "keywords": [
      {
        "word": "dashboard",
        "frequency": 3
      },
      {
        "word": "easy",
        "frequency": 2
      },
      {
        "word": "great",
        "frequency": 1
      },
      {
        "word": "upgrade",
        "frequency": 1
      }
    ],
    "sentiment": {
      "score": 0.924,
      "label": "positive",
      "positive_matches": [
        "great",
        "easy",
        "clear",
        "helpful",
        "fast",
        "easy",
        "reliable",
        "recommend"
      ],
      "negative_matches": [],
      "method": "lexicon"
    },
    "readability": {
      "flesch_score": 76,
      "grade_level": "middle_school"
    }
  },
  "credits_used": 3,
  "credits_remaining": 997,
  "processing_time": 310
}

Field Descriptions

data.content_lengthCharacter count of the analyzed text, after HTML stripping

data.avg_sentence_lengthWords per sentence, rounded to a whole number

data.languageISO 639-1 code, or `und`. Present only when `detect_language` is true (the default)

data.keywordsTop ten words over three characters by raw frequency. English function words are left out, and German ones too when the text is German

data.sentiment.scoreSum of matched lexicon weights divided by the square root of the word count, clamped to -1..1

data.sentiment.label`positive` above 0.05, `negative` below -0.05, otherwise `neutral` — or `not_applicable`, with `notApplicable`, on non-English text that matched no lexicon word

data.sentiment.positive_matchesEvery word that contributed positively, in order and with duplicates — this is what the score is made of

data.sentiment.methodAlways `lexicon`. No model is called

data.readability.flesch_scoreFlesch reading ease, clamped to 0-100. Higher is easier

data.readability.grade_level`middle_school`, `high_school` or `college`, derived from sentence length alone

credits_usedFixed 3 credits per call, whatever the text length

Error Handling

Neither url nor content (400 Bad Request)

VALIDATION_ERROR. At least one of the two must be present. The same status covers a malformed url and a timeout outside 1000-30000.

Page Too Large (413 Payload Too Large)

RESPONSE_TOO_LARGE. The fetched body exceeded the 25MB read limit and was refused rather than buffered. Pass the text yourself via content if you need to analyze it.

Target Timed Out (504 Gateway Timeout)

FETCH_TIMEOUT. The page stopped responding while sending its body. Raise timeout, up to 30000ms.

Fetch Failed (502 Bad Gateway)

FETCH_FAILED. The response body could not be read — connection reset, or a body that is not decodable text.

Analysis Failed (500 Internal Server Error)

TOOL_ERROR. Nothing is charged for a failed call; credits are only deducted after the analysis succeeds.

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Note:

The sentiment lexicon is English-only. On text in another language that matches no lexicon word, sentiment.label is not_applicable, sentiment.notApplicable is sentiment-lexicon-is-english-only and score is 0. Non-English text that happens to contain English sentiment words is still scored, so check data.language before trusting data.sentiment.