AnalysisDeterministic3 credits

analyze_content

Statistical content analysis: language detection, keyword frequency, weighted-lexicon sentiment, and Flesch reading ease. Point it at a URL and it fetches and strips the page for you, or pass text directly. No model call is involved, so the same input always produces the same output.

Use Cases

Content Classification

Route articles by detected language and dominant keywords before they reach a more expensive pipeline stage.

Sentiment Monitoring

Track the tone of reviews and feedback. Every score ships with the exact words that produced it, so a surprising result can be audited.

Readability Auditing

Flesch reading ease over a set of pages, to find documentation that has drifted denser than the rest.

Cheap Pre-filtering

Screen scraped pages on length, language and keywords before spending credits on summarize_content or extract_with_llm.

Multilingual Routing

Detect the language of a page and send it to the right downstream handler or translator.

Corpus Statistics

Word, sentence and average-sentence-length counts across a crawl, for reporting or quality gates.

Endpoint

POST/api/v1/tools/analyze_content

Auth Required

1 req/s on Free plan

3 credits

Parameters

Name	Type	Required	Default	Description
url	string	Optional	-	Page to fetch and analyze. Scripts, styles, nav, header and footer are stripped and the remaining body text is analyzed. Provide either `url` or `content`. Example: https://example.com/blog/post
content	string	Optional	-	Text to analyze directly, skipping the fetch. Takes effect only when `url` is omitted — if both are sent, the fetched page wins. Example: The dashboard is a great upgrade...
analyze_sentiment	boolean	Optional	false	Score sentiment against a weighted English lexicon. Off by default; the `sentiment` object is absent unless you ask for it. Example: true
extract_keywords	boolean	Optional	true	Return the ten most frequent words longer than three characters, with their counts. Example: true
detect_language	boolean	Optional	true	Detect the language and return an ISO 639-1 code, or `und` when the text is too short or the language is not recognised. Example: true
analyze_readability	boolean	Optional	false	Compute Flesch reading ease and a coarse grade level. Off by default; the `readability` object is absent unless you ask for it. Example: true
timeout	number	Optional	10000	Fetch timeout in milliseconds, between 1000 and 30000. Only applies when `url` is used. Example: 10000
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Request Examples

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/analyze_content \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/dashboard-review",
    "analyze_sentiment": true,
    "analyze_readability": true
  }'

Response Example

200 OK310ms

{
  "success": true,
  "data": {
    "content_length": 407,
    "word_count": 75,
    "sentence_count": 6,
    "avg_sentence_length": 13,
    "language": "en",
    "keywords": [
      {
        "word": "dashboard",
        "frequency": 3
      },
      {
        "word": "easy",
        "frequency": 2
      },
      {
        "word": "that",
        "frequency": 2
      },
      {
        "word": "great",
        "frequency": 1
      },
      {
        "word": "upgrade",
        "frequency": 1
      }
    ],
    "sentiment": {
      "score": 0.924,
      "label": "positive",
      "positive_matches": [
        "great",
        "easy",
        "clear",
        "helpful",
        "fast",
        "easy",
        "reliable",
        "recommend"
      ],
      "negative_matches": [],
      "method": "lexicon"
    },
    "readability": {
      "flesch_score": 76,
      "grade_level": "middle_school"
    }
  },
  "credits_used": 3,
  "credits_remaining": 997,
  "processing_time": 310
}

Field Descriptions

data.content_lengthCharacter count of the analyzed text, after HTML stripping

data.avg_sentence_lengthWords per sentence, rounded to a whole number

data.languageISO 639-1 code, or `und`. Present only when `detect_language` is true (the default)

data.keywordsTop ten words over three characters by raw frequency. There is no stop-word list, so common words such as `that` and `with` can appear

data.sentiment.scoreSum of matched lexicon weights divided by the square root of the word count, clamped to -1..1

data.sentiment.label`positive` above 0.05, `negative` below -0.05, otherwise `neutral`

data.sentiment.positive_matchesEvery word that contributed positively, in order and with duplicates — this is what the score is made of

data.sentiment.methodAlways `lexicon`. No model is called

data.readability.flesch_scoreFlesch reading ease, clamped to 0-100. Higher is easier

data.readability.grade_level`middle_school`, `high_school` or `college`, derived from sentence length alone

credits_usedFixed 3 credits per call, whatever the text length

Error Handling

Neither url nor content (400 Bad Request)

VALIDATION_ERROR. At least one of the two must be present. The same status covers a malformed url and a timeout outside 1000-30000.

Page Too Large (413 Payload Too Large)

RESPONSE_TOO_LARGE. The fetched body exceeded the 25MB read limit and was refused rather than buffered. Pass the text yourself via content if you need to analyze it.

Target Timed Out (504 Gateway Timeout)

FETCH_TIMEOUT. The page stopped responding while sending its body. Raise timeout, up to 30000ms.

Fetch Failed (502 Bad Gateway)

FETCH_FAILED. The response body could not be read — connection reset, or a body that is not decodable text.

Analysis Failed (500 Internal Server Error)

TOOL_ERROR. Nothing is charged for a failed call; credits are only deducted after the analysis succeeds.

Blocked by robots.txt (403 Forbidden)

The target site's robots.txt disallows this path for CrawlForge. Set respect_robots: false to override if you have your own agreement with the target — the override is recorded against your API key. The override does not reach a host on CrawlForge's permanent opt-out list, which is refused whatever respect_robots is set to.

Note: The sentiment lexicon is English-only. Running it over text in another language returns a score near zero rather than an error, so check data.language before trusting data.sentiment.

Credit Cost

3 credits

3 credits per call

Flat rate whether you pass a URL or raw text, and whichever analyses you enable. Failed calls are not charged.

What's Included:

Language detection (ISO 639-1, trigram-based)

Top-ten keyword frequency

Weighted-lexicon sentiment with the matched words

Flesch reading ease and grade level

Word, sentence and character counts

Plan Recommendations:

Free Plan: 1,000 one-time trial credits = 333 analyses

Hobby Plan: 5,000 credits = 1,666 analyses ($19/mo)

Professional Plan: 50,000 credits = 16,666 analyses ($99/mo)

Related Tools

extract_content

Pull clean article text out of a page first (2 credits)

summarize_content

Condense a page instead of measuring it (4 credits)

localization

Detect language plus hreflang and geo-targeting signals (2 credits)

batch_scrape

Collect many pages, then analyze each one (5 credits per URL)

Ready to try analyze_content? Sign up for free and get 1,000 credits to start building.

Use Cases

Content Classification

Route articles by detected language and dominant keywords before they reach a more expensive pipeline stage.

Sentiment Monitoring

Track the tone of reviews and feedback. Every score ships with the exact words that produced it, so a surprising result can be audited.

Readability Auditing

Flesch reading ease over a set of pages, to find documentation that has drifted denser than the rest.

Cheap Pre-filtering

Screen scraped pages on length, language and keywords before spending credits on summarize_content or extract_with_llm.

Multilingual Routing

Detect the language of a page and send it to the right downstream handler or translator.

Corpus Statistics

Word, sentence and average-sentence-length counts across a crawl, for reporting or quality gates.

Parameters

Name	Type	Required	Default	Description
url	string	Optional	-	Page to fetch and analyze. Scripts, styles, nav, header and footer are stripped and the remaining body text is analyzed. Provide either `url` or `content`. Example: https://example.com/blog/post
content	string	Optional	-	Text to analyze directly, skipping the fetch. Takes effect only when `url` is omitted — if both are sent, the fetched page wins. Example: The dashboard is a great upgrade...
analyze_sentiment	boolean	Optional	false	Score sentiment against a weighted English lexicon. Off by default; the `sentiment` object is absent unless you ask for it. Example: true
extract_keywords	boolean	Optional	true	Return the ten most frequent words longer than three characters, with their counts. Example: true
detect_language	boolean	Optional	true	Detect the language and return an ISO 639-1 code, or `und` when the text is too short or the language is not recognised. Example: true
analyze_readability	boolean	Optional	false	Compute Flesch reading ease and a coarse grade level. Off by default; the `readability` object is absent unless you ask for it. Example: true
timeout	number	Optional	10000	Fetch timeout in milliseconds, between 1000 and 30000. Only applies when `url` is used. Example: 10000
respect_robots	boolean	Optional	true	Respect the target site's robots.txt. Left at `true`, a path disallowed for `CrawlForge` is refused with 403 before anything is fetched and no credits are charged. Set it to `false` only for a target you have your own agreement with — the response then carries a `warnings` entry and the override is recorded against your API key. Example: true

Request Examples

terminalBash

curl -X POST https://crawlforge.dev/api/v1/tools/analyze_content \
  -H "X-API-Key: cf_test_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/dashboard-review",
    "analyze_sentiment": true,
    "analyze_readability": true
  }'

Response Example

200 OK310ms

{
  "success": true,
  "data": {
    "content_length": 407,
    "word_count": 75,
    "sentence_count": 6,
    "avg_sentence_length": 13,
    "language": "en",
    "keywords": [
      {
        "word": "dashboard",
        "frequency": 3
      },
      {
        "word": "easy",
        "frequency": 2
      },
      {
        "word": "that",
        "frequency": 2
      },
      {
        "word": "great",
        "frequency": 1
      },
      {
        "word": "upgrade",
        "frequency": 1
      }
    ],
    "sentiment": {
      "score": 0.924,
      "label": "positive",
      "positive_matches": [
        "great",
        "easy",
        "clear",
        "helpful",
        "fast",
        "easy",
        "reliable",
        "recommend"
      ],
      "negative_matches": [],
      "method": "lexicon"
    },
    "readability": {
      "flesch_score": 76,
      "grade_level": "middle_school"
    }
  },
  "credits_used": 3,
  "credits_remaining": 997,
  "processing_time": 310
}

Field Descriptions

data.content_lengthCharacter count of the analyzed text, after HTML stripping

data.avg_sentence_lengthWords per sentence, rounded to a whole number

data.languageISO 639-1 code, or `und`. Present only when `detect_language` is true (the default)

data.keywordsTop ten words over three characters by raw frequency. There is no stop-word list, so common words such as `that` and `with` can appear

data.sentiment.scoreSum of matched lexicon weights divided by the square root of the word count, clamped to -1..1

data.sentiment.label`positive` above 0.05, `negative` below -0.05, otherwise `neutral`

data.sentiment.positive_matchesEvery word that contributed positively, in order and with duplicates — this is what the score is made of

data.sentiment.methodAlways `lexicon`. No model is called

data.readability.flesch_scoreFlesch reading ease, clamped to 0-100. Higher is easier

data.readability.grade_level`middle_school`, `high_school` or `college`, derived from sentence length alone

credits_usedFixed 3 credits per call, whatever the text length

Error Handling

Neither url nor content (400 Bad Request)

VALIDATION_ERROR. At least one of the two must be present. The same status covers a malformed url and a timeout outside 1000-30000.

Page Too Large (413 Payload Too Large)

RESPONSE_TOO_LARGE. The fetched body exceeded the 25MB read limit and was refused rather than buffered. Pass the text yourself via content if you need to analyze it.

Target Timed Out (504 Gateway Timeout)

FETCH_TIMEOUT. The page stopped responding while sending its body. Raise timeout, up to 30000ms.

Fetch Failed (502 Bad Gateway)

FETCH_FAILED. The response body could not be read — connection reset, or a body that is not decodable text.

Analysis Failed (500 Internal Server Error)

TOOL_ERROR. Nothing is charged for a failed call; credits are only deducted after the analysis succeeds.

Blocked by robots.txt (403 Forbidden)

Note: The sentiment lexicon is English-only. Running it over text in another language returns a score near zero rather than an error, so check data.language before trusting data.sentiment.

Credit Cost

3 credits

3 credits per call

Flat rate whether you pass a URL or raw text, and whichever analyses you enable. Failed calls are not charged.

What's Included:

Language detection (ISO 639-1, trigram-based)

Top-ten keyword frequency

Weighted-lexicon sentiment with the matched words

Flesch reading ease and grade level

Word, sentence and character counts

Plan Recommendations:

Free Plan: 1,000 one-time trial credits = 333 analyses

Hobby Plan: 5,000 credits = 1,666 analyses ($19/mo)

Professional Plan: 50,000 credits = 16,666 analyses ($99/mo)