Tasks

Fetch Webpage Content

Fetches and extracts content from one or more web pages with optional AI-generated summaries

# Basic usage

Fetch full text content from a webpage.

# Parameters

# urls

required

Array<String>

Array of URLs to fetch content from. Multiple URLs can be provided to fetch content from multiple pages in a single task.

# settings

optional

object

Configuration settings for fetching webpage content.

# Result

# Properties

# contents

Array

An array of webpage content objects.

# contents.id

string

Unique identifier for the fetched content (typically the URL).

# contents.title

string

The title of the webpage.

# contents.url

string

The URL of the fetched webpage.

# contents.author

string

The author of the content (if available).

# contents.text

string

The full text content extracted from the webpage. Only present when settings.contents.full_text is set to true.

# contents.highlights

Array<string>

Array of highlighted sections or key passages from the content. Only present when settings.contents.highlights is set to true.

# contents.highlightScores

Array<number>

Relevance scores for each highlight. Only present when settings.contents.highlights is set to true.

# contents.summary

string

AI-generated summary of the content. Only present when settings.ai_summary.enabled is set to true.

# contents.image

string

URL to a preview image for the content (if available).

# contents.favicon

string

URL to the website's favicon (if available).

# Examples

# Fetching content from multiple URLs

Fetch content from multiple webpages in a single task:

# Result

# Fetching with AI summaries

Enable AI-generated summaries for fetched content:

# Result

# Fetching with highlights

Extract key highlights from webpage content:

# Result

# Using dynamic URLs from stub data

Fetch content from URLs stored in stub data:

This example uses Data Substitution to dynamically fetch URLs stored in stub.data.article_urls.

# Result

# Fetching documentation pages

Fetch technical documentation for knowledge base building:

# Result