Building a Custom Intelligence Pipeline with Four Apify Actors

An automated market intelligence pipeline ingests multi-channel competitive metrics by combining website traffic estimates, organic search positions, editorial news reporting, and ad transparency records. By orchestrating four targeted tools on Apify, engineering teams can continuously ingest domain traffic distribution, AI referral traffic, localized keyword rankings, publisher news coverage, and active ad creatives without building custom scraping infrastructure.

To construct an operational intelligence pipeline, four specialized scrapers fulfill distinct data collection roles:

1. Similarweb Scraper: Analyzes domain-level traffic volume, engagement metrics (bounce rate, pages per visit, average time on site), global and category ranks, six traffic source channels (direct, search, social, referrals, mail, paid referrals), top search keywords, and AI referral traffic insights including chatbot platform distribution and user prompts.
2. Google Search Results Scraper: Extracts organic search positions, site links, related search queries, and People Also Ask questions across 48 localized regional search engines while handling proxy rotation and rate limiting automatically.
3. Google News Scraper: Collects real-time headlines, publisher sources, direct article links without Google redirects, publication dates, and optional full-text body content extracted using trafilatura and rendered via Playwright without requiring proxies.
4. Google Ads Scraper: Extracts official ad creatives, ad formats (TEXT, IMAGE, VIDEO), active date ranges (first shown and last shown), preview links, and advertiser identifiers directly from the Google Ads Transparency Center.

Practical use cases

Use case 1: Domain visibility and AI referral tracking

Outcome: Benchmark your domain against industry competitors across total visit volumes, six traffic channels, and incoming AI referral traffic.

Question to answer: How does our domain traffic volume, channel mix, and AI chatbot referral presence compare against top market competitors?

Configure: Pass target competitor domain names to Similarweb Scraper with maxItems set to 100. Concurrently, provide core market search terms to Google Search Results Scraper with countryCode set to 'us' and maxResults set to 100.

Working method: Execute Similarweb Scraper to fetch totalVisits, trafficSourceSearch, trafficSourceReferrals, and aiTotalVisits. Run Google Search Results Scraper to extract organic ranking positions per search term. Join datasets on root domain names to correlate traffic volume with search positions.

Deliverable: A structured matrix mapping each domain's monthly visits, channel breakdown decimals, top search rankings, AI platform distribution, and user prompts sending traffic.

Stop condition: Processing halts when all input domain profiles finish parsing in Similarweb Scraper and all search query pages complete execution in Google Search Results Scraper.

Use case 2: Press coverage and consumer search intent monitoring

Outcome: Track editorial press mentions, news article body text, and related consumer search questions following market events or product releases.

Question to answer: Which news outlets are reporting on target company releases, what main text appears in articles, and what consumer questions appear in search engine results?

Configure: Set search keywords in Google News Scraper with extractFullText set to true and dateRange set to '7d'. Set matching brand keywords in Google Search Results Scraper with timePeriod set to 'week'.

Working method: Google News Scraper resolves direct publisher links, deduplicates articles across search terms, and uses Playwright and trafilatura to extract body text. Google Search Results Scraper extracts People Also Ask questions and related search queries. Pipeline scripts cross-reference article text themes against consumer search questions.

Deliverable: A unified media dataset containing resolved publisher URLs, publication timestamps, full article body text, hero image links, and consumer search questions.

Stop condition: Data collection finishes when deduplicated news articles reach maxResultsPerQuery and search results finish parsing for all query terms.

Use case 3: Ad transparency and competitor creative auditing

Outcome: Catalog competitor advertising creatives, format breakdowns, and active dates across regional markets.

Question to answer: What creative messaging, ad formats, and preview links are active for target competitors in specific regions?

Configure: Supply advertiser names or direct Transparency Center links to Google Ads Scraper. Set region to 'US' or 'DE' and resultsLimit to 100 ads per advertiser.

Working method: Execute Google Ads Scraper to query the Google Ads Transparency Center. Extract creativeId, advertiserId, format (TEXT, IMAGE, VIDEO), firstShown, lastShown, previewUrl, imageUrl, and videoUrl. Filter historical records by checking lastShown dates against active campaign thresholds.

Deliverable: An ad inventory file listing unique creative IDs, advertiser names, format types, active date windows, and direct media preview links.

Stop condition: Ingestion completes when Google Ads Scraper reaches the configured resultsLimit for each specified target advertiser.

Operational implementation workflow

Building an operational market intelligence pipeline requires coordinating execution across all four tools sequentially or in parallel:

1. Target configuration: Define master lists containing domain names, search keywords, media tracking phrases, and target advertiser search terms or start URLs.
2. Macro traffic collection: Run Similarweb Scraper to extract traffic metrics, engagement stats, 3 months of estimated monthly visit history, six traffic source decimal shares, top search keywords, and AI referral traffic fields including aiTrafficSplit, aiChatbotDistribution, and topPrompts.
3. Organic SERP collection: Trigger Google Search Results Scraper with target queries, setting countryCode, languageCode, timePeriod, and maxPagesPerQuery to extract organic rankings, site links, related queries, and People Also Ask entries.
4. Media news gathering: Trigger Google News Scraper with brand terms, setting dateRange, siteFilter, or dateFrom and dateTo. Enable extractFullText and includeImages to parse full article text with trafilatura and capture hero image links.
5. Advertising asset extraction: Trigger Google Ads Scraper using searchTerms or startUrls, filtering by region and limiting results via resultsLimit.
6. Dataset consolidation: Combine outputs into a central repository, joining domain traffic from Similarweb Scraper, search positions from Google Search Results Scraper, editorial text from Google News Scraper, and ad creatives from Google Ads Scraper.

Data quality boundaries and operational limits

Engineers must account for technical boundaries across all four scrapers when setting up pipeline workflows:

• Low traffic websites: Similarweb Scraper sets the isSmall field to true when SimilarWeb lacks sufficient traffic panel data for a domain. In these instances, traffic metrics can return as zero or null.
• Input domain batching: Similarweb Scraper processes up to 5,000 domains in a single run and normalizes bare domains, URLs with protocols, or URLs with query paths automatically.
• Dynamic ads omission in search: Google Search Results Scraper utilizes fast HTTP requests. Because Google loads ads via JavaScript after the initial page load, paid ads are not captured in search SERP output. Ad creative collection requires using Google Ads Scraper.
• SERP pagination bounds: Google Search Results Scraper supports up to 200 results per query across up to 10 pages of search results.
• Paywalled media content: Google News Scraper uses Playwright and trafilatura to extract article text, but fullText returns null for articles behind paywalls.
• Deduplication across news queries: Google News Scraper automatically deduplicates articles across multiple input search terms so each unique article is output once.
• Memory usage with browser concurrency: Setting maxConcurrency high in Google News Scraper increases extraction speed but consumes more memory because Playwright opens parallel browser instances.
• Speed trade-offs in ad scraping: Enabling skipDetails on Google Ads Scraper makes scraping much faster, but returns null for extended detail fields like specific image or video URLs, returning only basic IDs and URLs.

Frequently asked questions

How does Similarweb Scraper flag low traffic domains?

When SimilarWeb lacks sufficient traffic panel data to create reliable traffic and engagement estimates for a website, Similarweb Scraper sets the isSmall field to true in the output dataset row. While an output item is created, individual traffic metrics may be zero.

Why does Google Search Results Scraper omit paid ads from SERPs?

Google loads ads via JavaScript after the initial page load. Because Google Search Results Scraper uses fast HTTP requests to crawl SERP markup efficiently, paid ads are not captured. However, all organic search results, related search queries, and People Also Ask questions are fully extracted.

How does Google News Scraper handle duplicate articles across search terms?

When processing multiple search keywords in a single run, Google News Scraper automatically deduplicates articles across queries, ensuring that each unique article is recorded only once in the dataset.

What happens when Skip Details is enabled in Google Ads Scraper?

When skipDetails is set to true in Google Ads Scraper, the tool skips fetching full details for each ad. This operates much faster, but causes extended detail fields to return null while providing top-level IDs and URLs.

What AI referral traffic data can be extracted with Similarweb Scraper?

Similarweb Scraper extracts four dedicated AI fields: aiTrafficSplit showing platforms sending traffic, aiChatbotDistribution showing percentage breakdown per chatbot, aiTotalVisits showing total AI referral volume, and topPrompts showing user prompts asked to AI chatbots that mention the website.

What export formats are supported for Google Search Results Scraper?

Results from Google Search Results Scraper can be exported as JSON, CSV, Excel, HTML, XML, or RSS. Enabling the csvFriendlyOutput option outputs one result per row for spreadsheet analysis.

Resources

• Similarweb Scraper: Website traffic analytics, rankings, keyword distribution, and AI referral traffic insights.
• Google Search Results Scraper: Multi-country SERP organic rankings, related queries, and People Also Ask data.
• Google News Scraper: Real-time press monitoring, direct URL resolution, and full-text article extraction.
• Google Ads Scraper: Google Ads Transparency Center creative details, media preview links, and advertiser active dates.