Web Scraping APIs Guide: Features, Performance and Pricing

In recent years, the surge in web scraping activities has prompted the emergence of diverse APIs provided by proxy services and data collection firms.

This report delves into seven prominent vendors in the web scraping API landscape, analyzing their features, scraping capabilities, parsing efficiency, and cost-effectiveness.

Focusing on three key website categories—search engines, e-commerce platforms, and social media—we aim to provide insights into the evolving realm of web scraping APIs.

Evolution of Web Scraping APIs

Web scraping APIs act as remote web scrapers, accepting API requests with target URLs and optional parameters.

Behind the scenes, these APIs utilize proxies, headers, and even headless browsers to retrieve HTML content. Some advanced APIs employ AI vision and pattern recognition for sophisticated tasks.

Pricing models are often based on successful requests, ensuring predictability. However, some providers exhibit opaque pricing structures.

Key Insights

➡️ Data Output and Parsing:

• Six out of seven APIs return raw HTML, with advanced parsers available for specific websites.
• Google and Amazon are the most targeted websites, with Oxylabs offering a machine-learning model for parsing most e-commerce stores.

➡️ Data Transfer and Customization:

• APIs transfer data over open connections, often acting as proxies for seamless integration.
• Customization options include location selection, device specification, and custom headers.
• Four APIs accept CSS selectors and three support browser interactions for dynamic scraping scenarios.

➡️ Performance and Reliability:

• Performance tests reveal varying speeds, with some APIs excelling in Google and Amazon scraping.
• Social media, especially GraphQL, proves challenging for many APIs.
• Oxylabs, Smartproxy, and Bright Data emerge as the most reliable performers, boasting robust parsers.

➡️ Pricing Models:

• Bright Data charges a uniform price for all features, while Oxylabs and Smartproxy differentiate prices by target group.
• ScraperAPI and Zyte employ tiered pricing, with rates differing significantly based on the target website.

Participant Overview

We engaged with seven prominent companies offering web scraping APIs, including established names and proxy providers transitioning into this domain.

The participants willingly provided access to their APIs for scraping Google, Amazon, and a social media network.

Participant Snapshot

Feature Overview

Integration Methods

In theory, all web scraping APIs use the same basic structure: there's an endpoint where you pass URLs you want to scrape with one or more parameters.

In practice, the implementation can differ significantly. Here are the four main methods we've encountered:

• API (open connection): Open connection means sending requests to an API endpoint and waiting for the response. GET and POST methods are used, with variations in implementation.
• API (asynchronous): Asynchronous delivery allows sending API calls with an ID and fetching results over a webhook, which is useful for scraping in bulk.
• Proxy: Most APIs can integrate as proxies, easing the transition from regular proxy servers.
• Library/SDK: Some providers offer SDKs for additional convenience.

HTML Scraping

General-purpose APIs have one endpoint that attempts to scrape any website, returning pages in raw HTML.

All participants offer an API for general-purpose scraping:

Parameters like geolocation, residential proxies, device type, sessions, cookies, and data input are common among APIs.

Headless Scraping

Headless scraping is crucial for overcoming website protection systems.

Most providers manage headless browsers for you:

JavaScript rendering is universally available, and some providers allow interactions with the browser, such as clicking and scrolling.

Specialized APIs

Specialized APIs target specific website groups, ensuring compatibility and structured scraping:

Search engines and e-commerce sites are common targets, with Google and Amazon receiving the most attention.

Google Features

Amazon Features

Data Parsing

Parsing capabilities vary among providers. Some offer specialized APIs with built-in parsers, while others expose selectors for manual parsing. Overall parsing capabilities are as follows:

Pre-built parsers for Google are common, and manual parsing is offered by a few providers. Specialized parsers for Amazon are available, with Oxylabs supporting targets beyond Amazon.

Google Parsing

Amazon Parsing

Performance Benchmarks of Web Scraping APIs

In a comprehensive evaluation of web scraping APIs, a custom Python script utilizing Asyncio and AIOHTTP libraries was employed for asynchronous requests with a timeout of 150 seconds.

The focus was on assessing Google, Amazon, and a photo-centric social media platform across various scenarios.

import asyncio
import aiohttp
from aiohttp import ClientSession

async def fetch_data(session: ClientSession, url: str, timeout: int = 150) -> dict:
    try:
        async with session.get(url, timeout=timeout) as response:
            return await response.json()
    except aiohttp.ClientError as e:
        print(f"Error fetching data from {url}: {e}")
        return {}

async def scrape_google():
    google_url = "https://www.google.com"
    async with aiohttp.ClientSession() as session:
        google_data = await fetch_data(session, google_url)
        print("Google Data:", google_data)

async def scrape_amazon():
    amazon_url = "https://www.amazon.com"
    async with aiohttp.ClientSession() as session:
        amazon_data = await fetch_data(session, amazon_url)
        print("Amazon Data:", amazon_data)

async def main():
    tasks = [
        scrape_google(),
        scrape_amazon(),
    ]
    await asyncio.gather(*tasks)

if __name__ == "__main__":
    asyncio.run(main())

Google

Unparsed Results

Most APIs performed well, with notable exceptions. Shifter's general-purpose scraper faced challenges with Google, resulting in a 429 detection error every fifth request. The specialized API improved performance but experienced a decrease in speed.

Parsed Results

The use of a data parser had minimal impact on response time, except for Rayobyte, which exhibited a three-second delay in JSON results for unexplained reasons.

Amazon

Bright Data, Oxylabs, and Smartproxy consistently delivered excellent results. Rayobyte's slow response was attributed to defaulting to datacenter IPs for Amazon, necessitating multiple request retries. Zyte encountered 520 errors, and ScraperAPI mirrored its Google performance. Shifter performed well, but its scraper faced challenges.

Photo-Focused Social Media Platform

GraphQL Endpoint

The GraphQL endpoint posed a serious challenge, with Shifter struggling even with rendering enabled. ScraperAPI faced difficulties, while Zyte stood out with commendable performance.

Headless Rendering

The headless test was more forgiving, with Bright Data demonstrating superior results. Shifter was fast but faced errors. ScraperAPI had improved performance, and Oxylabs and Smartproxy maintained success rates at the expense of some speed.

Concurrency

Concurrency varied, with Bright Data, Smartproxy, and Oxylabs allowing for high parallel requests. Rayobyte and Zyte had more restrictive default limits, mainly applicable to enterprise-level needs.

Evaluation of Parsing Capabilities in Web Scraping APIs

In a nuanced examination of web scraping APIs, a qualitative test was conducted to assess their parsing abilities on four distinct types of pages: localized Google search desktop query, localized Google search mobile query, Google Shopping query, and Amazon product pages.

Google SERP, Localized Desktop Query

For the localized desktop query "best hairdresser near me" in London, the APIs were evaluated based on various elements:

While ScraperAPI and Rayobyte focused on essential information, others aimed to parse the entire SERP.

Notably, Bright Data even provided a screenshot of the map. Shifter faced issues with the location parameter, hindering local result retrieval.

Google SERP, Localized Mobile Query

The mobile query with the same parameters as the desktop query yielded the following results:

Bright Data, Oxylabs, and Smartproxy successfully returned complete and accurate results. However, ScraperAPI failed to scrape anything, and Shifter's mobile parser regressed to main page elements, missing local data.

Google Shopping

The Google Shopping query for "Nike Air Max" in London was analyzed for various aspects:

ScraperAPI provided the most comprehensive results, including related searches and the "you might like" block. It successfully retrieved ad results, a feature absent in other providers. Bright Data and Shifter failed to localize the page for this specific query.

Amazon Product Pages

Various product pages from art supplies, kitchenware, and electronics were targeted for parsing. The evaluation included elements such as breadcrumbs, item details, images, pricing, merchant information, availability, bestsellers rank, delivery, evaluation, and warranty.

All four APIs demonstrated the ability to parse most page elements. Oxylabs and Smartproxy provided the most comprehensive results, including discounts, delivery, and warranty information. Rayobyte's parser was less informative, excluding item variations, delivery, and warranty information. Shifter chose to exclude buy box data and experienced a few formatting errors.

In summary, this qualitative test unveiled the varying parsing capabilities of web scraping APIs, shedding light on their strengths and limitations across different types of web pages.

Pricing Models

Web scraping APIs predominantly adopt a pricing structure centered around successful requests, simplifying expense calculations. Providers typically charge based on the number of successful requests, allowing users to gauge costs easily. The standard metric for comparison is the CPM (cost per 1,000 requests).

The dominant pricing model remains the monthly subscription, but variations exist. Zyte introduces an intriguing approach where users set a monthly limit and pay half in advance each month. Notably, trials are available with most providers, with a standard offering of 5,000 requests.

Calculating Request Price

While the pricing model appears straightforward, some web scraping APIs introduce complexities in calculating a request's price.

Factors such as the target website, JavaScript rendering, residential proxies, and more contribute to price modifiers, leading to significant cost variations.

ScraperAPI stands out with a complex structure involving three tiers of proxy networks and JavaScript rendering.

The pricing varies based on factors like the use of residential proxies, headless scraping, and rates for specific websites such as Google, Amazon, and social media.

Oxylabs and Smartproxy adopt a differentiation approach, with higher costs for search engine scrapers and approximately double expenses for e-commerce scrapers.

Shifter follows a similar strategy for search engines, while its regular scraper aligns with ScraperAPI's structure.

Bright Data and Rayobyte maintain consistent pricing irrespective of whether they use custom scrapers or render JavaScript, simplifying the process of scraping challenging targets.

Zyte, on the other hand, dynamically calculates the price per request for each website, considering its difficulty, JavaScript rendering, screenshots, and browser actions. This dynamic approach makes it challenging to estimate expenses in advance.

Conclusion

The web scraping API landscape is dynamic, offering diverse features and pricing structures.

Key insights include the evolution of advanced features, the targeting of major websites like Google and Amazon, and the importance of parsing capabilities.

Performance and reliability vary, with Oxylabs, Smartproxy, and Bright Data emerging as reliable performers.

Pricing models are generally based on successful requests, but some providers introduce complexity with differentiated pricing.

Organizations should carefully assess their needs and budget constraints when choosing a web scraping API, considering factors like data output, customization, and parsing capabilities. Ongoing monitoring is essential in this competitive and evolving ecosystem.

Frequently Asked Questions

For further reading, you might be interested in the following:

• 6 Best German Proxy Providers for 2024
• 10 Best Residential Proxies Compared 2024
• How to Scrape Amazon Product Data: Extract Product Info with Ease