Optimizing Python Data Scraping Scripts Using Wenxin Yiyan API Integration

Integrating the Wenxin Yiyan (ERNIE Bot) API into Python scraping pipelines is the best way to solve the "dirty data" problem without writing a thousand fragile regex patterns. Most people use AI for generating the initial scraper code, but the real productivity gain happens when you use the LLM as a post-processing layer to structure unstructured HTML fragments into clean JSON.

I've been using Cursor to build this workflow, and the trick is to stop asking the AI to "write a scraper" and instead ask it to "build a cleaning pipeline."

The biggest gotcha with scraping is that website layouts change. If you hardcode your selectors, your script breaks. I've shifted to a hybrid approach: use httpx and BeautifulSoup to grab the raw text or the specific container, then feed that messy string into Wenxin Yiyan to extract the actual entities.

Here is the core logic I use for the API integration. I wrapped it in a helper function to handle the authentication and prompt engineering in one go:

import httpx
import json

def extract_structured_data(raw_html_text):
    url = "https://aip.baidubce.com/rpc/2.0/ai_custom/v1/wenxinworkshop/chat"
    access_token = "YOUR_ACCESS_TOKEN"
    
    # The prompt is the most critical part. Be explicit about the JSON schema.
    prompt = (
        f"Extract the product name, price, and availability from this text. "
        f"Return ONLY a valid JSON object. Text: {raw_html_text}"
    )
    
    payload = {
        "question": prompt,
        "chain_of_thought": "disable" # Disable to save tokens and get faster responses
    }
    
    headers = {"Content-Type": "application/json"}
    response = httpx.post(f"{url}?access_token={access_token}", json=payload, headers=headers)
    
    # Clean the response because LLMs sometimes wrap JSON in markdown code blocks
    result = response.json().get("result", "")
    return result.replace("

json", "").replace("

", "").strip()

To make this actually fast, you can't do synchronous requests. If you have 1,000 pages, calling the API one by one will take forever. I used Cursor's Composer mode to refactor my script into asyncio with httpx.AsyncClient. This cut my processing time from 20 minutes down to about 3 minutes.

Key config tips for productivity:

• Prompt Versioning: Don't hardcode prompts in your functions. Move them to a .env or a prompts.yaml file. When the API response starts drifting, you can tweak the prompt without touching the logic.
• Token Management: Wenxin Yiyan has specific token limits. Use BeautifulSoup to strip out <script>, <style>, and <nav> tags before sending the HTML to the API. Sending a full HTML page is a waste of money and often confuses the model.
• Error Handling: LLMs occasionally hallucinate a comma or a bracket, breaking json.loads(). I always wrap the parsing in a try-except block and have a fallback that logs the raw string for manual review.

The productivity jump here is massive. Instead of spending three hours debugging why a CSS selector changed from .price-tag to .product-price, I just tell the API to "find the price," and it works regardless of the HTML structure. This transforms scraping from a maintenance nightmare into a data engineering task.