15,047
edits
Web scrape troubleshooting: Difference between revisions
Jump to navigation
Jump to search
m
→List of technical issues
Tags: Mobile edit Mobile web edit |
|||
| (4 intermediate revisions by the same user not shown) | |||
| Line 8: | Line 8: | ||
1. Content of web page was changed (revision): The expected web content (of specified DOM element) became empty. | 1. Content of web page was changed (revision): The expected web content (of specified DOM element) became empty. | ||
* Multiple sources of same column such as different HTML DOM but have the same column value. | |||
* Backup the HTML text of parent DOM element | |||
* (optional) Complete HTML file backup | |||
2. The IP was banned from server | 2. The IP was banned from server | ||
* Random Delays: Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep | Sleep random seconds in programming]]. | |||
* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP | |||
* Smart Retry: '''Automatic retry''' or '''Exponential Backoff'''<ref>[https://en.wikipedia.org/wiki/Exponential_backoff Exponential backoff - Wikipedia]: "Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate."</ref> on network errors | |||
3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA] | 3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA] | ||
4. AJAX | 4. AJAX | ||
* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter] | |||
5. The web page needed to signed in | 5. The web page needed to signed in | ||
| Line 81: | Line 86: | ||
* ''target website'' + download / downloader site:github.com | * ''target website'' + download / downloader site:github.com | ||
* ''target website'' + browser client site:github.com | * ''target website'' + browser client site:github.com | ||
== Common Web Scraping Issues and Solutions == | |||
=== Complex Webpage Structure === | |||
One frequent challenge in web scraping is dealing with overly complex webpage structures that are difficult to parse. Here's how to address this: | |||
'''Solution: Find Alternative Page Versions''' | |||
Look for simpler versions of the same webpage content through 1. Mobile versions of the site 2. AMP (Accelerated Mobile Pages) versions | |||
'''Example:''' | |||
* Standard webpage: `https://www.ettoday.net/news/20250107/2888050.htm` | |||
* AMP version: `https://www.ettoday.net/amp/amp_news.php7?news_id=2888050&ref=mw&from=google.com` | |||
The AMP version typically offers a more streamlined structure that's easier to parse, while containing the same core content. | |||
== Further reading == | == Further reading == | ||