Web scrape troubleshooting: Difference between revisions

(3 intermediate revisions by the same user not shown)

Line 15:

2. The IP was banned from server

* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep | Sleep random seconds in programming]].

* Random Delays: Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep | Sleep random seconds in programming]].

* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP

* Smart Retry: '''Automatic retry''' or '''Exponential Backoff'''<ref>[https://en.wikipedia.org/wiki/Exponential_backoff Exponential backoff - Wikipedia]: "Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate."</ref> on network errors

3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]

Line 84:

Line 86:

* ''target website'' + download / downloader site:github.com

* ''target website'' + browser client site:github.com

== Common Web Scraping Issues and Solutions ==

=== Complex Webpage Structure ===

One frequent challenge in web scraping is dealing with overly complex webpage structures that are difficult to parse. Here's how to address this:

'''Solution: Find Alternative Page Versions'''

Look for simpler versions of the same webpage content through 1. Mobile versions of the site 2. AMP (Accelerated Mobile Pages) versions

'''Example:'''

* Standard webpage: `https://www.ettoday.net/news/20250107/2888050.htm`

* AMP version: `https://www.ettoday.net/amp/amp_news.php7?news_id=2888050&ref=mw&from=google.com`

The AMP version typically offers a more streamlined structure that's easier to parse, while containing the same core content.

== Further reading ==

@@ Line 15: / Line 15: @@
 . The IP was banned from server
-* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep | Sleep random seconds in programming]].
+* Random Delays: Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep | Sleep random seconds in programming]].
 * The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP
+* Smart Retry: '''Automatic retry''' or '''Exponential Backoff'''<ref>[https://en.wikipedia.org/wiki/Exponential_backoff Exponential backoff - Wikipedia]: "Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate."</ref> on network errors
 . [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]
@@ Line 84: / Line 86: @@
 * ''target website'' + download / downloader site:github.com
 * ''target website'' + browser client site:github.com
+== Common Web Scraping Issues and Solutions ==
+=== Complex Webpage Structure ===
+One frequent challenge in web scraping is dealing with overly complex webpage structures that are difficult to parse. Here's how to address this:
+'''Solution: Find Alternative Page Versions'''
+Look for simpler versions of the same webpage content through 1. Mobile versions of the site 2. AMP (Accelerated Mobile Pages) versions
+'''Example:'''
+* Standard webpage: `https://www.ettoday.net/news/20250107/2888050.htm`
+* AMP version: `https://www.ettoday.net/amp/amp_news.php7?news_id=2888050&ref=mw&from=google.com`
+The AMP version typically offers a more streamlined structure that's easier to parse, while containing the same core content.
 == Further reading ==