Web scrape troubleshooting - Revision history

Planetoid: /* List of technical issues */

2025-12-22T22:23:23Z

List of technical issues

← Older revision		Revision as of 06:23, 23 December 2025
Line 17:		Line 17:
	* Random Delays: Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].		* Random Delays: Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].
	* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP		* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP
	* Smart Retry: '''Automatic retry''' or '''Exponential Backoff'''<ref>[https://en.wikipedia.org/wiki/Exponential_backoff Exponential backoff - Wikipedia]: "Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate."</ref> on network errors ~~(up to 3 times)~~		* Smart Retry: '''Automatic retry''' or '''Exponential Backoff'''<ref>[https://en.wikipedia.org/wiki/Exponential_backoff Exponential backoff - Wikipedia]: "Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate."</ref> on network errors

Planetoid: /* List of technical issues */

2025-12-22T10:22:18Z

List of technical issues

← Older revision		Revision as of 18:22, 22 December 2025
Line 15:		Line 15:
	2. The IP was banned from server		2. The IP was banned from server

	* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].		* Random Delays: Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].
	* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP		* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP
			* Smart Retry: '''Automatic retry''' or '''Exponential Backoff'''<ref>[https://en.wikipedia.org/wiki/Exponential_backoff Exponential backoff - Wikipedia]: "Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate."</ref> on network errors (up to 3 times)


	3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]		3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]
Line 84:		Line 86:
	* ''target website'' + download / downloader site:github.com		* ''target website'' + download / downloader site:github.com
	* ''target website'' + browser client site:github.com		* ''target website'' + browser client site:github.com


	== Common Web Scraping Issues and Solutions ==		== Common Web Scraping Issues and Solutions ==

Planetoid: /* Complex Webpage Structure */

2025-01-10T03:40:47Z

Complex Webpage Structure

← Older revision		Revision as of 11:40, 10 January 2025
Line 92:		Line 92:

	'''Solution: Find Alternative Page Versions'''		'''Solution: Find Alternative Page Versions'''
	~~Look for simpler versions of the same webpage content through:~~

	1. Mobile versions of the site		Look for simpler versions of the same webpage content through 1. Mobile versions of the site 2. AMP (Accelerated Mobile Pages) versions
	2. AMP (Accelerated Mobile Pages) versions

	'''Example:'''		'''Example:'''
Line 102:		Line 100:

	The AMP version typically offers a more streamlined structure that's easier to parse, while containing the same core content.		The AMP version typically offers a more streamlined structure that's easier to parse, while containing the same core content.


	== Further reading ==		== Further reading ==

Planetoid: /* Further reading */

2025-01-10T03:38:34Z

Further reading

@@ Line 84: / Line 84: @@
 * ''target website'' + download / downloader site:github.com
 * ''target website'' + browser client site:github.com
 == Further reading ==

Planetoid: /* List of technical issues */

2025-01-10T03:33:34Z

List of technical issues

← Older revision		Revision as of 11:33, 10 January 2025
Line 8:		Line 8:

	1. Content of web page was changed (revision): The expected web content (of specified DOM element) became empty.		1. Content of web page was changed (revision): The expected web content (of specified DOM element) became empty.
	#* Multiple sources of same column such as different HTML DOM but have the same column value.
	#* Backup the HTML text of parent DOM element		* Multiple sources of same column such as different HTML DOM but have the same column value.
	#* (optional) Complete HTML file backup		* Backup the HTML text of parent DOM element
			* (optional) Complete HTML file backup

	2. The IP was banned from server		2. The IP was banned from server
	#* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].
	#* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP		* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].
			* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP

	3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]		3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]

	4. AJAX		4. AJAX
	#* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter]
			* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter]

	5. The web page needed to signed in		5. The web page needed to signed in

Planetoid: /* List of technical issues */

2025-01-10T03:32:52Z

List of technical issues

Planetoid: /* Further reading */

2024-11-26T02:56:59Z

Further reading

← Older revision		Revision as of 10:56, 26 November 2024
Line 89:		Line 89:
	* [https://blog.gslin.org/archives/2022/08/22/10850/%e5%8d%97%e9%9f%93%e6%9c%80%e9%ab%98%e6%b3%95%e9%99%a2%e4%b9%9f%e5%b0%8d-web-scraping-%e7%b5%a6%e5%87%ba%e4%ba%86%e9%a1%9e%e4%bc%bc%e7%be%8e%e5%9c%8b%e7%9a%84%e5%88%a4%e4%be%8b/ 南韓最高法院也對 Web Scraping 給出了類似美國的判例 – Gea-Suan Lin's BLOG]		* [https://blog.gslin.org/archives/2022/08/22/10850/%e5%8d%97%e9%9f%93%e6%9c%80%e9%ab%98%e6%b3%95%e9%99%a2%e4%b9%9f%e5%b0%8d-web-scraping-%e7%b5%a6%e5%87%ba%e4%ba%86%e9%a1%9e%e4%bc%bc%e7%be%8e%e5%9c%8b%e7%9a%84%e5%88%a4%e4%be%8b/ 南韓最高法院也對 Web Scraping 給出了類似美國的判例 – Gea-Suan Lin's BLOG]
	* [https://www.yzu.edu.tw/library/index.php/tw/news-tw/943-20181226-1 元智大學 Yuan Ze University - 圖書館 - 疑似侵權，你不可不知的小撇步]		* [https://www.yzu.edu.tw/library/index.php/tw/news-tw/943-20181226-1 元智大學 Yuan Ze University - 圖書館 - 疑似侵權，你不可不知的小撇步]
			* [http://www.naipo.com/Portals/1/web_tw/Knowledge_Center/Infringement_Case/IPNC_170125_0501.htm 北美智權報第177期：大數據與著作權之合理使用]

	== References ==		== References ==

Planetoid: /* Further reading */

2024-06-09T07:25:04Z

Further reading

← Older revision		Revision as of 15:25, 9 June 2024
Line 88:		Line 88:
	* [[Skill tree of web scraping]]		* [[Skill tree of web scraping]]
	* [https://blog.gslin.org/archives/2022/08/22/10850/%e5%8d%97%e9%9f%93%e6%9c%80%e9%ab%98%e6%b3%95%e9%99%a2%e4%b9%9f%e5%b0%8d-web-scraping-%e7%b5%a6%e5%87%ba%e4%ba%86%e9%a1%9e%e4%bc%bc%e7%be%8e%e5%9c%8b%e7%9a%84%e5%88%a4%e4%be%8b/ 南韓最高法院也對 Web Scraping 給出了類似美國的判例 – Gea-Suan Lin's BLOG]		* [https://blog.gslin.org/archives/2022/08/22/10850/%e5%8d%97%e9%9f%93%e6%9c%80%e9%ab%98%e6%b3%95%e9%99%a2%e4%b9%9f%e5%b0%8d-web-scraping-%e7%b5%a6%e5%87%ba%e4%ba%86%e9%a1%9e%e4%bc%bc%e7%be%8e%e5%9c%8b%e7%9a%84%e5%88%a4%e4%be%8b/ 南韓最高法院也對 Web Scraping 給出了類似美國的判例 – Gea-Suan Lin's BLOG]
			* [https://www.yzu.edu.tw/library/index.php/tw/news-tw/943-20181226-1 元智大學 Yuan Ze University - 圖書館 - 疑似侵權，你不可不知的小撇步]

	== References ==		== References ==

Unknown user: /* List of technical issues */

2024-01-16T10:03:31Z

List of technical issues

← Older revision		Revision as of 18:03, 16 January 2024
Line 11:		Line 11:
	#* Backup the HTML text of parent DOM element		#* Backup the HTML text of parent DOM element
	#* (optional) Complete HTML file backup		#* (optional) Complete HTML file backup

	# The IP was banned from server		# The IP was banned from server
	#* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].		#* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].
	#* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP		#* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP

	# [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]		# [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]

	# AJAX		# AJAX
	#* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter]		#* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter]
Line 26:		Line 29:
	# Language and [http://php.net/manual/en/function.urlencode.php URL-encodes string]		# Language and [http://php.net/manual/en/function.urlencode.php URL-encodes string]

	# [[~~Extract article text~~ from ~~web page~~]]		# [[How to extract content from websites]]

	# [[Data cleaning#Data_handling \| Data cleaning]] issues e.g. [https://en.wikipedia.org/wiki/Non-breaking_space Non-breaking space] or other [https://en.wikipedia.org/wiki/Whitespace_character Whitespace character]		# [[Data cleaning#Data_handling \| Data cleaning]] issues e.g. [https://en.wikipedia.org/wiki/Non-breaking_space Non-breaking space] or other [https://en.wikipedia.org/wiki/Whitespace_character Whitespace character]

Unknown user: /* Further reading */

2022-10-09T05:23:05Z

Further reading

← Older revision		Revision as of 13:23, 9 October 2022
Line 84:		Line 84:
	* [https://en.wikipedia.org/wiki/List_of_HTTP_status_codes List of HTTP status codes - Wikipedia]		* [https://en.wikipedia.org/wiki/List_of_HTTP_status_codes List of HTTP status codes - Wikipedia]
	* [[Skill tree of web scraping]]		* [[Skill tree of web scraping]]
			* [https://blog.gslin.org/archives/2022/08/22/10850/%e5%8d%97%e9%9f%93%e6%9c%80%e9%ab%98%e6%b3%95%e9%99%a2%e4%b9%9f%e5%b0%8d-web-scraping-%e7%b5%a6%e5%87%ba%e4%ba%86%e9%a1%9e%e4%bc%bc%e7%be%8e%e5%9c%8b%e7%9a%84%e5%88%a4%e4%be%8b/ 南韓最高法院也對 Web Scraping 給出了類似美國的判例 – Gea-Suan Lin's BLOG]

	== References ==		== References ==

← Older revision		Revision as of 11:32, 10 January 2025
Line 7:		Line 7:
	== List of technical issues ==		== List of technical issues ==

	# Content of web page was changed (revision): The expected web content (of specified DOM element) became empty.		1. Content of web page was changed (revision): The expected web content (of specified DOM element) became empty.
	#* Multiple sources of same column such as different HTML DOM but have the same column value.		#* Multiple sources of same column such as different HTML DOM but have the same column value.
	#* Backup the HTML text of parent DOM element		#* Backup the HTML text of parent DOM element
	#* (optional) Complete HTML file backup		#* (optional) Complete HTML file backup

	# The IP was banned from server		2. The IP was banned from server
	#* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].		#* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep \| Sleep random seconds in programming]].
	#* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP		#* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP

	# [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]		3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]

	# AJAX		4. AJAX
	#* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter]		#* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter]

	# The web page needed to signed in		5. The web page needed to signed in

	# Blocking the request without {{kbd \| key= Referer}} or other headers.		6. Blocking the request without {{kbd \| key= Referer}} or other headers.

	# Connection timeout during a http request. e.g. In PHP {{kbd \| key=default_socket_timeout}} is 30 seconds<ref>[https://www.php.net/manual/en/filesystem.configuration.php PHP: Runtime Configuration - Manual]</ref><ref>[https://curl.haxx.se/libcurl/c/libcurl-errors.html libcurl - Error Codes]</ref>.		7. Connection timeout during a http request. e.g. In PHP {{kbd \| key=default_socket_timeout}} is 30 seconds<ref>[https://www.php.net/manual/en/filesystem.configuration.php PHP: Runtime Configuration - Manual]</ref><ref>[https://curl.haxx.se/libcurl/c/libcurl-errors.html libcurl - Error Codes]</ref>.

	# Language and [http://php.net/manual/en/function.urlencode.php URL-encodes string]		8. Language and [http://php.net/manual/en/function.urlencode.php URL-encodes string]

	# [[How to extract content from websites]]		9. [[How to extract content from websites]]

	# [[Data cleaning#Data_handling \| Data cleaning]] issues e.g. [https://en.wikipedia.org/wiki/Non-breaking_space Non-breaking space] or other [https://en.wikipedia.org/wiki/Whitespace_character Whitespace character]		10. [[Data cleaning#Data_handling \| Data cleaning]] issues e.g. [https://en.wikipedia.org/wiki/Non-breaking_space Non-breaking space] or other [https://en.wikipedia.org/wiki/Whitespace_character Whitespace character]

	# Is link a permanent link?		11. Is link a permanent link?

	# Enable/Disable the CSS or JavaScript		12. Enable/Disable the CSS or JavaScript

	<div class="table-responsive" style="width:100%; min-height: .01%; overflow-x: auto;">		<div class="table-responsive" style="width:100%; min-height: .01%; overflow-x: auto;">