Web scrape troubleshooting: Difference between revisions

Jump to navigation Jump to search

Web scrape troubleshooting (edit)

Revision as of 06:23, 23 December 2025

1,417 bytes added ,  23 December 2025
m
→‎List of technical issues
Tags: Mobile edit Mobile web edit
 
(7 intermediate revisions by the same user not shown)
Line 7: Line 7:
== List of technical issues ==
== List of technical issues ==


# Content of web page was changed (revision): The expected web content (of specified DOM element) became empty.
1. Content of web page was changed (revision): The expected web content (of specified DOM element) became empty.
#* Multiple sources of same column such as different HTML DOM but have the same column value.
#* Backup the HTML text of parent DOM element
#* (optional) Complete HTML file backup


# The IP was banned from server
* Multiple sources of same column such as different HTML DOM but have the same column value.
#* Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep | Sleep random seconds in programming]].
* Backup the HTML text of parent DOM element
#* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP
* (optional) Complete HTML file backup


# [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]
2. The IP was banned from server


# AJAX
* Random Delays: Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep | Sleep random seconds in programming]].
#* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter]
* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP
* Smart Retry: '''Automatic retry''' or '''Exponential Backoff'''<ref>[https://en.wikipedia.org/wiki/Exponential_backoff Exponential backoff - Wikipedia]: "Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate."</ref> on network errors


# The web page needed to signed in


# Blocking the request without {{kbd | key= Referer}} or other headers.
3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]


# Connection timeout during a http request. e.g. In PHP {{kbd | key=default_socket_timeout}} is 30 seconds<ref>[https://www.php.net/manual/en/filesystem.configuration.php PHP: Runtime Configuration - Manual]</ref><ref>[https://curl.haxx.se/libcurl/c/libcurl-errors.html libcurl - Error Codes]</ref>.
4. AJAX


# Language and [http://php.net/manual/en/function.urlencode.php URL-encodes string]
* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter]


# [[How to extract content from websites]]
5. The web page needed to signed in


# [[Data cleaning#Data_handling | Data cleaning]] issues e.g. [https://en.wikipedia.org/wiki/Non-breaking_space Non-breaking space] or other [https://en.wikipedia.org/wiki/Whitespace_character Whitespace character]
6. Blocking the request without {{kbd | key= Referer}} or other headers.


# Is link a permanent link?
7. Connection timeout during a http request. e.g. In PHP {{kbd | key=default_socket_timeout}} is 30 seconds<ref>[https://www.php.net/manual/en/filesystem.configuration.php PHP: Runtime Configuration - Manual]</ref><ref>[https://curl.haxx.se/libcurl/c/libcurl-errors.html libcurl - Error Codes]</ref>.


# Enable/Disable the CSS or JavaScript
8. Language and [http://php.net/manual/en/function.urlencode.php URL-encodes string]
 
9. [[How to extract content from websites]]
 
10. [[Data cleaning#Data_handling | Data cleaning]] issues e.g. [https://en.wikipedia.org/wiki/Non-breaking_space Non-breaking space] or other [https://en.wikipedia.org/wiki/Whitespace_character Whitespace character]
 
11. Is link a permanent link?
 
12. Enable/Disable the CSS or JavaScript


<div class="table-responsive" style="width:100%;    min-height: .01%;    overflow-x: auto;">
<div class="table-responsive" style="width:100%;    min-height: .01%;    overflow-x: auto;">
Line 81: Line 86:
* ''target website'' + download / downloader site:github.com
* ''target website'' + download / downloader site:github.com
* ''target website'' + browser client site:github.com
* ''target website'' + browser client site:github.com
== Common Web Scraping Issues and Solutions ==
=== Complex Webpage Structure ===
One frequent challenge in web scraping is dealing with overly complex webpage structures that are difficult to parse. Here's how to address this:
'''Solution: Find Alternative Page Versions'''
Look for simpler versions of the same webpage content through 1. Mobile versions of the site 2. AMP (Accelerated Mobile Pages) versions
'''Example:'''
* Standard webpage: `https://www.ettoday.net/news/20250107/2888050.htm`
* AMP version: `https://www.ettoday.net/amp/amp_news.php7?news_id=2888050&ref=mw&from=google.com`
The AMP version typically offers a more streamlined structure that's easier to parse, while containing the same core content.


== Further reading ==
== Further reading ==
Line 88: Line 108:
* [[Skill tree of web scraping]]
* [[Skill tree of web scraping]]
* [https://blog.gslin.org/archives/2022/08/22/10850/%e5%8d%97%e9%9f%93%e6%9c%80%e9%ab%98%e6%b3%95%e9%99%a2%e4%b9%9f%e5%b0%8d-web-scraping-%e7%b5%a6%e5%87%ba%e4%ba%86%e9%a1%9e%e4%bc%bc%e7%be%8e%e5%9c%8b%e7%9a%84%e5%88%a4%e4%be%8b/ 南韓最高法院也對 Web Scraping 給出了類似美國的判例 – Gea-Suan Lin's BLOG]
* [https://blog.gslin.org/archives/2022/08/22/10850/%e5%8d%97%e9%9f%93%e6%9c%80%e9%ab%98%e6%b3%95%e9%99%a2%e4%b9%9f%e5%b0%8d-web-scraping-%e7%b5%a6%e5%87%ba%e4%ba%86%e9%a1%9e%e4%bc%bc%e7%be%8e%e5%9c%8b%e7%9a%84%e5%88%a4%e4%be%8b/ 南韓最高法院也對 Web Scraping 給出了類似美國的判例 – Gea-Suan Lin's BLOG]
* [https://www.yzu.edu.tw/library/index.php/tw/news-tw/943-20181226-1 元智大學 Yuan Ze University - 圖書館 - 疑似侵權，你不可不知的 小撇步]
* [http://www.naipo.com/Portals/1/web_tw/Knowledge_Center/Infringement_Case/IPNC_170125_0501.htm 北美智權報 第177期：大數據與著作權之合理使用]


== References ==
== References ==
Retrieved from "https://wiki.planetoid.info/index.php/Special:MobileDiff/24309...26129"

Navigation menu