Skill tree of web scraping - Revision history

Planetoid at 16:02, 28 April 2025

2025-04-28T16:02:49Z

← Older revision		Revision as of 00:02, 29 April 2025
Line 19:		Line 19:
	** submit the from after logged the account ★★★☆☆		** submit the from after logged the account ★★★☆☆
	* Detection of abnormal data		* Detection of abnormal data
	** [https://en.wikipedia.org/wiki/List_of_HTTP_status_codes HTTP status codes] ★★☆☆☆		** [https://en.wikipedia.org/wiki/List_of_HTTP_status_codes HTTP status codes] ([[Troubleshooting of HTTP errors]]) ★★☆☆☆
	** Data is wrong even the server throw HTTP 200 status code ★★★☆☆		** Data is wrong even the server throw HTTP 200 status code ★★★☆☆
	* Etiquette of web scraping		* Etiquette of web scraping

Unknown user at 07:10, 27 August 2020

2020-08-27T07:10:29Z

← Older revision		Revision as of 15:10, 27 August 2020
Line 31:		Line 31:
	* Character encoding ★☆☆☆☆		* Character encoding ★☆☆☆☆
	* [[Data cleaning]] e.g. unprintable characters ★★☆☆☆		* [[Data cleaning]] e.g. unprintable characters ★★☆☆☆
	* [~~https://en.wikipedia.org/wiki/Regular_expression~~ Regular expression] ★★★☆☆		* [[Regular expression]] ★★★☆☆
	* Selection of database engine ★★★★☆		* Selection of database engine ★★★★☆

Unknown user at 02:52, 27 August 2020

2020-08-27T02:52:31Z

← Older revision		Revision as of 10:52, 27 August 2020
Line 30:		Line 30:
	Data transforming		Data transforming
	* Character encoding ★☆☆☆☆		* Character encoding ★☆☆☆☆
	* Data cleaning e.g. unprintable characters ★★☆☆☆		* [[Data cleaning]] e.g. unprintable characters ★★☆☆☆
	* [https://en.wikipedia.org/wiki/Regular_expression Regular expression] ★★★☆☆		* [https://en.wikipedia.org/wiki/Regular_expression Regular expression] ★★★☆☆
	* Selection of database engine ★★★★☆		* Selection of database engine ★★★★☆

Unknown user at 03:16, 25 August 2020

2020-08-25T03:16:46Z

← Older revision		Revision as of 11:16, 25 August 2020
Line 38:		Line 38:
	[[Category:Data Science]]		[[Category:Data Science]]
	[[Category:Data collecting]]		[[Category:Data collecting]]
			[[Category:web scraping]]

Unknown user at 02:39, 25 August 2020

2020-08-25T02:39:04Z

← Older revision		Revision as of 10:39, 25 August 2020
Line 36:		Line 36:

	[[Category:Programming]]		[[Category:Programming]]
	[[Category:Data ~~science~~]]		[[Category:Data Science]]
	[[Category:Data collecting]]		[[Category:Data collecting]]

Unknown user at 02:20, 25 August 2020

2020-08-25T02:20:48Z

← Older revision		Revision as of 10:20, 25 August 2020
Line 6:		Line 6:

	* Understnding the web technology		* Understnding the web technology
	** HTTP GET/POST ★★☆☆☆ [Related page: [HTTP request and response data tool]]		** HTTP GET/POST ★★☆☆☆ Related page: [[HTTP request and response data tool]]
	** HTTP/CSS/Javascript ★★☆☆☆		** HTTP/CSS/Javascript ★★☆☆☆
	** CSS seletor and DOM (Document Object Model) elements ★★☆☆☆ Related page: [[Xpath tools]]		** CSS seletor, XPath expressions and DOM (Document Object Model) elements ★★☆☆☆ Related page: [[Xpath tools]]
	** AJAX (Asynchronous JavaScript and XML) ★★★★☆		** AJAX (Asynchronous JavaScript and XML) ★★★★☆
	* Using API to retrieve data ★★☆☆☆		* Using API to retrieve data ★★☆☆☆

Unknown user: Created page with "Data extraction * How they build the website & information architecture ** Understanding the navigation system e.g. pagination ★★☆☆☆ *..."

2020-08-24T09:44:57Z

Created page with "Data extraction * How they build the website & information architecture ** Understanding the navigation system e.g. pagination ★★☆☆☆ *..."

New page

Data extraction
* How they build the website & [[Information Architecture | information architecture]]
** Understanding the navigation system e.g. pagination ★★☆☆☆
*** Understanding the classfication system ★★☆☆☆
*** Parse the sitemap XML file ★★☆☆☆

* Understnding the web technology
** HTTP GET/POST ★★☆☆☆ [Related page: [HTTP request and response data tool]]
** HTTP/CSS/Javascript ★★☆☆☆
** CSS seletor and DOM (Document Object Model) elements ★★☆☆☆ Related page: [[Xpath tools]]
** AJAX (Asynchronous JavaScript and XML) ★★★★☆
* Using API to retrieve data ★★☆☆☆
* Parse remote files to retrieve data ★★☆☆☆
* Using unofficial method to retrieve data
** [https://curl.haxx.se/ curl] command or [https://www.gnu.org/software/wget/manual/wget.html GNU Wget] command ★★☆☆☆ Related page: [[Troubleshooting of curl errors]]
** [https://www.selenium.dev/ SeleniumHQ Browser Automation] ★★★☆☆
* Forum submit
** submit the from without loggin ★★☆☆☆
** submit the from after logged the account ★★★☆☆
* Detection of abnormal data
** [https://en.wikipedia.org/wiki/List_of_HTTP_status_codes HTTP status codes] ★★☆☆☆
** Data is wrong even the server throw HTTP 200 status code ★★★☆☆
* Etiquette of web scraping
** Limit of web request ★★☆☆☆
* Tom and Jerry
** VPN and proxy ★★☆☆☆
** Decode the [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA] ★★★★☆
** Decentralized web scraping ★★★★☆

Data transforming
* Character encoding ★☆☆☆☆
* Data cleaning e.g. unprintable characters ★★☆☆☆
* [https://en.wikipedia.org/wiki/Regular_expression Regular expression] ★★★☆☆
* Selection of database engine ★★★★☆

[[Category:Programming]]
[[Category:Data science]]
[[Category:Data collecting]]