Web scrape troubleshooting: Difference between revisions

Jump to navigation Jump to search

Web scrape troubleshooting (edit)

Revision as of 17:44, 24 August 2020

1,769 bytes removed ,  24 August 2020
m
→‎Skill tree of web scraping
Line 53: Line 53:
== Skill tree of web scraping ==
== Skill tree of web scraping ==


Data extraction
[[Skill tree of web scraping]]
* How they build the website & [[Information Architecture | information architecture]]
** Understanding the navigation system ★★☆☆☆
*** Understanding the classfication system ★★☆☆☆
*** Parse the sitemap XML file ★★☆☆☆
 
* Understnding the web technology
** HTTP GET/POST ★★☆☆☆ [Related page: [HTTP request and response data tool]]
** HTTP/CSS/Javascript ★★☆☆☆
** CSS seletor and DOM (Document Object Model) elements ★★☆☆☆ Related page: [[Xpath tools]]
** AJAX (Asynchronous JavaScript and XML) ★★★★☆
* Using API to retrieve data ★★☆☆☆
* Parse remote files to retrieve data ★★☆☆☆
* Using unofficial method to retrieve data
** [https://curl.haxx.se/ curl] command or [https://www.gnu.org/software/wget/manual/wget.html GNU Wget] command ★★☆☆☆ Related page: [[Troubleshooting of curl errors]]
** [https://www.selenium.dev/ SeleniumHQ Browser Automation] ★★★☆☆
* Forum submit
** submit the from without loggin ★★☆☆☆
** submit the from after logged the account ★★★☆☆
* Detection of abnormal data
** [https://en.wikipedia.org/wiki/List_of_HTTP_status_codes HTTP status codes] ★★☆☆☆
** Data is wrong even the server throw HTTP 200 status code ★★★☆☆
* Etiquette of web scraping
** Limit of web request ★★☆☆☆
* Tom and Jerry
** VPN and proxy ★★☆☆☆
** Decode the [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA] ★★★★☆
** Decentralized web scraping ★★★★☆
 
Data transforming
* Character encoding ★☆☆☆☆
* Data cleaning e.g. unprintable characters ★★☆☆☆
* [https://en.wikipedia.org/wiki/Regular_expression Regular expression] ★★★☆☆
* Selection of database engine ★★★★☆


== Further reading ==
== Further reading ==
Anonymous user
Retrieved from "https://wiki.planetoid.info/index.php/Special:MobileDiff/21992"

Navigation menu