Web scrape troubleshooting: Difference between revisions

Revision as of 21:17, 11 May 2019

list of technical issues

Content of web page was changed (revision): The expected web content (of specified DOM element) became empty.
- Multiple sources of same column such as different HTML DOM but have the same column value.
- Backup the HTML text of parent DOM element
- (optional) Complete HTML file backup
The IP was banned from server
- Setting the temporization (sleep time) between each request e.g.: PHP: sleep - Manual, AutoThrottle extension — Scrapy 1.0.3 documentation or Sleep random seconds in programming.
- The server responded with a status of 403: '403 forbidden' --> Change the network IP
CATCHA
AJAX
The web page needed to signed in
Blocking the request without Referer or other headers.
Connection timeout during a http request. e.g. In PHP default_socket_timeout is 30 seconds^[1]^[2].
Language and URL-encodes string
Data cleaning issues e.g. Non-breaking space or other Whitespace character

Difficulty in implementing	Approach	Comments
easy	Url is the resource of dataset
more difficult	Url is the resource of dataset. Require to simulate post form submit with the form data or user agent	Using HTTP request and response data tool or PHP: cURL
more difficult	Require to simulate the user behavior on browser such as click the button, submit the form and obtain the file finally.	Using Selenium or Headless Chrome
difficult	Ajax

Further reading

References

Troubleshooting of ...

Template

@@ Line 11: / Line 11: @@
 # The web page needed to signed in
 # Blocking the request without {{kbd | key= Referer}} or other headers.
-# Connection timeout during a http request. e.g. In PHP {{kbd | key=default_socket_timeout}} is 30 seconds<ref>[https://www.php.net/manual/en/filesystem.configuration.php PHP: Runtime Configuration - Manual]</ref>.
+# Connection timeout during a http request. e.g. In PHP {{kbd | key=default_socket_timeout}} is 30 seconds<ref>[https://www.php.net/manual/en/filesystem.configuration.php PHP: Runtime Configuration - Manual]</ref><ref>[https://curl.haxx.se/libcurl/c/libcurl-errors.html libcurl - Error Codes]</ref>.
 # Language and [http://php.net/manual/en/function.urlencode.php URL-encodes string]
 # [[Data cleaning#Data_handling | Data cleaning]] issues e.g. [https://en.wikipedia.org/wiki/Non-breaking_space Non-breaking space] or other [https://en.wikipedia.org/wiki/Whitespace_character Whitespace character]