Editing Web scrape troubleshooting (section)

== List of technical issues ==

1. Content of web page was changed (revision): The expected web content (of specified DOM element) became empty.

* Multiple sources of same column such as different HTML DOM but have the same column value.
* Backup the HTML text of parent DOM element
* (optional) Complete HTML file backup

2. The IP was banned from server

* Random Delays: Setting the temporization (sleep time) between each request e.g.: [http://php.net/manual/en/function.sleep.php PHP: sleep - Manual], [http://doc.scrapy.org/en/1.0/topics/autothrottle.html#topics-autothrottle AutoThrottle extension — Scrapy 1.0.3 documentation] or [[Sleep | Sleep random seconds in programming]].
* The server responded with a status of 403: '[https://zh.wikipedia.org/wiki/HTTP_403 403 forbidden]' --> Change the network IP
* Smart Retry: '''Automatic retry''' or '''Exponential Backoff'''<ref>[https://en.wikipedia.org/wiki/Exponential_backoff Exponential backoff - Wikipedia]: "Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate."</ref> on network errors


3. [https://en.wikipedia.org/wiki/CAPTCHA CAPTCHA]

4. AJAX

* [https://chrome.google.com/webstore/detail/autoscroll/kgkaecolmndcecnchojbndeanmiokofl/related Autoscroll] on {{Chrome}} or {{Edge}} written by [https://twitter.com/PeterLegierski Peter Legierski (@PeterLegierski) / Twitter]

5. The web page needed to signed in

6. Blocking the request without {{kbd | key= Referer}} or other headers.

7. Connection timeout during a http request. e.g. In PHP {{kbd | key=default_socket_timeout}} is 30 seconds<ref>[https://www.php.net/manual/en/filesystem.configuration.php PHP: Runtime Configuration - Manual]</ref><ref>[https://curl.haxx.se/libcurl/c/libcurl-errors.html libcurl - Error Codes]</ref>.

8. Language and [http://php.net/manual/en/function.urlencode.php URL-encodes string]

9. [[How to extract content from websites]]

10. [[Data cleaning#Data_handling | Data cleaning]] issues e.g. [https://en.wikipedia.org/wiki/Non-breaking_space Non-breaking space] or other [https://en.wikipedia.org/wiki/Whitespace_character Whitespace character]

11. Is link a permanent link?

12. Enable/Disable the CSS or JavaScript

<div class="table-responsive" style="width:100%;     min-height: .01%;    overflow-x: auto;">
<table class="wikitable sortable" style="width:100%">
  <tr>
    <th>Difficulty in implementing</th>
    <th>Descriptioin</th> 
    <th>Approach</th> 
    <th>Comments</th> 
  </tr>
  <tr>
    <td>Easy</td>
    <td>Well-formatted HTML elements</td> 
    <td>Url is the resource of dataset.</td> 
    <td></td>
  </tr>
  <tr>
    <td>Advanced</td>
    <td>Interactive websites</td>
    <td>Url is the resource of dataset. Require to simulate post form submit with the form data or [[User agent|user agent]]</td> 
    <td>Using [[HTTP request and response data tool]] or [http://php.net/curl PHP: cURL]</td>
  </tr>
  <tr>
    <td>more difficult</td>
    <td>Interactive websites</td>
    <td>Require to simulate the user behavior on browser such as click the button, submit the form and obtain the file finally.</td> 
    <td>Using [https://www.seleniumhq.org/ Selenium] or [https://developers.google.com/web/updates/2017/04/headless-chrome Headless Chrome]</td>
  </tr>
  <tr>
    <td>Difficult</td>
    <td>Interactive websites</td>
    <td>Ajax</td> 
    <td></td>
  </tr>
</table>
</div>



=== Search keyword strategy ===
'''How to find the unofficial (3rd-party) web crawler?''' Search keyword strategy suggested as follows:
* ''target website'' + crawler site:github.com
* ''target website'' + scraper site:github.com
* ''target website'' + bot site:github.com
* ''target website'' + download / downloader site:github.com
* ''target website'' + browser client site:github.com