Web scrape troubleshooting

Before start to writing the scirpt of web scraping (crawler)

1. Content of web page was changed (revision): The expected web content (of specified DOM element) became empty.

Multiple sources of same column such as different HTML DOM but have the same column value.
Backup the HTML text of parent DOM element
(optional) Complete HTML file backup

2. The IP was banned from server

Random Delays: Setting the temporization (sleep time) between each request e.g.: PHP: sleep - Manual, AutoThrottle extension — Scrapy 1.0.3 documentation or Sleep random seconds in programming.
The server responded with a status of 403: '403 forbidden' --> Change the network IP
Smart Retry: Automatic retry or Exponential Backoff^[1] on network errors (up to 3 times)

4. AJAX

5. The web page needed to signed in

6. Blocking the request without Referer or other headers.

7. Connection timeout during a http request. e.g. In PHP default_socket_timeout is 30 seconds^[2]^[3].

11. Is link a permanent link?

12. Enable/Disable the CSS or JavaScript

Difficulty in implementing	Descriptioin	Approach	Comments
Easy	Well-formatted HTML elements	Url is the resource of dataset.
Advanced	Interactive websites	Url is the resource of dataset. Require to simulate post form submit with the form data or user agent	Using HTTP request and response data tool or PHP: cURL
more difficult	Interactive websites	Require to simulate the user behavior on browser such as click the button, submit the form and obtain the file finally.	Using Selenium or Headless Chrome
Difficult	Interactive websites	Ajax

How to find the unofficial (3rd-party) web crawler? Search keyword strategy suggested as follows:

One frequent challenge in web scraping is dealing with overly complex webpage structures that are difficult to parse. Here's how to address this:

Solution: Find Alternative Page Versions

Look for simpler versions of the same webpage content through 1. Mobile versions of the site 2. AMP (Accelerated Mobile Pages) versions

Example:

The AMP version typically offers a more streamlined structure that's easier to parse, while containing the same core content.

↑ Exponential backoff - Wikipedia: "Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate."
↑ PHP: Runtime Configuration - Manual
↑ libcurl - Error Codes