Web scrape troubleshooting

list of technical issues

1. Content of web page was changed (revision): Th expected web content (of specified DOM element) became empty.
   • Multiple sources of same column such as different HTML DOM but have the same column value.
   • Backup the HTML text of parent DOM element
   • (optional) Complete HTML file backup
2. Server ip ban
   • setting the temporization (sleep time) between pages ex: PHP: sleep - Manual, AutoThrottle extension — Scrapy 1.0.3 documentation
3. CATCHA
4. AJAX

Further reading

• Stateless: Why say that HTTP is a stateless protocol? - Stack Overflow
• Stateful: What is stateful? Webopedia Definition