Editing
How to extract content from websites
(section)
Jump to navigation
Jump to search
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
== Methods for article extraction == === Open source solution for article extraction === {{Gd}} [https://github.com/mozilla/readability mozilla/readability: A standalone version of the readability lib]<ref>[https://videoinu.com/blog/firefox-reader-view-heuristics/ How does Firefox's Reader View work?]</ref> * Demo: * Pricing / Free Limit: free * Requirement: Node.js * License: Apache License, Version 2.0 {{Gd}} * Source code of client: Available on GitHub & Container: [https://hub.docker.com/r/phpdockerio/readability-js-server phpdockerio/readability-js-server - Docker Image | Docker Hub] {{Gd}} [https://github.com/postlight/parser postlight/parser: 📜 Extract meaningful content from the chaos of a web page] (Replacement of [https://github.com/postlight/mercury-parser postlight/mercury-parser]) * Demo: * Requirement: Node.js * License: Apache License, Version 2.0 or MIT license {{Gd}} * Container [https://github.com/timothytylee/full-text-rss timothytylee/full-text-rss: Fork of Full-Text RSS to improve handling of non UTF-8 sites] * Demo: [https://www.fivefilters.org/full-text-rss/ Full-Text RSS - FiveFilters.org] * Requirement: PHP * License: GNU Affero General Public License v3.0 {{Gd}} * Container [https://github.com/luin/readability luin/readability: 📚 Turn any web page into a clean view] * Demo: * Requirement: Node.js * License: Apache License 2.0 {{Gd}} * Container [https://github.com/adbar/trafilatura adbar/trafilatura: Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)] + [https://github.com/mozilla/readability mozilla/readability: A standalone version of the readability lib] * Demo: * Requirement: * License: * Container [https://totheweb.com/learning_center/tools-convert-html-text-to-plain-text-for-content-review/ Free Tool: Convert Your Webpage to Plain Text » ToTheWeb] * Demo: * Requirement: * License: GPL + Apache License Version 2.0 {{Gd}} * Container [https://github.com/crscheid/php-article-extractor crscheid/php-article-extractor: A PHP library to extract article text from web pages] * Pricing / Free Limit: free * Source code of client: Available on GitHub * License: === Commercial solution for article extraction === ''$'' [https://www.diffbot.com/products/extract/ Diffbot | Extract Content From Websites Automatically] two weeks free trial * Demo: * Pricing / Free Limit: 14 days free trial * Requirement: * License: * Source code of client: [https://www.diffbot.com/dev/docs/libraries/ Diffbot Libraries - Diffbot] ''$'' [https://www.fivefilters.org/full-text-rss/ Full-Text RSS - FiveFilters.org] * Demo: [http://ftr.fivefilters.org/ Full-Text RSS Feeds | from fivefilters.org] * Requirement: PHP * License: * Source code of client: [https://getpocket.com/developer/docs/v3/article-view Pocket Developer Program: Pocket API: Article View] not available {{exclaim}} * Pricing / Free Limit: * Source code of client: n/a * License: ''$'' [https://newsblur.com/ NewsBlur] > [https://newsblur.com/api The NewsBlur API] > GET /rss_feeds/original_story * Pricing: * License: * Source code of client: ''$'' [https://feedbin.com/ Feedbin] > API > [https://github.com/feedbin/feedbin-api/blob/master/content/extract-full-content.md Extracting Full Content] * Pricing: * License: * Source code of client:
Summary:
Please note that all contributions to LemonWiki共筆 are considered to be released under the Creative Commons Attribution-NonCommercial-ShareAlike (see
LemonWiki:Copyrights
for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource.
Do not submit copyrighted work without permission!
Cancel
Editing help
(opens in new window)
Navigation menu
Personal tools
Not logged in
Talk
Contributions
Log in
Namespaces
Page
Discussion
English
Views
Read
Edit
View history
More
Search
Navigation
Main page
Current events
Recent changes
Random page
Help
Categories
Tools
What links here
Related changes
Special pages
Page information