15,059
edits
HTTP request and response data tool: Difference between revisions
Jump to navigation
Jump to search
m
→References
mNo edit summary |
m (→References) |
||
| (15 intermediate revisions by the same user not shown) | |||
| Line 37: | Line 37: | ||
** Option {{kbd | key=<nowiki>--spider</nowiki>}} "When invoked with this option, Wget will behave as a Web spider, which means that it will not download the pages, just check that they are there." quoted from manual. | ** Option {{kbd | key=<nowiki>--spider</nowiki>}} "When invoked with this option, Wget will behave as a Web spider, which means that it will not download the pages, just check that they are there." quoted from manual. | ||
* ''$'' [https://paw.cloud/ Paw – The most advanced API tool] for {{Mac}} | |||
Example result after executed {{kbd | key=curl}} command: | |||
<pre> | <pre> | ||
$ curl -L -I https://www.google.com | $ curl -L -I https://www.google.com | ||
| Line 55: | Line 58: | ||
vary: Accept-Encoding | vary: Accept-Encoding | ||
</pre> | </pre> | ||
=== Redirect checker === | |||
* [https://wheregoes.com/ Link Checker | Redirect Checker - WhereGoes] {{Gd}} | |||
Curl command | |||
<pre> | |||
curl -sI -L "https://example.com/" | grep -i "^location:" | |||
</pre> | |||
This command '''traces the redirect chain of a URL''', printing only the destination of each redirect step. Here's the breakdown: | |||
<code>curl -sI -L "https://example.com/"</code> | |||
* '''<code>curl</code>''': a tool for sending HTTP requests | |||
* '''<code>-s</code>''' (silent): quiet mode, suppresses progress bars and other status info | |||
* '''<code>-I</code>''' (capital, <code>--head</code>): fetches only the '''HTTP headers''', not the page body — faster | |||
* '''<code>-L</code>''' (<code>--location</code>): if the server responds with a 3xx redirect status code (like 301, 302, 307), curl will '''automatically follow''' the new URL and keep requesting until it reaches the final page (or hits the redirect limit) | |||
* '''<code>"https://example.com/"</code>''': the target URL to query | |||
So this sends a HEAD request for each redirect step along the way, and prints all the response headers, which might look like: | |||
<pre> | |||
HTTP/1.1 301 Moved Permanently | |||
Location: https://www.example.com/ | |||
... | |||
HTTP/1.1 200 OK | |||
... | |||
</pre> | |||
<code>| grep -i "^location:"</code> | |||
* '''<code>|</code>''': pipes curl's output into <code>grep</code> | |||
* '''<code>grep -i</code>''': searches text, <code>-i</code> means case-insensitive (so it matches both <code>Location:</code> and <code>location:</code>) | |||
* '''<code>"^location:"</code>''': <code>^</code> means '''start of line''', so it only picks lines that '''begin with''' <code>location:</code> — that is, the new destination URL specified by the redirect header | |||
This command will: | |||
# Send a request to the target URL, fetching only headers | |||
# Automatically follow any redirects | |||
# Filter out every <code>Location:</code> line — i.e., the destination URL of '''each''' redirect step | |||
=== other tools === | === other tools === | ||
| Line 61: | Line 104: | ||
* [http://www.httpwatch.com/ HttpWatch 9.0: HTTP Sniffer] for {{IE}}, {{Fx}} and iPhone | * [http://www.httpwatch.com/ HttpWatch 9.0: HTTP Sniffer] for {{IE}}, {{Fx}} and iPhone | ||
** Steps to view the POST data: (1)start recording HTTP requests (2)click URL (3)switch the label to '''POST Data''' | ** Steps to view the POST data: (1)start recording HTTP requests (2)click URL (3)switch the label to '''POST Data''' | ||
echo | echo | ||
* [https://httpbin.org/ httpbin.org] "A simple HTTP Request & Response Service." | * [https://httpbin.org/ httpbin.org] "A simple HTTP Request & Response Service." | ||
web security | |||
* [https://tools.geekflare.com/tools/x-frame-options-test X-Frame-Options Header Checker Tool] {{access | date=2019-03-06}} | |||
* [https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project OWASP Zed Attack Proxy Project - OWASP] {{access | date=2019-03-06}} | |||
== HTTP & HTTPS Proxy == | |||
* [https://mitmproxy.org/ mitmproxy - an interactive HTTPS proxy] ([https://github.com/mitmproxy/mitmproxy/blob/main/LICENSE MIT license] {{Gd}}) on {{Win}}, {{Mac}} & {{Linux}}<ref>[https://docs.mitmproxy.org/stable/overview/installation/ Installation]</ref> | |||
* [https://portswigger.net/burp Burp Suite - Application Security Testing Software - PortSwigger] | |||
* ''$'' [https://www.charlesproxy.com/ Charles Web Debugging Proxy • HTTP Monitor / HTTP Proxy / HTTPS & SSL Proxy / Reverse Proxy] on {{Mac}} | |||
* ''$'' [https://www.telerik.com/fiddler Fiddler - Web Debugging Proxy - Telerik] on {{Win}}, {{Mac}} & {{Linux}} | |||
* ''$'' [https://proxyman.io/ Proxyman · Native, Modern Web Debugging Proxy · Inspect network from Mac, iOS, Android devices with ease] | |||
== Web page compression check == | == Web page compression check == | ||
| Line 78: | Line 131: | ||
* [http://stackoverflow.com/questions/3390547/capturing-ajax-requests html - capturing ajax requests - Stack Overflow] {{access | date = 2015-09-01}} | * [http://stackoverflow.com/questions/3390547/capturing-ajax-requests html - capturing ajax requests - Stack Overflow] {{access | date = 2015-09-01}} | ||
== | == References == | ||
<references/> | <references/> | ||
[[Category: | [[Category: Design]] | ||
[[Category:Programming]] | [[Category: Programming]] | ||
[[Category:Data collecting]] | [[Category: Data collecting]] | ||
[[Category:Security]] | [[Category: Security]] | ||
[[Category: Revised with LLMs]] | |||