Troubleshooting of curl errors: Difference between revisions
(23 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
[https://curl.haxx.se/ cUrl] "is used in command lines or scripts to transfer data."[https://curl.haxx.se/] | [https://curl.haxx.se/ cUrl] "is used in command lines or scripts to transfer data."[https://curl.haxx.se/] (alternative library to achive sampe purpose: (1) PHP naive [https://www.php.net/manual/en/book.curl.php Client URL Library] (2) [https://packagist.org/packages/curl/curl cURL class for PHP] (3) [https://www.oracle.com/webfolder/technetwork/tutorials/obe/cloud/objectstorage/installing_cURL/installing_cURL_on_Cygwin_on_Windows.html Installing cURL on Cygwin on Windows]) | ||
{{LanguageSwitcher | content = [[Troubleshooting of curl errors | EN]], [[Troubleshooting of curl errors in Mandarin | 漢字]] }} | |||
: [[Image:Owl icon.jpg]] How to debug the cUrl command: (1) Enable the [https://ec.haxx.se/usingcurl/verbose verbose] option e.g. "... run the command with the {{kbd | key=<nowiki>-v</nowiki>}} / {{kbd | key=<nowiki>--verbose</nowiki>}} option to get more information." or (2) Enable the [https://ec.haxx.se/usingcurl/verbose/trace trace options] e.g. {{kbd | key=<nowiki>--trace-ascii [filename]</nowiki>}} Note: trace options will be override by verbose option if enable both options. | |||
== How to resolve Command Execution: Works in DOS/Terminal but Not in PHP == | |||
Often, commands that run perfectly when manually executed in DOS or the Windows terminal (like [https://conemu.github.io/ ConEmu]) might not produce the expected outcome when run through PHP. A common issue is the command not generating the anticipated file. | |||
Example cURL command for Windows terminal | |||
<pre> | |||
curl "https://www.example.com/path/to/resource/" ^ | |||
-H "authority: www.example.com" ^ | |||
-H "accept: text/html" ^ | |||
--data-raw "PARAMS" --compressed > output.html | |||
</pre> | |||
Solution | |||
* '''Single Continuous String in PHP''': In PHP, when executing a shell command, avoid using the caret (^) for multi-line commands<ref>[https://stackoverflow.com/questions/69068/split-long-commands-in-multiple-lines-through-windows-batch-file/21000752#21000752 Split long commands in multiple lines through Windows batch file - Stack Overflow]</ref>, which is typical in the Command Prompt. Instead, ensure the command is one continuous string. | |||
* '''Unsupported curl Options''': If you encounter an error like "curl: option --compressed: the installed libcurl version doesn't support this", it's a clear indication to remove or adjust the problematic option in the curl command. | |||
Revised cURL command for PHP | |||
<pre> | |||
curl "https://www.example.com/path/to/resource/" -H "authority: www.example.com" -H "accept: text/html" --data-raw "PARAMS" > output.html | |||
</pre> | |||
== How to resolve cUrl error (#5): Unsupported proxy syntax == | == How to resolve cUrl error (#5): Unsupported proxy syntax == | ||
Line 37: | Line 62: | ||
Related articles | Related articles | ||
* [https://github.com/curl/curl/issues/4982 cURL error 56: OpenSSL SSL_read: No error information · Issue #4982 · curl/curl] | * [https://github.com/curl/curl/issues/4982 cURL error 56: OpenSSL SSL_read: No error information · Issue #4982 · curl/curl] | ||
== How to resolve the error: error code: 1010 == | |||
Identifying the Issue | |||
* While attempting to download a webpage content via the cURL command, I encountered an issue where the output displayed was "error code: 1010". To delve deeper, I reran the cURL command with verbose mode activated (using the {{kbd | key=<nowiki>-v</nowiki>}} flag). The headers revealed in this mode included: | |||
<pre> | |||
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=xxx"}],"group":"cf-nel","max_age":604800} | |||
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} | |||
< server: cloudflare | |||
</pre> | |||
Proposed Solution | |||
* Instead of relying on the cURL command, switching to a JavaScript-based crawler is recommended as a solution to this problem <ref>[https://www.zenrows.com/blog/cloudflare-error-1010#how-to-avoid Cloudflare Error 1010: What Is It and How to Avoid - ZenRows]</ref>. | |||
== How to resolve Error 405 Method Not Allowed == | |||
When I tried to crawl the web resource but met the error message: "Error 405 Method Not Allowed" | |||
Solution: Make sure using the correct HTTP request method<ref>[https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html HTTP/1.1: Method Definitions]</ref> e.g. {{kbd | key=GET}} or {{kbd | key=POST}} to access the web resource. | |||
== How to resolve Error 415 Unsupported Media Type == | |||
When I tried to POST JSON (as payload) but met the error message: "Error 415 Unsupported Media Type" | |||
Solution: Set the cUrl with the header {{kbd | key=<nowiki>Content-Type: application/json</nowiki>}}<ref>[https://stackoverflow.com/questions/11087872/php-curl-post-returns-a-415-unsupported-media-type PHP cURL POST returns a 415 - Unsupported Media Type - Stack Overflow]</ref> Example syntax of cUrl command as following: | |||
<pre> | |||
curl -X POST --header "Content-Type: application/json" --data '{"user" : "your_user_name", "token" : "your_token"}' "https://example.com/" | |||
</pre> | |||
== How to rseolve cURL returns binary data instead of html or json == | |||
* [https://stackoverflow.com/questions/28283822/curl-returns-binary-data-instead-of-html php - cURL returns binary data instead of html - Stack Overflow] | |||
* [https://errerrors.blogspot.com/2020/06/resolve-curl-returns-binary-data-instead-of-html-or-json.html 解決 cUrl 下載的網頁或 JSON 看起來像是亂碼的問題] | |||
== How to resolve the downloaded file was empty == | |||
Error condition | |||
* When I executed the Curl command {{kbd | key=<nowiki>curl -O https://path/to/plain_text.txt</nowiki>}}, but the downloaded file was empty. | |||
Solution | |||
* After I enabled [https://ec.haxx.se/usingcurl/usingcurl-verbose verbose] option, I found the HTTP status code {{kbd | key=HTTP/1.1 302 Moved Temporarily}}. I added the location ([https://curl.haxx.se/docs/manpage.html#-L -L]) parameter to resolve the downloaded file was empty: {{kbd | key=<nowiki>curl -L -O https://path/to/plain_text.txt</nowiki>}}, | |||
<pre> | |||
The truncated result after executed the command: curl -v -O https://path/to/plain_text.txt | |||
< HTTP/1.1 302 Moved Temporarily | |||
< Server: nginx | |||
< Date: Tue, 25 Aug 2020 03:45:13 GMT | |||
< Content-Type: text/html | |||
< Transfer-Encoding: chunked | |||
< Connection: keep-alive | |||
< Keep-Alive: timeout=60 | |||
< Location: /path/to/plain_text.txt | |||
</pre> | |||
== How to resolve the error: no matches found == | |||
Error condition | |||
* When I executed the Curl command {{kbd | key=<nowiki>curl http://website.test/http_status.php?case=500</nowiki>}}, but I met the error message: "zsh: no matches found: <nowiki>http://website.test/http_status.php?case=500</nowiki>" | |||
Solution<ref>[https://medium.com/%E5%BD%BC%E5%BE%97%E6%BD%98%E7%9A%84-swift-ios-app-%E9%96%8B%E7%99%BC%E5%95%8F%E9%A1%8C%E8%A7%A3%E7%AD%94%E9%9B%86/%E7%82%BA%E4%BB%80%E9%BA%BC-mac-terminal-curl-%E6%8C%87%E4%BB%A4%E7%9A%84%E7%B6%B2%E5%9D%80%E9%80%9A%E5%B8%B8%E8%A6%81%E5%8A%A0%E9%9B%99%E5%BC%95%E8%99%9F-8c3e9227be34 為什麼 Mac terminal curl 指令的網址通常要加雙引號 - 彼得潘的 Swift iOS App 開發問題解答集 - Medium]</ref> | |||
* Double quote the {{kbd | key=<nowiki>URL</nowiki>}} e.g. {{kbd | key=<nowiki>curl "http://website.test/http_status.php?case=500"</nowiki>}} | |||
== How to resolve the error: Remote file name has no length == | |||
Error message | |||
<pre> | |||
curl -O "https://website.com/?p=123" | |||
curl: Remote file name has no length | |||
curl: (23) Failed writing received data to disk/application | |||
</pre> | |||
Solution | |||
<pre> | |||
curl "https://website.com/?p=123" -o myfile.html | |||
</pre> | |||
Explanation of solution | |||
* When you encounter an error while downloading a file with `curl`, it may be because the URL does not include a file name, causing `curl` to be unable to determine the name to save the file under. The solution is to manually specify a file name using the `-o` option, for example, `curl -o myfile.html "https://website.com/?p=123"`. Additionally, make sure you have enough disk space and are in a directory where you have write permissions. | |||
== How to resolve the error: zsh: parse error near `&' == | |||
Error condition | |||
<pre> | |||
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today's_Biggest_Trends"}' | |||
dquote> | |||
</pre> | |||
Solution: Escape the single quote {{kbd | key=<nowiki>'</nowiki>}} became {{kbd | key=<nowiki>''</nowiki>}} (repeat the single quote twice) | |||
<pre> | |||
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today''s_Biggest_Trends"}' | |||
</pre> | |||
Trivial Solution {{exclaim}} '''NOT WORK''': single quote {{kbd | key=<nowiki>'</nowiki>}} became {{kbd | key=<nowiki>\'</nowiki>}} | |||
<pre> | |||
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today\'s_Biggest_Trends"}' | |||
dquote> | |||
</pre> | |||
== Further reading == | |||
* [https://ec.haxx.se/ Introduction - Everything curl] | |||
* [https://curl.haxx.se/docs/manpage.html curl - How To Use] | |||
* [https://stackoverflow.com/questions/32122586/curl-escape-single-quote bash - CURL escape single quote - Stack Overflow] | |||
== References == | == References == |
Latest revision as of 13:32, 20 March 2024
cUrl "is used in command lines or scripts to transfer data."[1] (alternative library to achive sampe purpose: (1) PHP naive Client URL Library (2) cURL class for PHP (3) Installing cURL on Cygwin on Windows)
- How to debug the cUrl command: (1) Enable the verbose option e.g. "... run the command with the -v / --verbose option to get more information." or (2) Enable the trace options e.g. --trace-ascii [filename] Note: trace options will be override by verbose option if enable both options.
How to resolve Command Execution: Works in DOS/Terminal but Not in PHP[edit]
Often, commands that run perfectly when manually executed in DOS or the Windows terminal (like ConEmu) might not produce the expected outcome when run through PHP. A common issue is the command not generating the anticipated file.
Example cURL command for Windows terminal
curl "https://www.example.com/path/to/resource/" ^ -H "authority: www.example.com" ^ -H "accept: text/html" ^ --data-raw "PARAMS" --compressed > output.html
Solution
- Single Continuous String in PHP: In PHP, when executing a shell command, avoid using the caret (^) for multi-line commands[1], which is typical in the Command Prompt. Instead, ensure the command is one continuous string.
- Unsupported curl Options: If you encounter an error like "curl: option --compressed: the installed libcurl version doesn't support this", it's a clear indication to remove or adjust the problematic option in the curl command.
Revised cURL command for PHP
curl "https://www.example.com/path/to/resource/" -H "authority: www.example.com" -H "accept: text/html" --data-raw "PARAMS" > output.html
How to resolve cUrl error (#5): Unsupported proxy syntax[edit]
Condition: the username of socks5 proxy contains @ symbol such as [email protected]
curl_setopt($ch, CURLOPT_PROXY, 'socks5://username:password@localhost:12345'); // result is ok curl_setopt($ch, CURLOPT_PROXY, 'socks5://[email protected]:password@localhost:12345'); // met cUrl error (#5): Unsupported proxy syntax
Solution: Escape the special symbol[2] using URL Encode and Decode - Online or urlencode function
curl_setopt($ch, CURLOPT_PROXY, 'socks5://bob%40email.com:password@localhost:12345'); // result is ok
How to resolve cUrl error (#56): OpenSSL SSL_read: No error.[edit]
Condition: The curl vesion 7.67.0 on windows met problem. Result of curl_version returned:
version: Array ( [version_number] => 475904 [age] => 5 [features] => 2953117 [ssl_version_number] => 0 [version] => 7.67.0 [host] => x86_64-pc-win32 [ssl_version] => OpenSSL/1.1.1d [libz_version] => 1.2.11 )
Solution: Use other version of curl command or curl on Cygwin or cURL class for PHP.
Related articles
How to resolve the error: error code: 1010[edit]
Identifying the Issue
- While attempting to download a webpage content via the cURL command, I encountered an issue where the output displayed was "error code: 1010". To delve deeper, I reran the cURL command with verbose mode activated (using the -v flag). The headers revealed in this mode included:
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=xxx"}],"group":"cf-nel","max_age":604800} < nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} < server: cloudflare
Proposed Solution
- Instead of relying on the cURL command, switching to a JavaScript-based crawler is recommended as a solution to this problem [3].
How to resolve Error 405 Method Not Allowed[edit]
When I tried to crawl the web resource but met the error message: "Error 405 Method Not Allowed"
Solution: Make sure using the correct HTTP request method[4] e.g. GET or POST to access the web resource.
How to resolve Error 415 Unsupported Media Type[edit]
When I tried to POST JSON (as payload) but met the error message: "Error 415 Unsupported Media Type"
Solution: Set the cUrl with the header Content-Type: application/json[5] Example syntax of cUrl command as following:
curl -X POST --header "Content-Type: application/json" --data '{"user" : "your_user_name", "token" : "your_token"}' "https://example.com/"
How to rseolve cURL returns binary data instead of html or json[edit]
How to resolve the downloaded file was empty[edit]
Error condition
- When I executed the Curl command curl -O https://path/to/plain_text.txt, but the downloaded file was empty.
Solution
- After I enabled verbose option, I found the HTTP status code HTTP/1.1 302 Moved Temporarily. I added the location (-L) parameter to resolve the downloaded file was empty: curl -L -O https://path/to/plain_text.txt,
The truncated result after executed the command: curl -v -O https://path/to/plain_text.txt < HTTP/1.1 302 Moved Temporarily < Server: nginx < Date: Tue, 25 Aug 2020 03:45:13 GMT < Content-Type: text/html < Transfer-Encoding: chunked < Connection: keep-alive < Keep-Alive: timeout=60 < Location: /path/to/plain_text.txt
How to resolve the error: no matches found[edit]
Error condition
- When I executed the Curl command curl http://website.test/http_status.php?case=500, but I met the error message: "zsh: no matches found: http://website.test/http_status.php?case=500"
Solution[6]
- Double quote the URL e.g. curl "http://website.test/http_status.php?case=500"
How to resolve the error: Remote file name has no length[edit]
Error message
curl -O "https://website.com/?p=123" curl: Remote file name has no length curl: (23) Failed writing received data to disk/application
Solution
curl "https://website.com/?p=123" -o myfile.html
Explanation of solution
- When you encounter an error while downloading a file with `curl`, it may be because the URL does not include a file name, causing `curl` to be unable to determine the name to save the file under. The solution is to manually specify a file name using the `-o` option, for example, `curl -o myfile.html "https://website.com/?p=123"`. Additionally, make sure you have enough disk space and are in a directory where you have write permissions.
How to resolve the error: zsh: parse error near `&'[edit]
Error condition
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today's_Biggest_Trends"}' dquote>
Solution: Escape the single quote ' became '' (repeat the single quote twice)
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today''s_Biggest_Trends"}'
Trivial Solution NOT WORK: single quote ' became \'
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today\'s_Biggest_Trends"}' dquote>
Further reading[edit]
References[edit]
- ↑ Split long commands in multiple lines through Windows batch file - Stack Overflow
- ↑ Log in to FTP using Windows Explorer with @ in the username - Super User
- ↑ Cloudflare Error 1010: What Is It and How to Avoid - ZenRows
- ↑ HTTP/1.1: Method Definitions
- ↑ PHP cURL POST returns a 415 - Unsupported Media Type - Stack Overflow
- ↑ 為什麼 Mac terminal curl 指令的網址通常要加雙引號 - 彼得潘的 Swift iOS App 開發問題解答集 - Medium
Troubleshooting of ...
- PHP, cUrl, Python, selenium, HTTP status code errors
- Database: SQL syntax debug, MySQL errors, MySQLTuner errors or PostgreSQL errors
- HTML/Javascript: Troubleshooting of javascript, XPath
- Software: Mediawiki, Docker, FTP problems, online conference software
- Test connectivity for the web service, Web Ping, Network problem, Web user behavior, Web scrape troubleshooting
Template