Troubleshooting of curl errors
cUrl "is used in command lines or scripts to transfer data."[1] (alternative library to achive sampe purpose: (1) PHP naive Client URL Library (2) cURL class for PHP (3) Installing cURL on Cygwin on Windows)
- How to debug the cUrl command: (1) Enable the verbose option e.g. "... run the command with the -v / --verbose option to get more information." or (2) Enable the trace options e.g. --trace-ascii [filename] Note: trace options will be override by verbose option if enable both options.
How to resolve Command Execution: Works in DOS/Terminal but Not in PHP
Often, commands that run perfectly when manually executed in DOS or the Windows terminal (like ConEmu) might not produce the expected outcome when run through PHP. A common issue is the command not generating the anticipated file.
Example cURL command for Windows terminal
curl "https://www.example.com/path/to/resource/" ^ -H "authority: www.example.com" ^ -H "accept: text/html" ^ --data-raw "PARAMS" --compressed > output.html
Solution
- Single Continuous String in PHP: In PHP, when executing a shell command, avoid using the caret (^) for multi-line commands[1], which is typical in the Command Prompt. Instead, ensure the command is one continuous string.
- Unsupported curl Options: If you encounter an error like "curl: option --compressed: the installed libcurl version doesn't support this", it's a clear indication to remove or adjust the problematic option in the curl command.
Revised cURL command for PHP
curl "https://www.example.com/path/to/resource/" -H "authority: www.example.com" -H "accept: text/html" --data-raw "PARAMS" > output.html
How to resolve cUrl error (#5): Unsupported proxy syntax
Condition: the username of socks5 proxy contains @ symbol such as [email protected]
curl_setopt($ch, CURLOPT_PROXY, 'socks5://username:password@localhost:12345'); // result is ok curl_setopt($ch, CURLOPT_PROXY, 'socks5://[email protected]:password@localhost:12345'); // met cUrl error (#5): Unsupported proxy syntax
Solution: Escape the special symbol[2] using URL Encode and Decode - Online or urlencode function
curl_setopt($ch, CURLOPT_PROXY, 'socks5://bob%40email.com:password@localhost:12345'); // result is ok
How to resolve cUrl error (#56): OpenSSL SSL_read: No error.
Condition: The curl vesion 7.67.0 on windows met problem. Result of curl_version returned:
version: Array ( [version_number] => 475904 [age] => 5 [features] => 2953117 [ssl_version_number] => 0 [version] => 7.67.0 [host] => x86_64-pc-win32 [ssl_version] => OpenSSL/1.1.1d [libz_version] => 1.2.11 )
Solution: Use other version of curl command or curl on Cygwin or cURL class for PHP.
Related articles
How to resolve the error: error code: 1010
Identifying the Issue
- While attempting to download a webpage content via the cURL command, I encountered an issue where the output displayed was "error code: 1010". To delve deeper, I reran the cURL command with verbose mode activated (using the -v flag). The headers revealed in this mode included:
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=xxx"}],"group":"cf-nel","max_age":604800} < nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} < server: cloudflare
Proposed Solution
- Instead of relying on the cURL command, switching to a JavaScript-based crawler is recommended as a solution to this problem [3].
How to resolve Error 405 Method Not Allowed
When I tried to crawl the web resource but met the error message: "Error 405 Method Not Allowed"
Solution: Make sure using the correct HTTP request method[4] e.g. GET or POST to access the web resource.
How to resolve Error 415 Unsupported Media Type
When I tried to POST JSON (as payload) but met the error message: "Error 415 Unsupported Media Type"
Solution: Set the cUrl with the header Content-Type: application/json[5] Example syntax of cUrl command as following:
curl -X POST --header "Content-Type: application/json" --data '{"user" : "your_user_name", "token" : "your_token"}' "https://example.com/"
How to rseolve cURL returns binary data instead of html or json
How to resolve the downloaded file was empty
Error condition
- When I executed the Curl command curl -O https://path/to/plain_text.txt, but the downloaded file was empty.
Solution
- After I enabled verbose option, I found the HTTP status code HTTP/1.1 302 Moved Temporarily. I added the location (-L) parameter to resolve the downloaded file was empty: curl -L -O https://path/to/plain_text.txt,
The truncated result after executed the command: curl -v -O https://path/to/plain_text.txt < HTTP/1.1 302 Moved Temporarily < Server: nginx < Date: Tue, 25 Aug 2020 03:45:13 GMT < Content-Type: text/html < Transfer-Encoding: chunked < Connection: keep-alive < Keep-Alive: timeout=60 < Location: /path/to/plain_text.txt
How to resolve the error: no matches found
Error condition
- When I executed the Curl command curl http://website.test/http_status.php?case=500, but I met the error message: "zsh: no matches found: http://website.test/http_status.php?case=500"
Solution[6]
- Double quote the URL e.g. curl "http://website.test/http_status.php?case=500"
How to resolve the error: Remote file name has no length
Error message
curl -O "https://website.com/?p=123" curl: Remote file name has no length curl: (23) Failed writing received data to disk/application
Solution
curl "https://website.com/?p=123" -o myfile.html
Explanation of solution
- When you encounter an error while downloading a file with `curl`, it may be because the URL does not include a file name, causing `curl` to be unable to determine the name to save the file under. The solution is to manually specify a file name using the `-o` option, for example, `curl -o myfile.html "https://website.com/?p=123"`. Additionally, make sure you have enough disk space and are in a directory where you have write permissions.
How to resolve the error: zsh: parse error near `&'
Error condition
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today's_Biggest_Trends"}' dquote>
Solution: Escape the single quote ' became '' (repeat the single quote twice)
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today''s_Biggest_Trends"}'
Trivial Solution NOT WORK: single quote ' became \'
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today\'s_Biggest_Trends"}' dquote>
Further reading
References
- ↑ Split long commands in multiple lines through Windows batch file - Stack Overflow
- ↑ Log in to FTP using Windows Explorer with @ in the username - Super User
- ↑ Cloudflare Error 1010: What Is It and How to Avoid - ZenRows
- ↑ HTTP/1.1: Method Definitions
- ↑ PHP cURL POST returns a 415 - Unsupported Media Type - Stack Overflow
- ↑ 為什麼 Mac terminal curl 指令的網址通常要加雙引號 - 彼得潘的 Swift iOS App 開發問題解答集 - Medium
Troubleshooting of ...
- PHP, cUrl, Python, selenium, HTTP status code errors
- Database: SQL syntax debug, MySQL errors, MySQLTuner errors or PostgreSQL errors
- HTML/Javascript: Troubleshooting of javascript, XPath
- Software: Mediawiki, Docker, FTP problems, online conference software
- Test connectivity for the web service, Web Ping, Network problem, Web user behavior, Web scrape troubleshooting
Template