Troubleshooting of curl errors
cUrl "is used in command lines or scripts to transfer data."[1] (alternative library to achive sampe purpose: (1) PHP naive Client URL Library (2) cURL class for PHP (3) Installing cURL on Cygwin on Windows)
How to debug the cUrl command: (1) Enable the verbose option e.g. "... run the command with the -v / --verbose option to get more information." or (2) Enable the trace options e.g. --trace-ascii [filename] Note: trace options will be override by verbose option if enable both options.
How to resolve Command Execution: Works in DOS/Terminal but Not in PHP[edit]
Often, commands that run perfectly when manually executed in DOS or the Windows terminal (like ConEmu) might not produce the expected outcome when run through PHP. A common issue is the command not generating the anticipated file.
Example cURL command for Windows terminal
curl "https://www.example.com/path/to/resource/" ^ -H "authority: www.example.com" ^ -H "accept: text/html" ^ --data-raw "PARAMS" --compressed > output.html
Solution
- Single Continuous String in PHP: In PHP, when executing a shell command, avoid using the caret (^) for multi-line commands[1], which is typical in the Command Prompt. Instead, ensure the command is one continuous string.
- Unsupported curl Options: If you encounter an error like "curl: option --compressed: the installed libcurl version doesn't support this", it's a clear indication to remove or adjust the problematic option in the curl command.
Revised cURL command for PHP
curl "https://www.example.com/path/to/resource/" -H "authority: www.example.com" -H "accept: text/html" --data-raw "PARAMS" > output.html
How to resolve cURL Binary Output Warning[edit]
When executing a cURL command, you might encounter this warning message:
Warning: Binary output can mess up your terminal. Use "--output -" to tell Warning: curl to output it to your terminal anyway, or consider "--output Warning: <FILE>" to save to a file.
To resolve this warning, append the --output flag to your cURL command:
- To save the output to a file: --output output.file
- To force output to terminal: --output -
Example:
curl https://example.com/binary-file --output output.file
How to resolve cURL error: HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)[edit]
Workaround solution[2]:
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
How to resolve cUrl error (3) bad range in URL position xx[edit]
Problem When using curl to download URLs containing special characters (like `[]`), you may encounter this error:
curl: (3) bad range in URL position 33
Solution Convert special characters to URL-encoded format:
# Incorrect example curl "https://example.com/api?param[0]=value" # Correct example curl "https://example.com/api?param%5B0%5D=value"
Common URL encoding references:
- `[` becomes `%5B`
- `]` becomes `%5D`
- Space becomes `%20`
Explanation
- URL specifications require certain special characters to be encoded for use
- Characters like `[]` have special meanings in URLs and can cause parsing errors if used directly
- URL encoding ensures these special characters can be correctly transmitted and parsed
If you're unsure whether encoding is needed, you can use an online URL encoder tool for conversion e.g. URL Encode and Decode - Online.
How to resolve cUrl error (#5): Unsupported proxy syntax[edit]
Condition: the username of socks5 proxy contains @ symbol such as [email protected]
curl_setopt($ch, CURLOPT_PROXY, 'socks5://username:password@localhost:12345'); // result is ok curl_setopt($ch, CURLOPT_PROXY, 'socks5://[email protected]:password@localhost:12345'); // met cUrl error (#5): Unsupported proxy syntax
Solution: Escape the special symbol[3] using URL Encode and Decode - Online or urlencode function
curl_setopt($ch, CURLOPT_PROXY, 'socks5://bob%40email.com:password@localhost:12345'); // result is ok
How to resolve cUrl error (#56): OpenSSL SSL_read: No error.[edit]
Condition: The curl vesion 7.67.0 on windows met problem. Result of curl_version returned:
version: Array ( [version_number] => 475904 [age] => 5 [features] => 2953117 [ssl_version_number] => 0 [version] => 7.67.0 [host] => x86_64-pc-win32 [ssl_version] => OpenSSL/1.1.1d [libz_version] => 1.2.11 )
Solution: Use other version of curl command or curl on Cygwin or cURL class for PHP.
Related articles
How to resolve cURL error: SSL: no alternative certificate subject name matches target host name xxx[edit]
Walkaround solution[4]:
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
How to resolve the error: error code: 1010[edit]
Identifying the Issue
- While attempting to download a webpage content via the cURL command, I encountered an issue where the output displayed was "error code: 1010". To delve deeper, I reran the cURL command with verbose mode activated (using the -v flag). The headers revealed in this mode included:
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=xxx"}],"group":"cf-nel","max_age":604800} < nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} < server: cloudflare
Proposed Solution
- Instead of relying on the cURL command, switching to a JavaScript-based crawler is recommended as a solution to this problem [5].
How to resolve 400 Bad Request Errors: UTF-8 Character Encoding Issues[edit]
Problem Analysis:
Example cURL command for macOS terminal with error:
$ curl "https://api.search.brave.com/res/v1/web/search?q=\"XXX\"+公司簡介+產品+服務" \ -H "Accept: application/json" \ -H "Accept-Encoding: gzip" \ -H "X-Subscription-Token: YOUR_API_KEY_HERE" > <html><body><h1>400 Bad request</h1> Your browser sent an invalid request. </body></html>
Solutions
Properly Handle URL Encoding: Chinese characters and special symbols need to be correctly URL-encoded to avoid 400 Bad Request errors (online tool: URL Encode and Decode - Online):
- "XXX" → %22XXX%22
- 公司簡介 → %E5%85%AC%E5%8F%B8%E7%B0%A1%E4%BB%8B
- 產品 → %E7%94%A2%E5%93%81
- 服務 → %E6%9C%8D%E5%8B%99
Revised cURL Command:
$ curl "https://api.search.brave.com/res/v1/web/search?q=%22XXX%22+%E5%85%AC%E5%8F%B8%E7%B0%A1%E4%BB%8B+%E7%94%A2%E5%93%81+%E6%9C%8D%E5%8B%99" \ -H "Accept: application/json" \ -H "Accept-Encoding: gzip" \ -H "X-Subscription-Token: YOUR_API_KEY_HERE"
How to resolve Error 405 Method Not Allowed[edit]
When I tried to crawl the web resource but met the error message: "Error 405 Method Not Allowed"
Solution: Make sure using the correct HTTP request method[6] e.g. GET or POST to access the web resource.
How to resolve Error 415 Unsupported Media Type[edit]
When I tried to POST JSON (as payload) but met the error message: "Error 415 Unsupported Media Type"
Solution: Set the cUrl with the header Content-Type: application/json[7] Example syntax of cUrl command as following:
curl -X POST --header "Content-Type: application/json" --data '{"user" : "your_user_name", "token" : "your_token"}' "https://example.com/"
How to rseolve cURL returns binary data instead of html or json[edit]
How to resolve the downloaded file was empty[edit]
Error condition
- When I executed the Curl command curl -O https://path/to/plain_text.txt, but the downloaded file was empty.
Solution
- After I enabled verbose option, I found the HTTP status code HTTP/1.1 302 Moved Temporarily. I added the location (-L) parameter to resolve the downloaded file was empty: curl -L -O https://path/to/plain_text.txt,
The truncated result after executed the command: curl -v -O https://path/to/plain_text.txt < HTTP/1.1 302 Moved Temporarily < Server: nginx < Date: Tue, 25 Aug 2020 03:45:13 GMT < Content-Type: text/html < Transfer-Encoding: chunked < Connection: keep-alive < Keep-Alive: timeout=60 < Location: /path/to/plain_text.txt
How to resolve the error: no matches found[edit]
Error condition
- When I executed the Curl command curl http://website.test/http_status.php?case=500, but I met the error message: "zsh: no matches found: http://website.test/http_status.php?case=500"
Solution[8]
- Double quote the URL e.g. curl "http://website.test/http_status.php?case=500"
Explanation by Claude
When you see a "no matches found" error in zsh (a type of shell), it's because the shell gets confused by special characters in your command. The main problem is the question mark (?) in the URL.
In a shell, some characters do special jobs:
- ? means "any one character"
- * means "any set of characters"
- [] is for grouping characters
- {} is for making lists of items
Here's what happens:
- When you run curl without quotes, the shell sees the ? and thinks "I need to find a file that matches this pattern"
- It looks in your current folder for matching files
- It can't find any files, so it shows the error
The fix is simple: put double quotes ("") around your URL. This tells the shell "just use these characters exactly as they are - don't try to do anything special with them." Now the ? goes straight to curl as part of the web address, instead of being used as a special character by the shell.
How to resolve the error: Remote file name has no length[edit]
Error message
curl -O "https://website.com/?p=123" curl: Remote file name has no length curl: (23) Failed writing received data to disk/application
Solution
curl "https://website.com/?p=123" -o myfile.html
Explanation of solution
- When you encounter an error while downloading a file with `curl`, it may be because the URL does not include a file name, causing `curl` to be unable to determine the name to save the file under. The solution is to manually specify a file name using the `-o` option, for example, `curl -o myfile.html "https://website.com/?p=123"`. Additionally, make sure you have enough disk space and are in a directory where you have write permissions.
How to resolve the error: zsh: parse error near `&'[edit]
Error condition
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today's_Biggest_Trends"}' dquote>
Solution: Escape the single quote ' became '' (repeat the single quote twice)
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today''s_Biggest_Trends"}'
Trivial Solution NOT WORK: single quote ' became \'
% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json" -d'{"url": "https://someweb.com/Today\'s_Biggest_Trends"}' dquote>
How to Suppress the Progress Meter[edit]
When executing the curl command, the typical output includes detailed information about the download and upload process as shown below:
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 90141 100 90141 0 0 20029 0 0:00:04 0:00:04 --:--:-- 22012
To suppress the progress meter of the curl command, you can use the -s (silent) option, which hides all output, or the -S (silent with error messages) option, which suppresses the progress meter but still displays error messages.
curl -sS [URL]
Alternatively, if you prefer to suppress all output including error messages, use the -s (silent) option alone:
curl -s [URL]
cURL JSON Payload Issues in Windows vs Mac/Linux Systems[edit]
1. Problem Error encountered when executing the following cURL command on Windows:
Invalid JSON payload received. Unknown name "": Root element must be a message.
2. Solution Windows system:
curl -X POST -H "Content-Type: application/json" "API_URL" -d "{\"key\":\"value\"}"
Mac/Linux system:
curl -X POST -H "Content-Type: application/json" "API_URL" -d '{"key":"value"}'
PHP code:
$is_windows = strtoupper(substr(PHP_OS, 0, 3)) === 'WIN'; if ($is_windows) { $command = sprintf('-d "{\"key\":\"%s\"}"', $value); } else { $command = sprintf("-d '%s'", $json_payload); }
3. Problem Description
- Root cause: Windows and Mac/Linux handle quotes in shell commands differently
- Windows: Requires double quotes and escaped internal quotes
- Mac/Linux: Can use single quotes to wrap the complete JSON string
- When using incorrect quote formatting, the JSON payload is parsed incorrectly, causing the API to return a format error message
Further reading[edit]
- Introduction - Everything curl
- curl - How To Use
- bash - CURL escape single quote - Stack Overflow
- explainshell.com - match command-line arguments to their help text
References[edit]
- ↑ Split long commands in multiple lines through Windows batch file - Stack Overflow
- ↑ PHP Curl error HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1) - Stack Overflow
- ↑ Log in to FTP using Windows Explorer with @ in the username - Super User
- ↑ Curl: Fix CURL (51) SSL error: no alternative certificate subject name matches - Stack Overflow
- ↑ Cloudflare Error 1010: What Is It and How to Avoid - ZenRows
- ↑ HTTP/1.1: Method Definitions
- ↑ PHP cURL POST returns a 415 - Unsupported Media Type - Stack Overflow
- ↑ 為什麼 Mac terminal curl 指令的網址通常要加雙引號 - 彼得潘的 Swift iOS App 開發問題解答集 - Medium
Troubleshooting of ...
- PHP, cUrl, Python, selenium, HTTP status code errors
- Database: SQL syntax debug, MySQL errors, MySQLTuner errors or PostgreSQL errors
- HTML/Javascript: Troubleshooting of javascript, XPath
- Software: Mediawiki, Docker, FTP problems, online conference software
- Test connectivity for the web service, Web Ping, Network problem, Web user behavior, Web scrape troubleshooting
Template