Troubleshooting of curl errors

From LemonWiki共筆
Jump to navigation Jump to search

cUrl "is used in command lines or scripts to transfer data."[1] (alternative library to achive sampe purpose: (1) PHP naive Client URL Library (2) cURL class for PHP (3) Installing cURL on Cygwin on Windows)

🌐 Switch language: EN, 漢字


Owl icon.jpg How to debug the cUrl command: (1) Enable the verbose option e.g. "... run the command with the -v / --verbose option to get more information." or (2) Enable the trace options e.g. --trace-ascii [filename] Note: trace options will be override by verbose option if enable both options.


How to resolve Command Execution: Works in DOS/Terminal but Not in PHP[edit]

Often, commands that run perfectly when manually executed in DOS or the Windows terminal (like ConEmu) might not produce the expected outcome when run through PHP. A common issue is the command not generating the anticipated file.

Example cURL command for Windows terminal 

curl "https://www.example.com/path/to/resource/" ^
-H "authority: www.example.com" ^
-H "accept: text/html" ^
--data-raw "PARAMS" --compressed > output.html

Solution

  • Single Continuous String in PHP: In PHP, when executing a shell command, avoid using the caret (^) for multi-line commands[1], which is typical in the Command Prompt. Instead, ensure the command is one continuous string.
  • Unsupported curl Options: If you encounter an error like "curl: option --compressed: the installed libcurl version doesn't support this", it's a clear indication to remove or adjust the problematic option in the curl command.

Revised cURL command for PHP 

curl "https://www.example.com/path/to/resource/" -H "authority: www.example.com" -H "accept: text/html" --data-raw "PARAMS" > output.html

How to resolve cURL Binary Output Warning[edit]

When executing a cURL command, you might encounter this warning message: 

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

To resolve this warning, append the --output flag to your cURL command:

  • To save the output to a file: --output output.file
  • To force output to terminal: --output -

Example: 

curl https://example.com/binary-file --output output.file

How to resolve cURL error: HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)[edit]

Workaround solution[2]: 

curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);

How to resolve cUrl error (3) bad range in URL position xx[edit]

Problem When using curl to download URLs containing special characters (like `[]`), you may encounter this error: 

curl: (3) bad range in URL position 33

Solution Convert special characters to URL-encoded format: 

# Incorrect example
curl "https://example.com/api?param[0]=value"

# Correct example
curl "https://example.com/api?param%5B0%5D=value"

Common URL encoding references:

  • `[` becomes `%5B`
  • `]` becomes `%5D`
  • Space becomes `%20`

Explanation

  • URL specifications require certain special characters to be encoded for use
  • Characters like `[]` have special meanings in URLs and can cause parsing errors if used directly
  • URL encoding ensures these special characters can be correctly transmitted and parsed

If you're unsure whether encoding is needed, you can use an online URL encoder tool for conversion e.g. URL Encode and Decode - Online.

How to resolve cUrl error (#5): Unsupported proxy syntax[edit]

Condition: the username of socks5 proxy contains @ symbol such as [email protected] 

curl_setopt($ch, CURLOPT_PROXY, 'socks5://username:password@localhost:12345');
// result is ok

curl_setopt($ch, CURLOPT_PROXY, 'socks5://[email protected]:password@localhost:12345');
// met cUrl error (#5): Unsupported proxy syntax

Solution: Escape the special symbol[3] using URL Encode and Decode - Online or urlencode function 

curl_setopt($ch, CURLOPT_PROXY, 'socks5://bob%40email.com:password@localhost:12345');
// result is ok

How to resolve cUrl error (#56): OpenSSL SSL_read: No error.[edit]

Condition: The curl vesion 7.67.0 on windows met problem. Result of curl_version returned: 

version: Array
(
    [version_number] => 475904
    [age] => 5
    [features] => 2953117
    [ssl_version_number] => 0
    [version] => 7.67.0
    [host] => x86_64-pc-win32
    [ssl_version] => OpenSSL/1.1.1d
    [libz_version] => 1.2.11
)

Solution: Use other version of curl command or curl on Cygwin or cURL class for PHP.

Related articles

How to resolve cURL error: SSL: no alternative certificate subject name matches target host name xxx[edit]

Walkaround solution[4]: 

curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);

How to resolve the error: error code: 1010[edit]

Identifying the Issue

  • While attempting to download a webpage content via the cURL command, I encountered an issue where the output displayed was "error code: 1010". To delve deeper, I reran the cURL command with verbose mode activated (using the -v flag). The headers revealed in this mode included:
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=xxx"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare

Proposed Solution

  • Instead of relying on the cURL command, switching to a JavaScript-based crawler is recommended as a solution to this problem [5].

How to resolve 400 Bad Request Errors: UTF-8 Character Encoding Issues[edit]

Problem Analysis:

Example cURL command for macOS terminal with error: 

$ curl "https://api.search.brave.com/res/v1/web/search?q=\"XXX\"+公司簡介+產品+服務" \
 -H "Accept: application/json" \
 -H "Accept-Encoding: gzip" \
 -H "X-Subscription-Token: YOUR_API_KEY_HERE"

> <html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>

Solutions

Properly Handle URL Encoding: Chinese characters and special symbols need to be correctly URL-encoded to avoid 400 Bad Request errors (online tool: URL Encode and Decode - Online):

  • "XXX" → %22XXX%22
  • 公司簡介 → %E5%85%AC%E5%8F%B8%E7%B0%A1%E4%BB%8B
  • 產品 → %E7%94%A2%E5%93%81
  • 服務 → %E6%9C%8D%E5%8B%99

Revised cURL Command: 

$ curl "https://api.search.brave.com/res/v1/web/search?q=%22XXX%22+%E5%85%AC%E5%8F%B8%E7%B0%A1%E4%BB%8B+%E7%94%A2%E5%93%81+%E6%9C%8D%E5%8B%99" \
 -H "Accept: application/json"  \
 -H "Accept-Encoding: gzip" \
 -H "X-Subscription-Token: YOUR_API_KEY_HERE"

How to resolve Error 405 Method Not Allowed[edit]

When I tried to crawl the web resource but met the error message: "Error 405 Method Not Allowed"

Solution: Make sure using the correct HTTP request method[6] e.g. GET or POST to access the web resource.

How to resolve Error 415 Unsupported Media Type[edit]

When I tried to POST JSON (as payload) but met the error message: "Error 415 Unsupported Media Type"

Solution: Set the cUrl with the header Content-Type: application/json[7] Example syntax of cUrl command as following: 

curl -X POST --header "Content-Type: application/json" --data '{"user" : "your_user_name", "token" : "your_token"}' "https://example.com/"

How to rseolve cURL returns binary data instead of html or json[edit]

How to resolve the downloaded file was empty[edit]

Error condition

  • When I executed the Curl command curl -O https://path/to/plain_text.txt, but the downloaded file was empty.

Solution

  • After I enabled verbose option, I found the HTTP status code HTTP/1.1 302 Moved Temporarily. I added the location (-L) parameter to resolve the downloaded file was empty: curl -L -O https://path/to/plain_text.txt,
The truncated result after executed the command: curl -v -O https://path/to/plain_text.txt

< HTTP/1.1 302 Moved Temporarily
< Server: nginx
< Date: Tue, 25 Aug 2020 03:45:13 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=60
< Location: /path/to/plain_text.txt

How to resolve the error: no matches found[edit]

Error condition

  • When I executed the Curl command curl http://website.test/http_status.php?case=500, but I met the error message: "zsh: no matches found: http://website.test/http_status.php?case=500"

Solution[8]

  • Double quote the URL e.g. curl "http://website.test/http_status.php?case=500"

Explanation by Claude

When you see a "no matches found" error in zsh (a type of shell), it's because the shell gets confused by special characters in your command. The main problem is the question mark (?) in the URL.

In a shell, some characters do special jobs:

  • ? means "any one character"
  • * means "any set of characters"
  • [] is for grouping characters
  • {} is for making lists of items

Here's what happens:

  1. When you run curl without quotes, the shell sees the ? and thinks "I need to find a file that matches this pattern"
  2. It looks in your current folder for matching files
  3. It can't find any files, so it shows the error

The fix is simple: put double quotes ("") around your URL. This tells the shell "just use these characters exactly as they are - don't try to do anything special with them." Now the ? goes straight to curl as part of the web address, instead of being used as a special character by the shell.

How to resolve the error: Remote file name has no length[edit]

Error message 

curl -O "https://website.com/?p=123"
curl: Remote file name has no length
curl: (23) Failed writing received data to disk/application

Solution 

curl "https://website.com/?p=123" -o myfile.html

Explanation of solution

  • When you encounter an error while downloading a file with `curl`, it may be because the URL does not include a file name, causing `curl` to be unable to determine the name to save the file under. The solution is to manually specify a file name using the `-o` option, for example, `curl -o myfile.html "https://website.com/?p=123"`. Additionally, make sure you have enough disk space and are in a directory where you have write permissions.

How to resolve the error: zsh: parse error near `&'[edit]

Error condition 

% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json"  -d'{"url": "https://someweb.com/Today's_Biggest_Trends"}'
dquote>

Solution: Escape the single quote ' became '' (repeat the single quote twice) 

% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json"  -d'{"url": "https://someweb.com/Today''s_Biggest_Trends"}'

Trivial Solution Icon_exclaim.gif NOT WORK: single quote ' became \' 

% curl -XPOST http://localhost:3000/ -H "Content-Type: application/json"  -d'{"url": "https://someweb.com/Today\'s_Biggest_Trends"}'
dquote>

How to Suppress the Progress Meter[edit]

When executing the curl command, the typical output includes detailed information about the download and upload process as shown below: 

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 90141  100 90141    0     0  20029      0  0:00:04  0:00:04 --:--:-- 22012

To suppress the progress meter of the curl command, you can use the -s (silent) option, which hides all output, or the -S (silent with error messages) option, which suppresses the progress meter but still displays error messages. 

curl -sS [URL]

Alternatively, if you prefer to suppress all output including error messages, use the -s (silent) option alone: 

curl -s [URL]

cURL JSON Payload Issues in Windows vs Mac/Linux Systems[edit]

1. Problem Error encountered when executing the following cURL command on Windows: 

Invalid JSON payload received. Unknown name "": Root element must be a message.

2. Solution Windows system: 

curl -X POST -H "Content-Type: application/json" "API_URL" -d "{\"key\":\"value\"}"

Mac/Linux system: 

curl -X POST -H "Content-Type: application/json" "API_URL" -d '{"key":"value"}'

PHP code: 

$is_windows = strtoupper(substr(PHP_OS, 0, 3)) === 'WIN';
if ($is_windows) {
    $command = sprintf('-d "{\"key\":\"%s\"}"', $value);
} else {
    $command = sprintf("-d '%s'", $json_payload);
}

3. Problem Description

  • Root cause: Windows and Mac/Linux handle quotes in shell commands differently
  • Windows: Requires double quotes and escaped internal quotes
  • Mac/Linux: Can use single quotes to wrap the complete JSON string
  • When using incorrect quote formatting, the JSON payload is parsed incorrectly, causing the API to return a format error message

Further reading[edit]

References[edit]

  1. ↑ Split long commands in multiple lines through Windows batch file - Stack Overflow
  2. ↑ PHP Curl error HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1) - Stack Overflow
  3. ↑ Log in to FTP using Windows Explorer with @ in the username - Super User
  4. ↑ Curl: Fix CURL (51) SSL error: no alternative certificate subject name matches - Stack Overflow
  5. ↑ Cloudflare Error 1010: What Is It and How to Avoid - ZenRows
  6. ↑ HTTP/1.1: Method Definitions
  7. ↑ PHP cURL POST returns a 415 - Unsupported Media Type - Stack Overflow
  8. ↑ 為什麼 Mac terminal curl 指令的網址通常要加雙引號 - 彼得潘的 Swift iOS App 開發問題解答集 - Medium


Troubleshooting of ...

Template

Retrieved from "https://wiki.planetoid.info/index.php?title=Troubleshooting_of_curl_errors&oldid=25511"