Fix garbled message text: Difference between revisions

Jump to navigation Jump to search

Fix garbled message text (edit)

Revision as of 14:21, 2 May 2024

5,068 bytes added ,  2 May 2024
→‎String starting from \u, \U or U+ symbol
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Ideas on how to fix garbled message text ==
== How to fix garbled message text ==
Ideas on how to fix garbled message text
# Possible cause
# Possible cause
#* Encoding issue: Choose the correct the language/encode of message text or auto detect the encode by tools
#* Encoding issue: Choose the correct the language/encode of message text or auto detect the encode by tools
Line 6: Line 7:
# (optional) Making text wrap to window size
# (optional) Making text wrap to window size


== List of the (look like but not) garbled text and possible cause ==
List of the (look like but not) garbled text and possible root cause
<table border="1" style="width: 100%; table-layout: fixed;" class="wikitable sortable">
<table border="1" style="width: 100%; table-layout: fixed;" class="wikitable sortable">
<tr>
<tr>
Line 43: Line 44:
<td>Unicode HTML code. "Unicode number in decimal, hex or octal"<ref>[http://www.amp-what.com/help.html &what Help]</ref></td>
<td>Unicode HTML code. "Unicode number in decimal, hex or octal"<ref>[http://www.amp-what.com/help.html &what Help]</ref></td>
<td>[https://www.php.net/manual/en/function.html-entity-decode.php PHP: html_entity_decode] ↔ (See the following section to understand how to encode)</td>
<td>[https://www.php.net/manual/en/function.html-entity-decode.php PHP: html_entity_decode] ↔ (See the following section to understand how to encode)</td>
</tr>
<tr>
<td>HTML source code starting from {{kbd | key=<nowiki>& ... ;</nowiki>}} symbols</td>
<td style="word-wrap: break-word;">{{kbd | key=<nowiki>& a m p ;</nowiki>}} (without whitespace) is {{kbd | key=<nowiki>&amp;</nowiki>}}</td>
<td>"all characters which have HTML character entity equivalents are translated into these entities"<ref>[https://www.php.net/manual/en/function.htmlentities.php PHP: htmlentities - Manual]</ref></td>
<td>[https://www.php.net/manual/en/function.htmlspecialchars-decode.php PHP: htmlspecialchars_decode] ↔ [https://www.php.net/manual/en/function.htmlentities.php PHP: htmlentities]</td>
</tr>
</table>
Possible approaches to encode the message text:
<table border="1" style="width: 100%; table-layout: fixed;" class="wikitable sortable">
<tr>
<th style="width: 20%;"> Approach
</th>
<th style="width: 25%"> Goal
</th>
<th style="width: 20%"> Is Chinese text garbled/encoded?
</th>
<th style="width: 35%;"> Sample text before encoded or after encoded
</th>
</tr>
<tr>
<th> [https://www.w3schools.com/jsref/jsref_encodeURIComponent.asp JavaScript encodeURIComponent()] <br />↔<br /> [http://www.w3schools.com/jsref/jsref_decodeuricomponent.asp JavaScript decodeURIComponent()]<ref>[http://stackoverflow.com/questions/9901027/how-to-encode-url-contains-unicode-characters-with-php urlencode - How to Encode URL Contains Unicode Characters with PHP - Stack Overflow]</ref>
</th>
<td>  "converts characters into a format that can be transmitted over the Internet ... " Cited from [http://www.w3schools.com/tags/ref_urlencode.asp w3schools]
</td>
<td> TRUE
</td>
<td style="word-wrap: break-word;"> <ul><li>before: {{kbd | key=<nowiki>http://www.中文網址.tw/my test.asp?name=ståle&car=saab</nowiki>}} <li>after: {{kbd | key=<nowiki>http%3A%2F%2Fwww.%E4%B8%AD%E6%96%87%E7%B6%B2%E5%9D%80.tw%2Fmy%20test.asp%3Fname%3Dst%C3%A5le%26car%3Dsaab</nowiki>}} </ul>
</td>
</tr>
<tr>
<th> [http://meyerweb.com/eric/tools/dencoder/ URL Decoder/Encoder]<ref>PHP [http://php.net/manual/en/function.urlencode.php urlencode()]</ref>
</th>
<td> (same as above)
</td>
<td> TRUE
</td>
<td style="word-wrap: break-word;"> (same as above)
</td>
</tr>
<tr>
<th> [http://php.net/manual/en/function.json-encode.php PHP: json_encode]<br />↔<br />[http://php.net/manual/en/function.json-decode.php PHP: json_decode]
</th>
<td>Save array in mysql database
</td>
<td> TRUE
</td>
<td style="word-wrap: break-word;"> <ul><li>before: {{kbd | key=<nowiki>array("作者" => "馬克吐溫", "名言" => "\"To a man with a hammer, everything looks like a nail.\" He said.");</nowiki>}} <li>after: {{kbd | key=<nowiki>{"\u4f5c\u8005":"\u99ac\u514b\u5410\u6eab","\u540d\u8a00":"\"To a man with a hammer, everything looks like a nail.\" He said."}</nowiki>}}</ul>
</td>
</tr>
<tr>
<th> [http://php.net/serialize PHP: serialize] <br />↔<br /> [http://php.net/manual/en/function.unserialize.php PHP: unserialize]
</th>
<td>[http://stackoverflow.com/questions/10686333/save-array-in-mysql-database Save array in mysql database]
</td>
<td> <span style="color: #999">FALSE</span>
</td>
<td style="word-wrap: break-word;"> <ul><li>before: {{kbd | key=<nowiki>array("作者" => "馬克吐溫", "名言" => "\"To a man with a hammer, everything looks like a nail.\" He said.");</nowiki>}} <li>after: {{kbd | key=<nowiki>a:2:{s:6:"作者";s:12:"馬克吐溫";s:6:"名言";s:64:""To a man with a hammer, everything looks like a nail." He said.";}</nowiki>}}</ul>
</td>
</tr>
<tr>
<th> [http://php.net/manual/en/function.htmlentities.php PHP: htmlentities][http://www.w3schools.com/html/html_entities.asp] <br />↔<br /> [http://php.net/manual/en/function.html-entity-decode.php PHP:  html_entity_decode]
</th>
<td> Replace reserved characters e.g. double quote symbol
</td>
<td> <span style="color: #999">FALSE</span>
</td>
<td style="word-wrap: break-word;"> <ul><li>before: {{kbd | key=<nowiki>馬克吐溫名言 "To a man with a hammer, everything looks like a nail."</nowiki>}} <li>after: {{kbd | key=<nowiki>
馬克吐溫名言 &amp;quot;To a man with a hammer, everything looks like a nail.&amp;quot;</nowiki>}}</ul>
</td>
</tr>
</tr>
</table>
</table>
Other functions
* [https://www.w3schools.com/js/js_json_parse.asp JSON.parse()] or [http://api.jquery.com/jquery.parsejson/ jQuery.parseJSON() | jQuery API Documentation]


=== String contains {{kbd | key=<nowiki>%2</nowiki>}} or {{kbd | key=<nowiki>%20</nowiki>}} symbols ===
=== String contains {{kbd | key=<nowiki>%2</nowiki>}} or {{kbd | key=<nowiki>%20</nowiki>}} symbols ===
Line 71: Line 148:
echo "encoded string: " . json_encode("🐘") . PHP_EOL; // print "\ud83d\udc18"
echo "encoded string: " . json_encode("🐘") . PHP_EOL; // print "\ud83d\udc18"
</pre>
</pre>
when using the heredoc syntax (<<<EOT ... EOT;), it's possible that unnecessary whitespace or hidden characters at the beginning or end of the block might cause json_decode to fail in parsing the string correctly. Direct assignment avoids potential whitespace or format issues from heredoc.
<pre>
$encoded = '"\u8c61"';
echo "decoded string: " . json_decode($encoded, true) . PHP_EOL; // print 象
echo "encoded string: " . json_encode("象") . PHP_EOL; // print "\u8c61"
</pre>


Using PHP v. 7.0 [https://wiki.php.net/rfc/unicode_escape Unicode Codepoint Escape Syntax]<ref>[https://secure.php.net/manual/en/migration70.new-features.php#migration70.new-features.unicode-codepoint-escape-syntax PHP: New features - Manual]</ref>
Using PHP v. 7.0 [https://wiki.php.net/rfc/unicode_escape Unicode Codepoint Escape Syntax]<ref>[https://secure.php.net/manual/en/migration70.new-features.php#migration70.new-features.unicode-codepoint-escape-syntax PHP: New features - Manual]</ref>
Line 160: Line 245:
=== String starting from &# symbols ===
=== String starting from &# symbols ===
Using PHP [https://www.w3schools.com/php/func_string_html_entity_decode.asp html_entity_decode() Function]<ref>[https://blog.longwin.com.tw/2011/06/php-html-unicode-convert-2011/ PHP 將 文字 轉換成 &#xxxxx; UNICODE 碼 | Tsung's Blog]</ref><ref>[http://hinablue.blogspot.com/2008/01/php-tech-unicode-html-convert.html [php tech.] unicode html convert | HINA::工程幼稚園] unicode html 字碼來元是由原本的編碼，轉換為 UCS-2 之後，再取二進制轉換，再取一次 16 to 10 進制轉換，在加上 &# 而得到這個字碼。</ref>
Using PHP [https://www.w3schools.com/php/func_string_html_entity_decode.asp html_entity_decode() Function]<ref>[https://blog.longwin.com.tw/2011/06/php-html-unicode-convert-2011/ PHP 將 文字 轉換成 &#xxxxx; UNICODE 碼 | Tsung's Blog]</ref><ref>[http://hinablue.blogspot.com/2008/01/php-tech-unicode-html-convert.html [php tech.] unicode html convert | HINA::工程幼稚園] unicode html 字碼來元是由原本的編碼，轉換為 UCS-2 之後，再取二進制轉換，再取一次 16 to 10 進制轉換，在加上 &# 而得到這個字碼。</ref>
To decode the text
<pre>
<pre>
$unicode_html = '&amp;#128024;';
$unicode_html = '&amp;#128024;';
Line 166: Line 253:
$unicode_html = '&amp;#128024;';
$unicode_html = '&amp;#128024;';
echo mb_convert_encoding($unicode_html, 'UTF-8', 'HTML-ENTITIES') . PHP_EOL; // print 🐘
echo mb_convert_encoding($unicode_html, 'UTF-8', 'HTML-ENTITIES') . PHP_EOL; // print 🐘
</pre>


To encode the text
<pre>
$input = "🐘";
$input = "🐘";
$unicode_html = base_convert(bin2hex(mb_convert_encoding($input, 'UTF-32', 'utf-8')), 16, 10);
$unicode_html = base_convert(bin2hex(mb_convert_encoding($input, 'UTF-32', 'utf-8')), 16, 10);
Line 240: Line 330:
* [http://en.wikipedia.org/wiki/Character_encoding Character encoding - Wikipedia, the free encyclopedia]
* [http://en.wikipedia.org/wiki/Character_encoding Character encoding - Wikipedia, the free encyclopedia]
* [https://pjchender.blogspot.com/2018/06/guide-unicode-javascript.html (Guide) 瞭解網頁中看不懂的編碼：Unicode 在 JavaScript 中的使用 ~ PJCHENder 那些沒告訴你的小細節]
* [https://pjchender.blogspot.com/2018/06/guide-unicode-javascript.html (Guide) 瞭解網頁中看不懂的編碼：Unicode 在 JavaScript 中的使用 ~ PJCHENder 那些沒告訴你的小細節]
* [https://www.multiutil.com/base64-to-text-converter/ Base64 to Text Converter]
* [https://www.multiutil.com/gzip-to-text-decompress/ Gzip to Text Decompress using gzip, deflate and brotli algoithms]
* [[URL Encoding]]
* [[URL Encoding]]


Unicode table
Unicode table
Retrieved from "https://wiki.planetoid.info/index.php/Special:MobileDiff/22628...24656"

Navigation menu