Find and remove duplicates: Difference between revisions
Jump to navigation
Jump to search
m
→Remove duplicate values in Cygwin/BASH
| Line 22: | Line 22: | ||
[http://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html GNU Coreutils: sort invocation] OS: {{Linux}}, cygwin of {{Win}}. | [http://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html GNU Coreutils: sort invocation] OS: {{Linux}}, cygwin of {{Win}}. | ||
Data preparation: | |||
* [[Alternative_Linux_commands#Merge_multiple_plain_text_files | Merge multiple plain text files]] | |||
case 1: To remove duplicate lines of entire paragraph | case 1: To remove duplicate lines of entire paragraph | ||
| Line 30: | Line 33: | ||
* (1) ignore first one line: {{kbd | key=<nowiki>(head -n 1 <file> && tail -n +2 <file> | sort -us) > newfile</nowiki>}} | * (1) ignore first one line: {{kbd | key=<nowiki>(head -n 1 <file> && tail -n +2 <file> | sort -us) > newfile</nowiki>}} | ||
* (2) ignore first two lines: {{kbd | key=<nowiki>(head -n 2 <file> && tail -n +3 <file> | sort -us) > newfile</nowiki>}} | * (2) ignore first two lines: {{kbd | key=<nowiki>(head -n 2 <file> && tail -n +3 <file> | sort -us) > newfile</nowiki>}} | ||
== Find and remove duplicates in MySQL == | == Find and remove duplicates in MySQL == | ||