Editing Anomaly detection


Outlier / Anomaly detection

== Anomaly detection of numeric data ==
* Median
* Range Checks
* All values is event or odd
* The values are the same even the column is totally different

== Anomaly detection of categorical data (qualitative variable) ==
* Normal distribution e.g. The interest of audiences should be very different NOT coherent

== Anomaly detection for time series data ==
* Trend: For example, holiday and seasonal factors typically drive consistent, predictable shopping behavior, but an unexpected shift in product category suddenly occurs
* Dramatic increase or decrease in row count for each time period
** Example: A regularly scheduled web scraper that normally collects 9k records per week suddenly drops to 3k records
* Future data: Data timestamps fall outside (after) the expected time range

== Anomaly detection for consumer data ==
For consumer data

* Season issue: consumption data of coat should increase in cold weather
* Holiday issue: consumption data of some gift e.g. moon cake should increase in special holiday e.g. Mid-Autumn Festival

== Anomaly detection for string data ==

* Created time of the text message
* Time frequency of the text message
* Length of the text message
* NULL or empty value
* Minor differences of text content<ref>[https://medium.com/@ahmetmnirkocaman/how-to-measure-text-similarity-a-comprehensive-guide-6c6f24fc01fe How to Measure Text Similarity: A Comprehensive Guide | by Ahmet Münir Kocaman | Medium]</ref>
* Character encoding e.g. [[Fix garbled message text]]

More on: [https://en.wikipedia.org/wiki/Outlier#Identifying_outliers Outlier - Wikipedia]

== Further Reading ==

# [https://www.amazon.com/Bad-Data-Handbook-Cleaning-Back-ebook/dp/B00A3IGAIA Amazon.com: Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work eBook : McCallum, Q. Ethan: Kindle Store] (ISBN: 9781449324964)
# [https://www.oreilly.com/library/view/bad-data-ji-shu-shou-ce/9789862768952/ Bad Data 技術手冊 [Book]] (ISBN: 9789862768952)

== References ==

<references />

[[Category: Data hygiene]]
[[Category: Data Science]]