Anomaly detection: Difference between revisions
No edit summary |
|||
| (11 intermediate revisions by the same user not shown) | |||
| Line 7: | Line 7: | ||
* All values is event or odd | * All values is event or odd | ||
* The values are the same even the column is totally different | * The values are the same even the column is totally different | ||
== Anomaly detection of categorical data (qualitative variable) == | == Anomaly detection of categorical data (qualitative variable) == | ||
| Line 15: | Line 12: | ||
== Anomaly detection for time series data == | == Anomaly detection for time series data == | ||
* Trend | * Trend: For example, holiday and seasonal factors typically drive consistent, predictable shopping behavior, but an unexpected shift in product category suddenly occurs | ||
* | * Dramatic increase or decrease in row count for each time period | ||
** Example: A regularly scheduled web scraper that normally collects 9k records per week suddenly drops to 3k records | |||
* Future data: Data timestamps fall outside (after) the expected time range | |||
== Anomaly detection for | == Anomaly detection for consumer data == | ||
For consumer data | |||
* | * Season issue: consumption data of coat should increase in cold weather | ||
* | * Holiday issue: consumption data of some gift e.g. moon cake should increase in special holiday e.g. Mid-Autumn Festival | ||
* | |||
== Anomaly detection for string data == | |||
* Created time of the text message | |||
* Time frequency of the text message | |||
* Length of the text message | |||
* NULL or empty value | |||
* Minor differences of text content<ref>[https://medium.com/@ahmetmnirkocaman/how-to-measure-text-similarity-a-comprehensive-guide-6c6f24fc01fe How to Measure Text Similarity: A Comprehensive Guide | by Ahmet Münir Kocaman | Medium]</ref> | |||
* Character encoding e.g. [[Fix garbled message text]] | |||
More on: [https://en.wikipedia.org/wiki/Outlier#Identifying_outliers Outlier - Wikipedia] | More on: [https://en.wikipedia.org/wiki/Outlier#Identifying_outliers Outlier - Wikipedia] | ||
[[Category: | == Further Reading == | ||
[[Category:Data Science]] | |||
# [https://www.amazon.com/Bad-Data-Handbook-Cleaning-Back-ebook/dp/B00A3IGAIA Amazon.com: Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work eBook : McCallum, Q. Ethan: Kindle Store] (ISBN: 9781449324964) | |||
# [https://www.oreilly.com/library/view/bad-data-ji-shu-shou-ce/9789862768952/ Bad Data 技術手冊 [Book]] (ISBN: 9789862768952) | |||
== References == | |||
<references /> | |||
[[Category: Data hygiene]] | |||
[[Category: Data Science]] | |||
Latest revision as of 16:23, 6 September 2026
Outlier / Anomaly detection
Anomaly detection of numeric data[edit]
- Median
- Range Checks
- All values is event or odd
- The values are the same even the column is totally different
Anomaly detection of categorical data (qualitative variable)[edit]
- Normal distribution e.g. The interest of audiences should be very different NOT coherent
Anomaly detection for time series data[edit]
- Trend: For example, holiday and seasonal factors typically drive consistent, predictable shopping behavior, but an unexpected shift in product category suddenly occurs
- Dramatic increase or decrease in row count for each time period
- Example: A regularly scheduled web scraper that normally collects 9k records per week suddenly drops to 3k records
- Future data: Data timestamps fall outside (after) the expected time range
Anomaly detection for consumer data[edit]
For consumer data
- Season issue: consumption data of coat should increase in cold weather
- Holiday issue: consumption data of some gift e.g. moon cake should increase in special holiday e.g. Mid-Autumn Festival
Anomaly detection for string data[edit]
- Created time of the text message
- Time frequency of the text message
- Length of the text message
- NULL or empty value
- Minor differences of text content[1]
- Character encoding e.g. Fix garbled message text
More on: Outlier - Wikipedia
Further Reading[edit]
- Amazon.com: Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work eBook : McCallum, Q. Ethan: Kindle Store (ISBN: 9781449324964)
- Bad Data 技術手冊 [Book] (ISBN: 9789862768952)