Anomaly detection
Outlier / Anomaly detection
Anomaly detection of numeric data
- Median
- Range Checks
- All values is event or odd
- The values are the same even the column is totally different
Anomaly detection of categorical data (qualitative variable)
- Normal distribution e.g. The interest of audiences should be very different NOT coherent
Anomaly detection for time series data
- Trend: For example, holiday and seasonal factors typically drive consistent, predictable shopping behavior, but an unexpected shift in product category suddenly occurs
- Dramatic increase or decrease in row count for each time period
- Example: A regularly scheduled web scraper that normally collects 9k records per week suddenly drops to 3k records
- Future data: Data timestamps fall outside (after) the expected time range
Anomaly detection for consumer data
For consumer data
- Season issue: consumption data of coat should increase in cold weather
- Holiday issue: consumption data of some gift e.g. moon cake should increase in special holiday e.g. Mid-Autumn Festival
Anomaly detection for string data
- Created time of the text message
- Time frequency of the text message
- Length of the text message
- NULL or empty value
- Minor differences of text content[1]
- Character encoding e.g. Fix garbled message text
More on: Outlier - Wikipedia
Further Reading
- Amazon.com: Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work eBook : McCallum, Q. Ethan: Kindle Store (ISBN: 9781449324964)
- Bad Data 技術手冊 [Book] (ISBN: 9789862768952)