Anomaly detection

Outlier / Anomaly detection

Anomaly detection of numeric data

• Median
• Range Checks
• All values is event or odd
• The values are the same even the column is totally different

Anomaly detection of categorical data (qualitative variable)

• Normal distribution e.g. The interest of audiences should be very different NOT coherent

Anomaly detection for time series data

• Trend: For example, holiday and seasonal factors typically drive consistent, predictable shopping behavior, but an unexpected shift in product category suddenly occurs
• Dramatic increase or decrease in row count for each time period
  • Example: A regularly scheduled web scraper that normally collects 9k records per week suddenly drops to 3k records
• Future data: Data timestamps fall outside (after) the expected time range

Anomaly detection for consumer data

For consumer data

• Season issue: consumption data of coat should increase in cold weather
• Holiday issue: consumption data of some gift e.g. moon cake should increase in special holiday e.g. Mid-Autumn Festival

Anomaly detection for string data

• Created time of the text message
• Time frequency of the text message
• Length of the text message
• NULL or empty value
• Minor differences of text content
• Character encoding e.g. Fix garbled message text

More on: Outlier - Wikipedia

Further Reading

1. Amazon.com: Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work eBook : McCallum, Q. Ethan: Kindle Store (ISBN: 9781449324964)
2. Bad Data 技術手冊 [Book] (ISBN: 9789862768952)