https://wiki.planetoid.info/api.php?action=feedcontributions&feedformat=atom&user=2001%3AB011%3AD804%3A9E4D%3A75B0%3A760%3A554F%3A27C4 2026-09-10T12:49:23Z User contributions MediaWiki 1.43.9 https://wiki.planetoid.info/index.php?title=Anomaly_detection&diff=26450 2026-09-06T08:23:24Z

<p>2001:B011:D804:9E4D:75B0:760:554F:27C4: </p> <hr /> <div><br /> Outlier / Anomaly detection<br /> <br /> == Anomaly detection of numeric data ==<br /> * Median<br /> * Range Checks<br /> * All values is event or odd<br /> * The values are the same even the column is totally different<br /> <br /> == Anomaly detection of categorical data (qualitative variable) ==<br /> * Normal distribution e.g. The interest of audiences should be very different NOT coherent<br /> <br /> == Anomaly detection for time series data ==<br /> * Trend: For example, holiday and seasonal factors typically drive consistent, predictable shopping behavior, but an unexpected shift in product category suddenly occurs<br /> * Dramatic increase or decrease in row count for each time period<br /> ** Example: A regularly scheduled web scraper that normally collects 9k records per week suddenly drops to 3k records<br /> * Future data: Data timestamps fall outside (after) the expected time range<br /> <br /> == Anomaly detection for consumer data ==<br /> For consumer data<br /> <br /> * Season issue: consumption data of coat should increase in cold weather<br /> * Holiday issue: consumption data of some gift e.g. moon cake should increase in special holiday e.g. Mid-Autumn Festival<br /> <br /> == Anomaly detection for string data ==<br /> <br /> * Created time of the text message<br /> * Time frequency of the text message<br /> * Length of the text message<br /> * NULL or empty value<br /> * Minor differences of text content&lt;ref&gt;[https://medium.com/@ahmetmnirkocaman/how-to-measure-text-similarity-a-comprehensive-guide-6c6f24fc01fe How to Measure Text Similarity: A Comprehensive Guide | by Ahmet Münir Kocaman | Medium]&lt;/ref&gt;<br /> * Character encoding e.g. [[Fix garbled message text]]<br /> <br /> More on: [https://en.wikipedia.org/wiki/Outlier#Identifying_outliers Outlier - Wikipedia]<br /> <br /> == Further Reading ==<br /> <br /> # [https://www.amazon.com/Bad-Data-Handbook-Cleaning-Back-ebook/dp/B00A3IGAIA Amazon.com: Bad Data Handbook: Cleaning Up The Data So You Can Get Back To Work eBook : McCallum, Q. Ethan: Kindle Store] (ISBN: 9781449324964)<br /> # [https://www.oreilly.com/library/view/bad-data-ji-shu-shou-ce/9789862768952/ Bad Data 技術手冊 [Book]] (ISBN: 9789862768952)<br /> <br /> == References ==<br /> <br /> &lt;references /&gt;<br /> <br /> [[Category: Data hygiene]]<br /> [[Category: Data Science]]</div>

2001:B011:D804:9E4D:75B0:760:554F:27C4