https://wasi0013.com/categories/data-mining--analysis/Recent content in Data Mining & Analysis on Wasi's LogHugoenSun, 21 Sep 2025 19:22:41 +0600https://wasi0013.com/2025/09/21/detecting-and-fixing-data-quality-problems-in-data-pipelines-and-ml-systems/Sun, 21 Sep 2025 19:22:41 +0600https://wasi0013.com/2025/09/21/detecting-and-fixing-data-quality-problems-in-data-pipelines-and-ml-systems/<p>Early in my career, I built many web scrapers, some of these early scrapy scrapers used to aggregate product pricing data from multiple e-commerce websites. The initial dataset looked complete, and basic analysis suggested reliable trends. Once I integrated the data into a dashboard for decision-making, inconsistencies appeared due to slight differences in HTML structure across sites. These hidden issues caused subtle quiet failure, but the impact stacked up quickly after weeks. This was a nice example of the hidden risk of poor data quality problems in data pipelines.</p>