Visualization of Uncertainties and Noise in Dark Data: Methods & Techniques

Abstract

Big Data generated in today’s digital world is overwhelming the capacity of institutions and researchers managing and making use of it, leading into a new crisis called the data deluge. Forbes estimates 2.5 quintillion bytes of data created every day, 90% of it being unexplored and unanalyzed complex Dark Data. As much as 80% of business organisations rely on costly manual processes for locating, organizing and processing a small fraction of such untapped data. The lack of automation
tools to improve the productivity and Dark Data utilization motivates to explore domain and define problems. This work aims to minimize expensive storage and security issues and reduces dark data volume which is still unanalyzed and disvalued. The methodology proposed to improve spatial-temporal efficiency by injecting intelligent insights into diverse and continuously evolving data silos by uncovering and transforming the value of unanalyzed Dark Data through application of Cognitive
Visualytics and Intelligent Process Automation (IPA) techniques. Moreover, the work intends to apply cognitive visualization techniques to finding the solution of the uncertainty faced in dark data leading to the conclusion of the work by offering not only dynamic, interpretable and multidimensional Dark Data visualizations, but will also allow diverse data providers make an informed decision by predicting Dark Data analysis.