This blog post examines whether the potential of big data hinges on technology or on the analyst’s insight and capabilities.
Just five years ago, big data was a niche concept known only to a select few. At the time, while information and technology were advancing rapidly and the volume of data was exploding, there was little thought given to how this data could be efficiently utilized. Traditional data analysis methods couldn’t handle the sheer volume, and as a result, much of the data was simply collected and left unused or used only as basic statistical material. However, the emergence of the big data concept gradually began to change this situation.
In 2012, big data was selected as one of the top ten technologies alongside the Internet of Things at the World Economic Forum, drawing significant attention. As interest in big data grew, attempts to actively utilize it increased in various sectors, including businesses. Big data was not merely a product of technological advancement; it was a concept with the potential to bring significant change across society. As the realization spread that data could be used to understand and predict people’s behavioral patterns, big data soon began to establish itself as a strategic tool for businesses.
In Korea, Seoul City estimates nighttime population flow using citizens’ late-night taxi boarding and alighting records and KT call volume data. Operating night buses by presenting the most suitable route maps and dispatch schedules based on this data is a prime example of big data utilization. Beyond this, experiments and challenges exploring big data’s potential continue across various industries in Korea. For instance, in healthcare, research is actively underway to analyze patient treatment records and lifestyle data to propose personalized treatment plans. In this way, big data is breaking the molds of traditional industries and enabling new innovations.
Samsung SDS’s smart factory solution reportedly predicts equipment failures by analyzing real-time operational data and historical defect records. Nevertheless, the question “Has big data significantly changed the world?” cannot be readily answered in the affirmative. Some argue that its impact remains largely unseen because big data is primarily used internally within companies. Additionally, some experts point to data security and privacy concerns as one reason big data’s potential remains largely untapped. As data volumes increase, worries about personal information protection grow, potentially hindering corporate big data adoption. Without progress in policies and technologies addressing these ethical issues, big data utilization will inevitably remain limited.
However, according to a survey by the Korea Information Society Development Institute, which found that only 4% of Korean companies utilize big data, it appears that big data is not being effectively used within companies either. So, why are companies failing to achieve significant results with this seemingly limitless potential of big data? Is it because the technology to manage and analyze big data is still lacking? I believe the reason for this problem lies not in technological limitations, but in the insufficient insight and capabilities of the people utilizing big data. This connects to the problem of ‘information overload’ that big data presents. Extracting meaningful information from vast data requires more than just simple technology; it demands insight. Perhaps due to the intimidating nature of the term ‘big data,’ many assume it requires immense technical skill. However, big data simply means massive amounts of data – internet search histories, SNS posts, CCTV footage, and so on. With servers to store this big data and software tools to analyze it, anyone can perform big data analysis. Of course, various technologies are being developed to efficiently store larger data sets and process them quickly, but these do not inherently make analysis better. Ultimately, the analyst’s role is paramount: determining what data to analyze from a mass of worthless raw data, judging its significance, and transforming it into valuable information. Even in the late-night bus example mentioned earlier, what others might see as mere call volume data can reveal information about population flow when viewed and analyzed correctly. This population flow data can then be used to determine optimal bus routes and scheduling intervals.
I became convinced of this perspective during my six-month internship at a big data research institute. The most memorable data I handled during that internship was Seoul National University’s KT free Wi-Fi access data. This dataset contained the IDs of all devices connecting to SNU’s Wi-Fi, along with their connection start and end times. Analyzing the data itself wasn’t difficult. The problem was a lack of fresh perspectives on how to view the data. The greatest challenge in big data analysis lies not in ‘how’ to analyze the data, but in deciding ‘what’ to analyze. While I successfully calculated Seoul National University’s floating population using this data, I couldn’t think of a useful application for it. Ultimately, until I finished my internship, this Wi-Fi access data remained just a mineral, never becoming a gem.
Thus far, we’ve described how big data utilization often falls short not due to technical limitations, but rather the analyst’s lack of capability. So, how can big data be analyzed effectively? Kevin Lyons, Senior VP at eXelate, a digital marketing data management company, emphasized that diversity is the most crucial factor in building a big data analytics team. He reportedly strives to staff data analysis teams not only with computer experts but also with individuals from diverse backgrounds like accounting and insurance. This diversity extends beyond mere technical knowledge; it enhances problem-solving abilities by bringing in different perspectives and experiences. When people from diverse fields collaborate to analyze big data, they can devise ways to utilize data in multiple areas that an individual might never conceive alone. Of course, individual analysts must also possess broad knowledge across society and a deep understanding of their industry.
New types of data continue to pour in constantly. It is highly anticipated how data experts, honed by extensive experience and collaboration, will transform the world through big data. Moving forward, big data holds the potential to transcend being a mere trend and become a core driver of innovation. Realizing this potential requires the creative thinking and collaboration of analysts.
