This blog post examines how big data has become a crucial factor determining innovation and competitiveness in our society, using various case studies.
The hot topic in the IT industry today is undoubtedly big data. Simply put, big data refers to large volumes of information. In the past, managing and utilizing large datasets was difficult, so they were often merely stored or used only by a few companies or institutions. However, with recent technological advancements, big data has now become an essential concept mentioned in nearly every field.
Big Data refers to massive collections of structured or unstructured data that exceed the capabilities of traditional database management tools for data collection, storage, management, and analysis. Structured data refers to organized data like numbers, while unstructured data includes text, images, videos, and other non-organized information. As data forms become increasingly diverse and complex, processing big data presents intricate technical challenges beyond simple data management. While it holds vast amounts of information commensurate with its size, unrefined data is merely useless clutter occupying memory space. Consequently, the meaning of the term “big data” has expanded beyond simply referring to large volumes of information itself to encompass the techniques used to analyze it and derive meaningful results.
The dramatic reduction in storage costs driven by advancements in data storage technology has enabled the creation of this big data, and the volume of accumulated data continues to grow exponentially. For instance, it is said that the amount of information accumulated over the 5 million years since Homo sapiens sapiens, humanity’s direct ancestors, first appeared on Earth, has doubled in just over a year and a half. This fact starkly illustrates the explosive growth of large-scale data. This exponentially increasing volume of information deeply impacts the lives of businesses and individuals, ushering in an era where data itself is considered an asset. Even at this very moment, our information is constantly transmitted and recorded through devices like smartphones, building yet another database.
Current trends in big data are even reshaping market dominance. Until recently, information power was paramount in the marketplace. Who possessed what information became the benchmark for competitiveness. However, in today’s deluge of data, where vast amounts remain buried and unused, the importance of how information is handled has come to the fore. In data analysis, it’s not just about accuracy; it’s crucial to efficiently analyze vast volumes of big data in real time to derive real-time strategies. Even the most precise analysis is useless if it can’t keep pace with rapidly changing trends. Consequently, ‘Big Data Analytics’—the methodology for analyzing and utilizing big data—is gaining significant attention. This goes beyond simple data analysis; it involves extracting insights from massive datasets and applying them in real time.
Big Data Analytics is the field concerning these techniques. Analyzing data volumes beyond common sense in real time, big data analytics is akin to mining sparkling gems from a dark, murky data mine. For this reason, Big Data Analytics is also called ‘Data Mining’. Data mining is the technology for discovering meaningful patterns within big data, playing an essential role in helping companies secure competitiveness amidst a flood of information.
Examples of data mining applications are also frequently seen in product marketing. In marketing, it is crucial to quickly gather consumer opinions and derive appropriate product strategies based on them. Knowing sales information about who buys what and when makes it easier to select key customer segments and conduct effective marketing. For retailers like large supermarkets, loyalty cards are the primary channel for obtaining this information. Most people have experienced being asked, “Do you have a loyalty card?” when shopping. Even if we don’t have the card with us, the fact that points can be accumulated simply by entering a phone number shows how flexibly this system collects customer information.
A company’s point accumulation strategy aims to digitally link the personal information stored on our point cards with the products we purchase. In other words, retailers earn points for customers while simultaneously gathering data on who buys what, when, and in what quantities. This data is crucial for product marketing. However, until recently, manufacturers could only supply products to retailers like large supermarkets and had no way to access this customer information. Therefore, manufacturers found it difficult to obtain high-quality material for marketing.
However, retailers, who already possess extensive customer information, make the following proposals to manufacturers: “We will provide you with the customer information we hold!” “We will develop marketing strategies for your products using our information.” By making such proposals, retailers have secured a favorable position in their dealings with manufacturers. Thus, distributors have generated substantial profits not only through distribution but also by leveraging customer information. As this example shows, until recently, companies possessing information held hegemony in market competition.
However, with the massive rise of social media like SNS and blogs, manufacturers have launched a counterattack. Even if manufacturers cannot directly reach customers, they can now hear customer feedback about their products through the open information source of the internet. Consequently, competition between companies has shifted beyond simply possessing information to effectively utilizing it, altering the dynamics of dominance.
A prime example of this reversal is the instant rice product, Haetban. While large supermarkets and distributors knew which customers purchased Haetban, there was one crucial piece of information they lacked: why customers bought and consumed it. CJ, the Korean manufacturer of Hapban, collected social media posts about the product and analyzed them using data mining techniques for handling unstructured text data. Before the analysis, many predicted Hapban consumption primarily occurred outside the home, such as during travel. However, the manufacturer’s analysis revealed that Hapban was frequently used by mothers to feed their families when there was no rice prepared at home. The data mining analysis didn’t stop there. Sentiment analysis of the words in the text also revealed that the mothers preparing Hapban meals for their families felt guilty. Based on these findings, the manufacturer created advertisements and conducted marketing campaigns. This is reflected in the recent Hapban ad copy: “Don’t feel guilty about the rice your mom made!” Advertisements that effectively reflected these trend analysis results significantly boosted product profits.
Manufacturers who effectively utilized the vast amount of information publicly available online through data mining techniques were able to conduct successful marketing without relying on transactions with distributors. In this way, within the given data mine, the hegemony among companies in market competition is shifting based on who can best utilize data mining techniques to uncover the gems.
The utilization of big data is no longer confined to specific companies. The public sector is also employing big data for diverse policy development and service improvements. Governments are actively leveraging big data to devise solutions for social issues based on large-scale data and to provide customized services to citizens. This data-driven approach enables more efficient and transparent administration while contributing to improved quality of life for citizens. Furthermore, big data is bringing significant changes to the medical field. By collecting and analyzing individual health data, it enables personalized treatment and prevention, predicts the spread of infectious diseases, and allows for rapid response.
Ultimately, despite its vast volume and complexity, big data can be a powerful tool capable of driving innovation across our entire society, provided it is handled correctly. Amidst this ever-growing flood of information, businesses, governments, and even individuals must recognize the importance of big data and seek ways to utilize it effectively.
To keep pace with these changes, continuous research and technological development in big data are essential. Alongside the advancement of big data analysis techniques, discussions on the ethical use of data are also becoming increasingly important. Protecting individual privacy and ensuring transparent data usage are critical challenges we must address in the big data era. To resolve these issues, legal and social discussions must proceed alongside technological advancements.
In conclusion, big data is often called the new oil of the 21st century. Its value depends not only on the sheer volume of data but also on how effectively it is utilized. The ability to handle data has now become a crucial factor determining the competitiveness of companies and nations. Recognizing the importance of big data and developing strategies for its utilization is the path to preparing for the future.
