In this blog post, we will examine the concept of big data, its 3V characteristics, storage, processing, and analysis technologies, the data production environment expanded by smartphones and social media, and use cases such as Seoul’s late-night buses, Google, and Amazon.
The Emergence and Definition of Big Data
Recently, Seoul City has been operating late-night buses, which have been very well received. So, how did Seoul City determine the routes and frequency of these late-night buses? To determine the routes, the Seoul Metropolitan Government analyzed areas with high call volumes using approximately 3 billion mobile call records from the late-night hours between midnight and 5 a.m. They then combined this data with 5 million records of late-night taxi pick-up and drop-off locations used by citizens, reanalyzed the results, and established optimal routes and frequency intervals with a focus on areas with high foot traffic. The vast amount of data used in this process—such as the 3 billion call records and 5 million boarding and alighting records—is referred to as “big data.”
Big data, as the term implies, refers to an extremely large volume of data, encompassing both structured and unstructured data that is difficult to process using conventional data collection, storage, and analysis tools. Nowadays, vast amounts of information are generated daily from various sources—such as news articles, credit card transaction records, and social media—making this an era of information overload. In this context, “big data” has emerged as a key concept.
Prior to the 2000s, producing data was not easy. Content flowed in a single direction, with experts or a select few producing it while the majority consumed it. However, with the emergence of blogs and online communities, followed by the rise of social media and the widespread adoption of smartphones in recent years, anyone can now easily produce data. Furthermore, as the internet has evolved into a user-participation-centered environment where data can be easily shared, data now flows in both directions. Big Data emerged as a result of these diverse data producers and sharers.
Characteristics, Technologies, and Applications of Big Data
The characteristics of big data can be summarized as the 3Vs. Here, the 3Vs refer to volume, velocity, and variety. First, volume refers to the large quantity of data being produced. Traditional data analysis dealt with terabytes of data. However, big data analysis often handles petabytes of data—approximately 1,000 times the volume of traditional data.
Another key characteristic of big data, Velocity, refers to data processing capability. The process of collecting, processing, and analyzing data must be performed either at regular intervals or in real time. Finally, Variety refers to the diverse forms of data. While traditional data analysis primarily involved structured numerical data, big data analysis includes not only structured data but also unstructured data—such as text, video, and location data—that lacks a fixed format.
Although it has been quite some time since big data began to emerge, the reason it has recently garnered significant attention is that technologies capable of analyzing and processing such data have been extensively developed, broadening its scope of application.
Big data technologies can be broadly categorized into big data storage technologies, big data processing technologies, and big data analysis technologies.
Big data storage technologies have already reached a high level of maturity thanks to companies like Google, Apple, and Yahoo. Open-source examples include HDFS from Hadoop. Such programs enable the distributed storage of large volumes of data.
Data processing technology refers to the ability to perform large-scale data cleansing within a reasonable timeframe, made possible by the widespread adoption of the MapReduce programming model proposed by Google. MapReduce divides big data and performs processing on the data stored on each of the hundreds or thousands of servers where it is distributed.
Data analysis technology refers to the process of storing data and analyzing all or part of the relevant data to extract trends and meaningful values, or to discover previously unknown facts and transform them into knowledge. To achieve this, technologies from various fields—such as artificial intelligence, machine learning, statistics, and databases—are comprehensively applied, with the R programming language being a prime example. R is a program that loads all the data to be processed into the memory of a single computer and analyzes it using a single CPU. As technologies capable of analyzing and processing big data have been developed, it has become possible to utilize big data even more effectively.
So, how can this processed information actually be utilized? Big data is being utilized in both the private and public sectors.
Google is a leader in big data applications. Through its internet search engine, Google demonstrates that the more data there is, the higher the quality of the information that can be obtained. It scans every accessible webpage to measure how closely the title and content relate to the search query, converting this into an index. Google’s automatic translation system is another example of big data utilization. Google refers to the technology behind its automatic translation system as “statistical machine translation.” This approach does not teach computers grammar but instead allows them to discover translation rules between languages by analyzing patterns in hundreds of millions of documents already translated by humans. The more reference documents available, the higher the likelihood of improved translation quality.
In addition to Google, Amazon, a pioneer in online shopping, also has a long history of utilizing big data. Amazon developed a system that analyzes customers’ book purchase data to recommend books that people who bought a specific title are likely to purchase next. This is a typical data-driven marketing method that recommends books customers are expected to read while offering discount coupons.
Big data is being utilized not only in the private sector but also in the public sector. In addition to the late-night buses in Seoul mentioned earlier, notable examples include the governments of Singapore and the United States. They are utilizing big data in the fields of security and risk management. Since 2004, the Singaporean government has been implementing the National Risk Management System (RAHS, Risk Assessment & Horizon Scanning) to prepare for an uncertain future, such as disaster prevention, terrorism detection, and the spread of infectious diseases. It collects and analyzes various national risk data to make predictions in advance and explore response strategies. The U.S. Federal Bureau of Investigation (FBI)’s DNA indexing system is another example of how big data is used to apprehend criminals quickly.
As such, big data is already being utilized in many areas, and its scope of application is expected to expand in the future. In the past, producing and sharing data was not easy, but now that the process has become much simpler, many people are producing and sharing data, leading to an explosive increase in the volume of data itself. As a result, big data has emerged, characterized by vast volumes, rapid collection and analysis speeds, and diverse data formats. With the advancement of technologies capable of processing these massive amounts of data, big data has recently garnered even more attention and is often referred to as the “crude oil” of the 21st century. Just as crude oil has been used in every aspect of daily life, big data will become a vital foundation in nearly every field in the future.
