Blog

While the previous post covered the conceptual importance of ETL (Extract, Transform, Load) in the context of the paradigm shift in materials research, this post introduces a robust architecture based on the Elastic Stack to bring those concepts to life.

Data generated in materials research fields is incredibly diverse in both type and format. In particular, time-series data streaming in real-time from experimental equipment is intricately intertwined with various process conditions controlled by researchers. Manually collecting and matching this data not only carries a high risk of data loss but also consumes an enormous amount of time when attempting to analyze the root causes of experimental results.

To solve these challenges, the Elastic Stack (ELK Stack) offers unparalleled utility for managing experimental data in the materials science sector. The Elastic Stack goes beyond the concept of a simple data lake for storing numerical values; it provides an integrated pipeline capable of instantly ingesting real-time experimental data and organically enriching it with the underlying process variables (such as temperature, pressure, and speed).

The ability to perform real-time integrated management of the correlation between process variables and measurement results—the core of materials research—and immediately convert them into machine-learning-ready datasets makes it the most powerful engine for Data-driven Materials Science.

Building on this utility, let’s explore a specific plan for constructing an ETL pipeline that utilizes the Elastic Stack to collect and process measurement data from experimental equipment and core process conditions in real-time.

Materials Data Pipeline Architecture Based on the Elastic Stack

The Elastic Stack provides an integrated pipeline—spanning collection (Beats/Logstash), storage and analysis (Elasticsearch), and visualization (Kibana)—making it the optimal tool for transforming fragmented experimental data into “AI-ready data” and, ultimately, “actionable knowledge.” The flow from laboratory equipment to the database consists of three main stages: Collection, Refinement/Transformation, and Storage/Indexing. Each component performs the following specialized functions:

• Beats (Lightweight Shippers): Real-time Data Capture and Transmission Beats act as agents installed on the control PCs connected to various experimental instruments (such as GC, XRD, SEM, etc.). They immediately detect when a device generates experimental results in formats like CSV, TXT, XML, or log files. With an extremely low resource footprint, they do not interfere with the performance of precision instruments. Furthermore, they ensure reliable data transmission through “back-pressure sensitivity,” which holds and retransmits data in the event of network failures.
• Logstash (Data Processing Engine): Process Condition Integration and AI-Ready Processing Logstash is the core engine that turns raw data into knowledge. Moving beyond simple data migration, it performs data processing tasks essential for materials research. For example, it can enrich raw voltage data output by a device with external “process condition” metadata, such as the experimental temperature at that moment, the concentration of materials used, and researcher information. It also standardizes data formats from different instruments into a single JSON schema, restructuring it into a state that AI models can immediately use for training.
• Elasticsearch (Distributed Search and Analytics Engine): High-speed Indexing and Multi-dimensional Querying Elasticsearch acts as the “brain,” storing and managing large-scale processed materials data. To handle millions of data points, Elasticsearch utilizes distributed storage to dramatically increase search speeds. Through advanced indexing technology, it provides instantaneous responses even when researchers perform complex, multi-dimensional queries, such as: “Show all experimental results where the additive ratio was over 10% and the test was conducted at temperatures above 50°C.”
• Kibana (Visualization Platform): Real-time Monitoring and Insight Generation Kibana serves as the interface for researchers to intuitively understand the data stored in Elasticsearch. By configuring real-time experimental values into graphs or dashboards, it allows researchers to immediately verify if an experiment is proceeding as intended. Furthermore, it functions as a “command center” where researchers can compare and analyze trends in physical properties relative to changes in process variables across historical data, helping them formulate new hypotheses and gain deeper insights.

Detailed Strategy for Experimental Equipment Data Collection and Processing: Beats

Data collection from experimental equipment involves more than just moving files between devices; it requires a sophisticated design to ensure that complex experimental conditions and measured values are integrated without loss. To achieve this, a strategy can be employed that standardizes data generation using NOMAD CAMELS—an experimental instrument control library—and extracts it in real-time through the Beats family. This combination acts as a core process that integrates fragmented data pouring in from heterogeneous devices into a single, organic flow.

The process of automated data extraction through the integration of NOMAD CAMELS and Beats is as follows:

First, NOMAD CAMELS plays the role of establishing a standardized framework from the very moment data is generated. By providing unified control over heterogeneous instruments with different connection methods through a single Python interface, it outputs raw measurement data—which often comes in disparate formats—into user-defined structures (such as JSON or refined CSV) that are easy to analyze. Particularly useful is its metadata integration feature, which links core process conditions entered at the start of an experiment—such as sample name, set temperature, and pressure—with the measured values in real-time. This ensures that researchers can conveniently secure datasets where causal relationships are already linked, eliminating the need to manually cross-reference conditions for every individual data point later on.

The data generated in this manner is delivered to a central server in real-time via Filebeat installed on the instrument’s PC. Filebeat is a lightweight agent that does not strain PC performance; it detects new experimental values added to a file—even a single line—within milliseconds and initiates transmission. During this process, complex results consisting of multiple lines are grouped into single, meaningful “experimental events” using multiline processing features, thereby preserving the data’s context. Furthermore, it automatically reads information such as dates or sample numbers embedded in file names or paths and attaches them as tags, drastically reducing the need for manual classification by researchers.

Finally, by utilizing various collection tools such as Metricbeat, environmental variables that may affect experimental results can be managed in a multi-dimensional way.

• Environment and System Integration: By linking with IoT sensors, laboratory temperature and humidity data at the time of the experiment are aligned with the measurement data on the same timeline. Simultaneously, the status of the collection PCs is monitored in real-time to fundamentally prevent any data omissions.
• Ensuring Transmission Reliability: Even if the network becomes unstable, high reliability is guaranteed through the “back-pressure” function. This feature temporarily stores data locally and resumes transmission from the exact point of interruption as soon as the connection is restored, ensuring that not a single line of experimental data is lost.

In conclusion, the collaborative structure—where NOMAD CAMELS facilitates convenient data generation at the experimental site and Beats delivers it safely and swiftly—transforms all laboratory data into immediately analyzable assets the moment they are created. Through this strategy, researchers can be liberated from the tedious burden of manual data organization.

Process Condition Integration and Standardization via Logstash (Transformation)

If Beats is the transport team that securely carries laboratory data, Logstash acts as the processing center that transforms raw measurement data into useful information ready for immediate analysis. The greatest advantage of Logstash is that data transformation rules can be written easily and conveniently using configuration files similar to JSON, without the need for complex coding. Its intuitive structure, much like assembling Lego blocks, allows even non-IT experts to understand the transformation process at a glance and modify or deploy rules instantly as needed.

The core processing steps performed at this stage are as follows:

Precision Extraction of Unstructured Data (Grok Filter)

The text logs output by experimental equipment often contain a mix of essential research values, unnecessary symbols, and proprietary equipment formats. By utilizing Logstash’s Grok filter, researchers can precisely extract key metrics—such as voltage, current, viscosity, and temperature—from unstructured text blobs using regular expressions. For example, a string like “Temp: 25.4C, Volt: 3.2V” is automatically categorized into discrete numerical values like ‘Temperature: 25.4’ and ‘Voltage: 3.2’, making them ready for analysis.

Multi-dimensional Data Completion (Lookup/Enrich Function)

It is often difficult to grasp the full context of an experiment using only the raw measurement data. Logstash excels at enriching data by combining external information that was not included in the original instrument output at the time of measurement.

• Lookup Function: Based on the sample ID sent by the experimental equipment, Logstash retrieves information in real-time from external databases (DB) or Excel configuration files, such as the researcher’s name, raw material lot numbers, and the chemical composition of the samples.
• Enrich Function: It further layers the combined data with environmental variables collected earlier, such as the laboratory’s humidity or power status on the day of the experiment.

Generation of Complete Datasets and Standardization

Through this filtering and enrichment process, fragmented measurement results are transformed into a single, complete dataset where process variables and the experimental environment are perfectly integrated. Researchers no longer need to open multiple files for manual cross-referencing; with the standardized data provided by Logstash, they can immediately analyze “who experimented with which raw materials, under what environment, and what the results were.”

From a materials research perspective, this method is a vital technical means of securing various “hidden metadata” that are typically not recorded alongside actual measurement data. While the physical properties of materials can vary significantly depending on subtle environmental changes or the history of raw materials, traditional equipment is often limited to storing only measured numerical values. Data enhancement through Logstash plays a critical role in determining data reliability for future AI modeling or root-cause analysis by firmly binding all scattered laboratory contextual information to the measurement values.

Ultimately, Logstash serves as a decisive bridge that evolves raw laboratory data into refined knowledge for research insights by allowing complex data transformation processes to be managed transparently and simply.

Utilizing Elasticsearch’s Flexible Schema (Dynamic Mapping) (Loading)

Refined data from the Logstash processing center is ultimately stored in Elasticsearch. Materials research is characterized by frequently changing or adding measurement items whenever a new hypothesis is tested. Elasticsearch perfectly addresses the rigidity of traditional Relational Databases (SQL) by providing a Dynamic Mapping feature optimized for such a volatile research environment.

The reasons why Elasticsearch is the ideal technology for materials data management at this stage are as follows:

Seamless Compatibility with JSON, the Standard for Materials Data

The JSON (JavaScript Object Notation) format, widely used as the standard distribution system for materials information, is exceptionally well-suited for representing hierarchical data structures. Since Elasticsearch uses JSON as its native storage format, it accepts complex experimental data structures arriving from Logstash exactly as they are, without requiring additional conversion processes. This prevents information distortion during transmission and allows researchers to immediately project their intended data structures into the database.

Change-Resilient Schemaless Architecture

In traditional SQL approaches, adding even a single new process variable required the cumbersome task of modifying the entire database schema. In contrast, Elasticsearch’s Schemaless (flexible schema) nature allows new fields to be created instantly as data is ingested, without the need to stop the database or restructure it. This flexibility serves as a powerful asset in materials research labs—where experimental conditions change constantly—enabling uninterrupted data collection.

An Environment Friendly to Ontology Construction

Materials data must move beyond simple lists of numerical values toward the construction of ontologies—knowledge systems that encompass relationships between substances and experimental contexts. Elasticsearch allows for the flexible definition of complex nested objects and associations between data points, making it highly advantageous for designing the intricate links between material properties, processes, and performance. These characteristics transform fragmented experimental data into a knowledge base, providing the foundational groundwork for AI to infer correlations between materials in the future.

Indexing-Based Dynamic Schema: A Data Collection Strategy to Maximize Research Autonomy

The greatest advantage is that researchers can attempt new experiments at any time without having to worry about specific database structures. In materials development—where experimental flexibility directly translates into research competitiveness—this serves as the decisive technical foundation for resolving data collection bottlenecks. This autonomy is possible because Elasticsearch implements a feature that automatically analyzes data types during the indexing process to generate schemas on the fly.

Indexing is more than just stacking experimental data in storage; it is the process of creating a reference system to instantly locate necessary information within massive datasets. Much like an index at the back of a book, Elasticsearch creates an Inverted Index structure optimized for search the moment data enters the system. This allows researchers to find specific experimental conditions or property values among hundreds of millions of records within milliseconds.

The flexible schema configuration implemented during this process follows these detailed steps: When new experimental data reaches Elasticsearch in JSON format, the system activates its Dynamic Mapping engine. Even without a predefined schema, the indexing engine analyzes each field value of the incoming data in real-time. For instance, it automatically determines whether a newly added ‘catalyst concentration’ field is a numeric type (e.g., 2.5) or if a ‘synthesis method’ field is a string (e.g., “Sol-gel”), and immediately assigns the most appropriate data type (such as Float or Keyword) to the index.

This automated configuration process enables a “collect first, define later” approach in the research field. While traditional database methods required pausing experiments to design or modify schemas whenever new variables were added, Elasticsearch allows the index structure to expand in real-time simply by ingesting data containing those new variables. Consequently, researchers can respond instantly to changes in experimental design or the emergence of unexpected variables without system constraints. This flexibly expanded index becomes the foundation for powerful multi-dimensional datasets used in materials correlation analysis.

Elasticsearch serves as more than just a storage repository; it acts as an intelligent data hub that evolves alongside the pace of materials research. Researchers can explore new experiments and variables at any time without database-related restrictions, while the system indexes them in real-time to transform them into immediate analytical assets. This accumulation of flexible, multi-dimensional datasets becomes a core asset of digital transformation, maximizing research efficiency and uncovering complex correlations between materials.

Multi-dimensional Data Analysis and Visualization via Kibana (Visualization)

The vast amount of materials data accumulated in Elasticsearch is finally translated into the language of researchers through Kibana, a powerful visualization platform. Moving beyond a tool that simply displays numbers, Kibana serves as a “Data Integrated Command Center” that tracks the evolution of data over time and provides multi-dimensional analysis of complex correlations between materials.

The core utilities Kibana offers for materials data analysis are as follows:

• Time-Series Trend Analysis and Long-term Research Tracking By utilizing Kibana’s robust time-filter features, researchers can instantly and conveniently analyze all experimental data accumulated over specific periods. For example, one can track how material properties or process efficiencies evolve by date or time through various tables and charts. This is exceptionally useful for capturing inflection points where research performance improves or back-tracking the impact of equipment status changes on data quality.
• Verification of Data Distribution and Reproducibility Kibana visualizes the qualitative distribution of data beyond simple average values. Researchers can use histograms or heatmaps to observe how non-uniform data distributions from the early stages of research (e.g., January) become concentrated around target values after process optimization (e.g., June). This shift in distribution serves as quantitative evidence of improvements in experimental reproducibility and process precision.
• Correlation Exploration through Multi-dimensional Filtering Kibana allows researchers to instantly filter thousands of records to isolate specific catalyst groups or temperature ranges, enabling a separate analysis of those specific segments. This facilitates cross-verification of how particular variables influence results within the overall data flow, providing critical insights when researchers review historical data to formulate new hypotheses.
• Location-based and Laboratory Environment Integrated Monitoring (Maps & Canvas) Materials research can be influenced by spatial factors, such as the location of a lab or the layout of equipment. Using Kibana Maps, data flowing in from multiple laboratories can be visualized geographically. Additionally, the Canvas feature allows for the creation of infographic-style dashboards to manage equipment utilization rates and experimental progress in real-time.
• Anomaly Detection and Machine Learning Integration Kibana can detect outliers that deviate from general patterns within vast amounts of experimental data in real-time. If an unexpected variable intervenes or an equipment error occurs during an experiment, Kibana provides immediate alerts to prevent data contamination. Furthermore, through integrated machine learning capabilities, researchers can visually verify the pre-processing results of models designed to predict future material properties.

In this way, Kibana visualizes the complex information stored within the vast “brain” of Elasticsearch into charts, dashboards, and maps, helping researchers intuitively grasp the full context of their data. This ensures that researchers are not overwhelmed by fragmented experimental results, but can instead significantly enhance the quality of their research through data-driven decision-making.

Infinite Scalability and High-Speed Analytics: Realizing the Value of Materials Big Data with the Elastic Stack

As materials research accelerates, the volume of data generated increases exponentially. The ability to process and analyze this data without latency is a critical factor in the success of research. Designed with a distributed processing architecture, the Elastic Stack allows for consistent performance even as data volumes grow by simply adding nodes, making it overwhelmingly advantageous for long-term materials big data construction and large-scale processing.

Notably, its full-text search and powerful filtering capabilities enable researchers to pinpoint specific data—such as conditions where “a specific additive exceeds 5%” or environments with a “drying temperature of 80°C”—with millisecond precision among tens of thousands of records. Furthermore, data managed this way becomes “AI-Ready” by default and can be immediately called into Python environments (such as Pandas or PyTorch) via REST APIs, offering excellent compatibility with AI modeling workflows.

The Elastic Stack connects sophisticated collection and processing through Beats and Logstash, flexible storage and high-speed search via Elasticsearch, and multi-dimensional visual analysis through Kibana into a single, organic flow. It functions as the most powerful and reliable data engine—not merely by storing vast amounts of data, but by uncovering patterns within complex, interconnected datasets in real-time, ultimately empowering researchers to derive the optimal material recipes.