Process of Data mining
Data mining is an iterative process that typically involves the following phases:
Problem definition
A data mining project starts with the understanding of the business problem. Data mining experts, business experts, and domain experts work closely together to define the project objectives and the requirements from a business perspective. The project objective is then translated into a data mining problem definition.
In the problem definition phase, data mining tools are not yet required.
Data exploration
Domain experts understand the meaning of the metadata. They collect, describe, and explore the data. They also identify quality problems of the data. A frequent exchange with the data mining experts and the business experts from the problem definition phase is vital.
In the data exploration phase, traditional data analysis tools, for example, statistics, are used to explore the data.
Data preparation
Domain experts build the data model for the modeling process. They collect, cleanse, and format the data because some of the mining functions accept data only in a certain format. They also create new derived attributes, for example, an average value.
In the data preparation phase, data is tweaked multiple times in no prescribed order. Preparing the data for the modeling tool by selecting tables, records, and attributes, are typical tasks in this phase. The meaning of the data is not changed.
Modeling
Data mining experts select and apply various mining functions because you can use different mining functions for the same type of data mining problem. Some of the mining functions require specific data types. The data mining experts must assess each model.
In the modeling phase, a frequent exchange with the domain experts from the data preparation phase is required.
The modeling phase and the evaluation phase are coupled. They can be repeated several times to change parameters until optimal values are achieved. When the final modeling phase is completed, a model of high quality has been built.
Evaluation
Data mining experts evaluate the model. If the model does not satisfy their expectations, they go back to the modeling phase and rebuild the model by changing its parameters until optimal values are achieved. When they are finally satisfied with the model, they can extract business explanations and evaluate the following questions:
Does the model achieve the business objective?
Have all business issues been considered?
At the end of the evaluation phase, the data mining experts decide how to use the data mining results.
Deployment
Data mining experts use the mining results by exporting the results into database tables or into other applications, for example, spreadsheets.
The Intelligent Miner™ products assist you to follow this process. You can apply the functions of the Intelligent Miner products independently, iteratively, or in combination.