Stages of Data Processing Cycle

Data processing stages consist of those activities necessary to transform data into information.

The stages of the Data Processing cycle are:

• Data Collection
• Data Preparation
• Data Input
• Data Analysis
• Data Interpretation
• Data Storage

Data Collection

The Data Collection process starts after collecting raw data from all possible sources. After raw data collection, they are converted into a computer friendly format, example tables, texts, images etc. to form a repository of data stored in both the natural and transformed formats. Major types of data collection includes statistical populations, research experiments, sample surveys, and byproduct operations. The collection and handling of data is not always an easy task. Most often the real world data possess noise, redundancy, and/or contradiction in the data.

Data Preparation

The data preparation stage involves pre-processing. Raw data are cleaned, organized, and checked for errors. The purpose of this stage is to deal with the missing values and eliminate redundant, incomplete, duplicate, and incorrect records. Significant domain knowledge may be required to correctly prepare the data, and possession of this knowledge is important because data that are not carefully prepared and screened can result in misleading information.

Data Input

As the data have been cleaned and entered into their destination location and translated into the desired format which can be easily understood. Understanding data means having a grasp of their key characteristics, including distribution, trends, and attribute relationships. This time-consuming process must be performed with speed and accuracy, and many organizations prefer to outsource this stage.

Data Analysis

The data analysis stage may be performed through multiple threads of simultaneously-executed instructions using machine learning and artificial intelligence algorithms. The time needed for this stage depends on the specifications of the processing device used and the complexity and amount of input data. This stage is the “heart” of data processing and may include converting the data to a more suitable format. This step has multiple sub-steps as follows:

• Feature Extraction : Data are represented by a number of fixed features which can be categorical, binary or continuous.
• Correlation Analysis : The focus of this step is to determine which pairs of data features have the highest degree of correlation. When two features are found to have a high correlation coefficient within a defined threshold, one of them can be removed from the feature set.
• Feature Selection : During this step, informative and relevant features are selected by applying correlation analysis to separate redundant features, keeping the features which show high correlation with the target variable. Relevant features are those that have a low degree of intercorrelation with other features and a high level of changeability across data records.
• Machine Learning : In this step, a learning mathematical algorithm is developed to extract knowledge from and uncover the properties of the data and predict future outcomes should new data be inserted. Descriptive analysis are used to understand underlying data patterns, predictive analysis are used to estimate new or future data based on past data performance, and prescriptive analytics are used to optimize the dependent action. Which learning technique to use is also determined: Unsupervised or Supervised.
• Extracting valuable insights : After the model is evaluated for accuracy and performance, the most important and relevant information contained in the input data is retrieved and presented. At this point the model is ready to be used for predicting future events and the probable gains/losses that a business can expect under different scenarios. However, in practice it is often difficult to judge the impact the model has on a business from the model’s performance; hence, both model evaluation metrics and key performance indicators (KPIs) are used to judge the model’s performance and degree of impact.

Data Interpretation

After the data analysis, it is time to interpret the data. To do so, the outcomes of the machine learning predictions need to be translated into actions. The outcomes must be interpreted to obtain beneficial information that can guide a company’s future decisions. It is a critical step because the outputs of the developed model (or the model itself) need to be presented to business managers in a user-friendly form so that the managers can take appropriate actions and make better decisions. For example, tables, audio, videos, and images. Although the insights obtained in the data analysis stage are important, the actions taken — either automatically or as decided by humans — are the more valuable outputs.

Data Storage

The final stage of data processing is the storing the data, instructions, developed numerical models, and information for future use. Data should be stored in such a manner that they can be accessed quickly and are available for retrieval when needed.