| Name: | Description: | Size: | Format: | |
|---|---|---|---|---|
| 11.39 MB | Adobe PDF |
Authors
Advisor(s)
Abstract(s)
Na vida das organizações, é muito frequente o acesso a grandes quantidades de dados, imprescindíveis aos processos de tomada de decisão. Por vezes, os dados apresentam problemas de qualidade que afetam a qualidade das decisões. Nos armazéns de dados, durante a manipulação dos dados, para se criar formatos que facilitem os processos de tomada de decisão, podem ser identificados mais problemas de qualidades nos dados (PQD). Existem diversas abordagens para detetar e corrigir PQD. Nestas abordagens pretende-se classificar os diferentes tipos de PQD que possam ocorrer nos dados e indicar caminhos possíveis de deteção e correção dos PQD. Existem algumas ferramentas que se baseiam em abordagens de deteção e correção de PQD existentes. Estas ferramentas detetam e corrigem PQD, no entanto, normalmente servem para detetar e corrigir PQD específicos e têm custos de aquisição consideráveis. O processo de deteção e correção pode ser complexo e é moroso. Normalmente estes processos de deteção e correção estão encadeados, e podem assumir nomes diferentes como “limpeza de dados” e “eliminação de dados sujos”. Foram definidos objetivos para desenvolver uma solução de deteção de PQD, freeware, com bom desempenho e de fácil utilização. Fez-se levantamento do estado da arte, apresentando conceitos importantes para a compreensão do tema da dissertação. Estudaram-se diferentes tecnologias uteis no desenvolvimento da solução. Foi desenvolvida uma solução de deteção de PQD, robusta, de fácil utilização que permite desenhar um workflow com uma sequência de operações de PQD escolhida pelo utilizador. O utilizador pode guardar o workflow para execução posterior com o mesmo ou com outras fontes de dados. A solução contém uma estrutura facilmente expansível para detetar novos tipos de PQD e com novos motores e algoritmos de deteção. A avaliação da solução revela que a solução disponibiliza uma interface gráfica facilitadora do processo de desenho de workflow e configuração das operações de PQD. A solução apresenta um bom desempenho, utilizando programação de pipelines Java com streams paralelas.
In organizations life, access to large amounts of data, which are essential to decision-making processes, is very common. Sometimes the data has quality problems that affect the quality of decisions. In data warehouses, during data manipulation, to create formats that facilitate decision-making processes, more problems of quality in the data (PQD) may arise. There are several approaches to detect and correct PQD. In these approaches it is intended to classify the different types of PQD that may occur in the data, and to indicate possible ways of detection and correction of PQD. There are some tools, which are based on existing PQD detection and correction approaches. These tools, detect and correct PQD, however they usually serve to detect and correct specific PQD, and have considerable acquisition costs. The PQD detection and correction process can be complex and time consuming. Usually, these detection and correction processes are linked, and may take different names such as “data cleaning” and “dirty data elimination”. The goals to develop a PQD detection solution were defined. Development of a solution freeware, with good performance and easy to use. A state-of-the-art survey was carried out, presenting important concepts for the understanding the dissertation theme. Different useful technologies were studied in the development of the solution. A robust, easy-to-use PQD detection solution was developed that allows designing a workflow with a sequence of PQD operations chosen by the user. The user can save the workflow for later execution with the same or with other data sources. The solution contains an easily expandable structure to detect new types of PQD, with new detection engines and algorithms. The evaluation of the solution reveals that the solution provides a graphical interface that facilitates designing the workflow and PQD operations configuration. The solution presents a good performance, using Java pipelines programming with parallel streams.
In organizations life, access to large amounts of data, which are essential to decision-making processes, is very common. Sometimes the data has quality problems that affect the quality of decisions. In data warehouses, during data manipulation, to create formats that facilitate decision-making processes, more problems of quality in the data (PQD) may arise. There are several approaches to detect and correct PQD. In these approaches it is intended to classify the different types of PQD that may occur in the data, and to indicate possible ways of detection and correction of PQD. There are some tools, which are based on existing PQD detection and correction approaches. These tools, detect and correct PQD, however they usually serve to detect and correct specific PQD, and have considerable acquisition costs. The PQD detection and correction process can be complex and time consuming. Usually, these detection and correction processes are linked, and may take different names such as “data cleaning” and “dirty data elimination”. The goals to develop a PQD detection solution were defined. Development of a solution freeware, with good performance and easy to use. A state-of-the-art survey was carried out, presenting important concepts for the understanding the dissertation theme. Different useful technologies were studied in the development of the solution. A robust, easy-to-use PQD detection solution was developed that allows designing a workflow with a sequence of PQD operations chosen by the user. The user can save the workflow for later execution with the same or with other data sources. The solution contains an easily expandable structure to detect new types of PQD, with new detection engines and algorithms. The evaluation of the solution reveals that the solution provides a graphical interface that facilitates designing the workflow and PQD operations configuration. The solution presents a good performance, using Java pipelines programming with parallel streams.
Description
Keywords
Problemas de qualidade nos dados Limpeza de dados Deteção de PQD ETL Workflow Data quality problems Data cleaning PQD detection