Data import system for data analysis system

Details Data import system for data analysis system Data import system for data analysis system

2000-09-29

2004-04-06

Metjahic, Safet (Department: 2171)

Data processing: database and file management or data structures

Database design

Data structure types

Reexamination Certificate

active

06718336

ABSTRACT:

I. FIELD OF THE INVENTION
This invention relates generally to a system for importing data into a data analysis system and, more particularly, to a system for importing data into a data analysis system for graphically displaying data relationships for the imported data.
II. BACKGROUND OF THE INVENTION
To make informed decisions regarding large volumes of collected records, automated processes are used to analyze and identify relationships that exist among the records. One of the most useful processes for gaining a quick and useful understanding of these relationships is the use of data analysis visualization tools for visualization-based exploratory data analysis. Data analysis visualization tools are software packages that create graphical representations of the relationships that exist among data sets or records.
Visualization-based exploratory data analysis involves three primary steps: importing the data from one or more sources, creating the visualizations from that data, and exploring and mining the data. A common problem for data analysis systems is getting data into a form that can be handled appropriately by the analysis software. The data streams and formats that are encountered are as varied as the creators of that information. Thus, it has been difficult to provide an input process or system to accommodate the variety of data formats.
These difficulties are amplified if analysis is desired for information of many types. For example, free text, numeric, categorical information, and genomic sequence data are different types of data that may be encountered in records. In addition to desiring systems that can handle data of various types, users desire flexibility in transforming data from one format to another. Depending on the nature of the data, the user may desire a variety of mathematical or other operations to be performed on the information, such as normalization schemes. This is particularly important with experimentally derived data.
There are several standard data formats, such as tab-delimited ASCII files, that can be read by spreadsheet and other programs. Automated and semi-automated methods exist for importing data from these standard formats into software analysis packages (e.g., the Text Import Wizard in Microsoft Excel). However, even within these standard formats, the organization of information may not be consistent with the desired, ensuing data analysis.
R. Ford, R. Thompson and D. Thompson, Supporting heterogeneous data import for data visualization, Proceedings of the 1995 ACM Symposium on Applied Computing, pp.81-85, 1995, have described one approach for combining datasets from various sources and formats. They use format converters that can be created for preexisting data formats and apply these to convert digital geographic information into a format that can be used by IBM Data Explorer software. This approach has general utility, but the output types are not sufficiently structured for enabling integrated data visualization and mining across disparate data types and structures.
H. Davulcu, G. Yang, M. Kifer, and I. Ramkrishnan, Computational Aspects of Resilient Data Extraction from Semistructured Sources, Proceedings of the 19
th
ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 136-144, 2000, describe a method for extracting information from semistructured data sources such as HTML pages. The variability in construction of those data sources and the need to extract common attributes from those sources presents difficulties. The approaches rely on extraction expressions as a means for identifying target data within a larger context. U.S. Pat. No. 5,913,214 to Madnick, et al., entitled Data extraction from word wide web pages, also describes a system for extracting disparate, heterogeneous data from multiple semi-structured data sources. In this system, a data translator is added to provide the data context needed for queries and retrieval. In a related patent, U.S. Pat. No. 5,953,716, entitled Querying heterogeneous data sources distributed over a network using context interchange, methods are provided to generally handle problems in data set creation, but these methods do not handle unstructured data sources, do not provide means for integrating multiple data types, and do not permit user-defined manipulations of the data.
Another approach for combining information from different sources is discussed in U.S. Pat. No. 6,023,694 to Kouchi, et al., entitled Data retrieval method and apparatus with multiple source capability. In the '694 patent, information is retrieved from two or more data sources by using drivers specific for each data format and converted to a database with a standardized data structure using key categories for organizing the data. U.S. Pat. No. 5,721,912 to Stepczyk, et al., entitled Graphical user interface for creating database integration specifications, also uses a data translation step in the framework of a graphically represented workflow.
C. Squire, Data Extraction and Transformation for the Data Warehouse, Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, pp. 446-447, 1995, discusses the integration and transformation of data for a data warehouse. However, Squire does not provide a general process for importing data into visualization and mining systems and does not describe a system that supports integration of different data types.
To enable analytical and visualization methods to be applied to multiple data types the final data format is standardized. D. King, Using Dimensions to Represent Data Base Attributes, Conference Proceedings on APL and the Future, pp. 151-155, 1985; and G. Bringham and E. Shaw, An N-Dimensional Data Structure in Support of Electronic Data Interchange (EDI) Translation, Proceedings of the International Conference on APL '91, pp. 71-79, 1991, describe a standard database data structure in which each data object can be thought of as a collection of attributes. As such, each object may be represented by an N-dimensional collection of elements. The discussed N-dimensional collection is a valuable construct for data visualization, but does not have the flexibility to incorporate data from multiple sources, the ability to manipulate the data, and the ability to represent some data types (e.g., genomic sequence or text).
U.S. Pat. No. 5,361,326 to Aparicio et al., entitled Enhanced interface for a neural network engine, describes one approach for bringing together some aspects of the data import requirements. A generalized data translator is provided along with an ability to manipulate or transform the raw data as part of the front end to a neural network analysis.
After importing data for analysis, a number of manipulations may be made. These include operations or transformations with the data or merging overlapping information. In order to process these steps correctly, the data type for those operations should be defined. The data type can be defined by the user at the time of the process or an automatic method can be used for identifying data type. One method for identifying data type is described in U.S. Pat. No. 5,991,714 to Shaner, entitled Method of identifying data type and locating in a file. The '714 patent uses an n-gram-based approach to characterize an electronic file of unknown data type. The n-grams are weighted and compared to defined thresholds for known data types. Automatic data type determination is also described in U.S. Pat. No. 6,014,661 to Ahlberg, et al., entitled System and method for automatic analysis of data bases and for user-controlled dynamic querying. The data is subjected to a series of tests that assess consistency with defined data types. For example, if the data contains only alphabetic ASCII characters, the data is assumed to be text or if the values detected are limited to 0 and 1, the data is assumed to be binary.
Data type recognition has also been applied to data streams as discussed in U.S. Pat. No. 5,784,544 to Stevens, entitled Method and system for determining the data type of

Affiliated with

Also associated with

Rating

Data import system for data analysis system does not yet have a rating. At this time, there are no reviews or comments for this patent.

If you have personal experience with Data import system for data analysis system, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Data import system for data analysis system will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFUS-PAI-O-3220200