FOCUS(ID:2041/foc013)Hierarchical database languageHierarchical database language. Information Builders Inc. Initially for 390, then for UNIX, then PC Structures: References: in Datamation 26(06) June 1980 view details in Datamation 26(06) June 1980 view details in Datamation 26(06) June 1980 view details FOCUS is a self-contained data management and reporting system marketed by Information Builders Incorporated. It also has a number of interfaces allowing it to retrieve data from databases such as IMS (DL/I), TOTAL, ADABAS and IDMS, the last of these being a CODASYL system. When using FOCUS as a query language for IDMS the user has to think in terms of FOCUS data structures - these are segment hierarchies (like DL/I) which can be 'rehung' to provide inverted views. Although the actual data is in IDMS, the user is unaware of the IDMS structure. Only when a totally new view is required does the user (or database administrator) have to set up a new FOCUS description and the correspondence rules stating how the IDMS structure is to be traversed. Pansophic's EASYTRIEVE also has an IDMS interface with a somewhat similar approach. in ACM SIGMOD Record 13(03) April 1983 view details Introduction An issue facing designers and users of computer applications is the treatment of missing and/or incomplete data. Such data include fields for which no value was provided by input transactions, fields for which values have been rejected by validation criteria, uninitialized variables, null parent records generated by a Database Management System (DBMS) when data for a child record only are given, lost data created by computer or communication systems malfunction, and so on. When data are missing, lost, or incomplete, procedures may work incorrectly; computations may lose reliability, and their results may have to be interpreted as estimates (if they are even valid at all). Special procedures will often be necessary to identify, correct, and deal with these phenomena. Several approaches are used for the management of missing data. One extreme tactic is to not identify such data and to abort processing once they are encountered (this is the method employed by IBM's FORTRAN, for example, [10]). Other systems ignore fields or records for which data are missing, in effect reducing the amount of data available to the user. This is an option available in some statistical packages (e.g., SPSS [19] and BMDP [24]) and the approach of some input commands in some DEC FORTRAN compilers [4]. The opposite extreme tactic is to not recognize missing data as such, but to replace them with default, imputed, or user'selected values. This is the policy of many financial reporting/modeling packages, which replace a missing value with a zero (e.g., TABOL IIl [6]), of some student compilers (e.g., PL/C [14] and WATFIV [17]), and of systems that allow classes of default values (e.g., VISICALC [26] and SAS [2]). The greatest liability of this tactic is that it tends to give the user the erroneous impression that all required data are available. Several other forms of "data impurity" are directly related to the missing data issue: ? Partially available data are manifest in the lack of "exact" value for an item (e.g., a person is known to be an adult or to belong to a certain age group, but his or her age is not known exactly) or the lack of any information for certain fields in a record (e.g., the first name of a person is unknown, even though his or her family name is available). ? Ambiguous data occur when different values for the same item are stored by the system (e.g., different names for the same person are stored in different records or files). Data that are known to be erroneous data may be provided to the application (e.g., a "Mr." may accidentally be put before a woman's name and then go undetected by the data entry validation procedures). The lack of certain data can sometimes be as informative and pertinent for inference and decision making as the existence of these data would be--hence the rather curious title of this paper. A programmer would like to know all the variables that are being used without being initialized; an executive would like to know all departments for which accounting or production data are not available; a database administrator would like to control and minimize the automatic generation of null parent records. If a person's name is given in a credit information file without any information on credit default, it can be assumed that that person is a good credit risk; if a customer record does not contain any open billing, it can be assumed that he or she does not owe money, and so forth. In all of these situations, the lack of particular data actually contributes to a user's knowledge and understanding of a particular situation. What are the implications of missing data and of various alternatives for treating such data? The next section considers a few of the many environments in which it is necessary to deal with missing data. Next there is a discussion of some standard procedures for treating missing data; further sections extend the spectrum of options that should be provided for handling missing data and suggest ways of implementing these options in high-level programming languages. The final section outlines solutions for missing data phenomena in the Current processing environment and discusses the possible effects of enhanced missing data processing on future data processing activities. Extract: Instances of Missing Data: Controlling Program Variables Instances of Missing Data: Controlling Program Variables One of the most common programming errors is reference to an undefined variable, which can either be an uninitialized variable that is encountered for the first time during a program execution or a variable for which input data are requested (from keyboard or file} but not provided (or for which it is incorrectly supplied). In the former case, no reasonable value can be substituted for the missing datum since there is no point of reference. In the latter case, though, prior processing may suggest a suitable or assumed value. In some languages a missing input item is automatically assigned a zero value, whereas in others the last assigned value of this variable is used (e.g., the READ and ACCEPT of DEC'S PDP-11 FORTRAN [4]). In large application systems, different programs may be written by different people, and one program may use the results of another. This means that data transferred between subroutines and programs (via COMMON, parameters, or files) may be contaminated. One program may set values for some missing or erroneous items and provide default values for others. Without consistent methods for handling or representing such data, subsequent programs may not be able to correctly interpret such assignments. Much of the effort in system design and development is directed at controlling and minimizing this kind of occurrence. In financial reporting and modeling systems (e.g., IFPS [5] and TABOL III [6]), the provided data are often incomplete. There are, for instance, certain expenditures that occur only at infrequent or irregular periods, base data from which projections are developed that may not be available, reports that may contain columns without data, etc. Such systems usually replace these missing data with zeros or with other default values that would be appropriate in many situations. Someone reading a report from such a system would, however, be unable to determine whether a printed zero is indeed an input or computed zero, or a substitute for missing data. Extract: Instances of Missing Data: Statistical and Mathematical Computations Instances of Missing Data: Statistical and Mathematical Computations By its very nature, the data in statistical computations tend to have missing values and outliers. In this category of missing data are: ? Missing values, which usually result from problems in sampling and data collection. Either the data cannot be collected, or they are deemed questionable and are therefore rejected. ? Outliers, which are values out of line with other data. Outliers result from the random nature of the collected data or from special events. They may severely bias the satistical result: One very low value can substantially reduce the mean of a large sample or increase its variance. To assure the acceptability and accuracy of the statistical results, some means of data correction and/or data conditioning are required. A problem that often occurs in statistical and demographic computation is the aggregation of data into classes or ranges, resulting in a lack of exact single values. This kind of grouping may be caused by the sampling methods used, by the inexactness of the sampled data, or by a need to minimize the quantity of the available data. Often, this kind of phenomenon can create a great deal of extra work--the computation of life tables from census data, for instance, requires special techniques to overcome grouping into age ranges. Missing data are also a problem in sensitivity analysis in mathematical programming and other optimization and numerical algorithms. If information about the possible range of various parameters and coefficients, necessary for the analysis, is missing, the program may assume a range that makes no sense in reality. It might even make such an assumption without informing the user. Extract: Instances of Missing Data: Databases Instances of Missing Data: Databases An orphan record results when a parent record in a hierarchical order is missing. This is a common phenomenon in hierarchical database systems and can occur in other kinds as well. Consider a database structure in which employee records are grouped together under a parent department record. Suppose that a department has been eliminated but that its employees have not yet been assigned to other departments. In order to maintain the integrity of the database, the department record must be eliminated; but to assure that the orphan employee records are not lost, a null parent record must be generated for them until they are redistributed among actual departments. Free-lance employees, not assigned to any particular department, would also need a null parent department. Orphan records can occur when such popular DBMSs as IMS [11], SYSTEM 2000 [18], and FOCUS [12] are used. Manual and optional set membership in the CODASYL data model [3] are constructs that also permit orphan records without contrived parent records or links. A similar problem occurs when a database structure is modified so that new fields are added to records or new record types are introduced. What values should initially be stored in these new fields and records? What data should be loaded into existing records that are now to include data from, or links to, new records? Should the validation rules applied to updates of a field with an existing null value be different from those applied to a nonnull value? Similarly, how can a database system respond to queries when the data it has to work with are incomplete or ambiguous? What should be the response to "list all red objects" when for some objects the color designation is missing and others are identified as being "not of a certain color"? Should the response be a list of all objects that are surely red, a list of all objects that might be red, or some hybrid of the two? Should the list of all objects that lack a color value also be reported? Still another database-related problem occurs in relational databases, where each tuple (or record) is assumed to have data for all of its key attributes and is uniquely determined by its key attribute values. How should tuples with partial key data be stored and processed? Should duplicate tuples be eliminated when key attribute values are missing? This is a particularly acute problem for relational databases, because of the axiomatic nature of their processing and their strict structural requirements [25]. Advances in data processing, and in distributed processing particularly, serve to emphasize the problems inherent in missing and ambiguous data. In many distributed systems, local databases are refreshed from a central database, or refresh such a database, at certain times. Consequently, the local and central databases might differ in their content between refreshing epochs, with all the resulting problems of missing and ambiguous data. This is particularly noticeable when refreshing epochs occur between update runs, where the second supplements the data entered in the first run and corrects data that were rejected as errors or flagged as possible errors. in [ACM] CACM 27(08) (August 1984) view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details in Proceedings of the 1984 annual conference of the ACM on The fifth generation challenge 1984 view details Resources
|