What are the roles of information policy and data administration in administration management?

Data Administration

The data administration area consists of the personnel who are involved in the capturing of the business requirements from the business problem area. Also, they are responsible for integrating with and receiving model constructs and high-level definitions from the corporate architects and capturing these within reusable constructs such as case tools and data dictionary/repositories. They are also responsible for maintaining these model structures over time and ensuring that they reflect the business.

Data administration’s focus is on managing data from a conceptual, DBMS-independent perspective. It coordinates the strategies for information and metadata management by controlling the requirements gathering and modeling functions. Data modeling supports individual application development with tools, methodology[ies], naming standards, and internal modeling consulting. It also provides the upward integration and bridging of disparate application and software package models into the overall data architecture. This overall data architecture is the enterprise data model and is critical in the organization’s ability to assess business risk and the impact of business changes.

7.2.5 Set Auxiliary Tables

In the January 2005 issue of The Data Administration Newsletter [www.TDAN.com], I published an article on a look-up table solution to a more difficult problem. If you watch the Food channel on cable or if you just like Memphis-style BBQ, you know the name “Corky’s.” The chain started in 1984 in Memphis by Don Pelts and has grown by franchise at a steady rate ever since. They will never be a McDonald’s because all the meats are slow cooked for up to 22 hours over hickory wood and charcoal, and then every pork shoulder is hand-pulled. No automation, no mass production.

They sell a small menu of items by mail order via a toll-free number or from their website [www.corkysbbq.com] and ship the merchandise in special boxes sometimes using dry ice. Most of the year, their staff can handle the orders. But at Christmas time, they have the problem of success.

Their packing operation consists of two lines. At the start of the line, someone pulls a box of the right size, and puts the pick list in it. As it goes down the line, packers put in the items, and when it gets to the end of the line, it is ready for shipment. This is a standard business operation in lots of industries. Their people know what boxes to use for the standard gift packs and can pretty accurately judge any odd sized orders.

At Christmas time, however, mail-order business is so good that they have to get outside temp help. The temporary help does not have the experience to judge the box sizes by looking at a pick list. If a box that is too small starts down the line, it will jam up things at some point. The supervisor has to get it off the line, and re-pack the order by hand. If a box that is too large goes down the line, it is a waste of money and creates extra shipping costs.

Mark Tutt [On The Mark Solutions, LLC] has been consulting with Corky’s for years and set up a new order system for them on Sybase. One of the goals of the new system is print the pick list and shipping labels with all of the calculations done, including what box size the order requires.

Following the rule that you do not re-invent the wheel, Mr. Tutt went to the newsgroups to find out if anyone had a solution already. The suggestions tended to be along the lines of getting the weights and shapes of the items and using a Tetris program to figure out the packing.

Programmers seem to love to face every new problem as if nobody has ever done it before and nobody will ever do it again. The “Code first, research later!” mentality is hard to overcome.

The answer was not in complicated 3-D math, but in the past 4 or 5 years of orders in the database. Human beings with years of experience had been packing orders and leaving a record of their work to be mined. Obviously the standard gift packs are easy to spot. But most of the orders tend to be something that had occurred before, too. Here are the answers, if you will bother to dig them out.

First, Mr. Tutt found all of the unique configurations in the orders, how often they occurred and the boxes used to pack them. If the same configuration had two or more boxes, then you should go with the smaller size. As it turned out, there were ~ 5000 unique configurations in the custom orders which covered about 99.5% of the cases.

Next, this table of configurations was put into a stored procedure that did a slightly modified exact relational division to obtain the box size required. A fancy look-up table with the divisor set as its parameter!

Fixing the Problem

A large organization with multiple databases should probably be involved in data administration, a process distinct from database administration. Data administration keeps track of where data are used throughout an organization and how the data are represented. It provides oversight for data at an organizational level, rather than at the database level. When the time comes to use the data in a database or application program, the developers can consult the metadata [data about data] that have been identified through the data administration process and then determine how the data should be represented to ensure consistency.1

It is important to keep in mind, however, that even the best data administration can’t totally protect against inconsistent names and addresses. Although organizational metadata can specify that the abbreviation for street is always “St.” and that the title for a married woman is always stored as “Mrs.”, there is no way to ensure that names and addresses are always spelled consistently. Human error will always be a factor. When that occurs, the best strategy may be just to smile sweetly to the complaining customer and fix the problem.

Table-based Search/Edit

For the simplest type of screen, typically used to maintain administration data [e.g., lists of countries or product types, etc.], the UI pattern we use in an ADF application is very similar to the pattern used in the Forms application. The main difference is that a separate component is used to perform the search.

If you are familiar with the ADF 11g framework, you know that the simplest way to provide CRUD capabilities is to create an entity based on the table being edited. A View object would then be created for that entity and exposed to the user in the page. To provide the search capability to the user, the developer would create a set of view criteria. These view criteria are then used to create an ADF Query Panel that the user of the application will use to search the data. To take advantage of the benefits of the ADF Query Panel, you need to include the Query Panel on the page.

In order to alleviate any unnecessary button clicks, the ADF Panel Query is provided fully disclosed. The user can then search and see the results on the same screen. Edits to the returned data can be performed in the table itself, and records can be deleted from the returned results. When the user wants to add a record, the user simply clicks the Add button which will insert an empty row into the table, and then he or she can enter data directly into that row.

The task flow for this type of search and edit functionality in Figure 12.5 explicitly shows the messages being set and cleared to indicate to the user that an action has been successfully performed.

Users of the original application normally react well to a change such as this. Though the UI has changed from the original form, it provides them some additional functionality that they did not have before through the ADF Query Panel. If you were to use the Oracle Metadata repository, users could even save their favorite queries using this panel.

Basic System Goals

The CEO has defined the following goals for the reengineering project:

Develop a corporation-wide data administration plan that includes a requirements document detailing organizational functions that require technology support and the functions that the reengineered system will provide.

Provide an application roadmap that documents all application programs that will be needed to support corporate operations.

Investigate alternatives for developing and hosting a Web site that will allow online orders. Conduct a cost-benefit analysis of those alternatives before beginning development.

Document all databases to be developed for the corporation. This documentation will include ER diagrams and data dictionaries.

Create a timeline for the development of applications and their supporting databases.

Specify hardware changes and/or acquisitions that will be necessary to support the reengineered information systems.

Plan and execute a security strategy for an expanded corporate network that will include both internal and external users.

Implement the planned systems.

Note: It is important to keep in mind that the implementation of a Web presence for the company is relatively independent of the database design. The Web application will require the same data as all the other types of data entry. Concerns about hosting and security are rarely the job of the database designer.

VI.A. Changes

Database technology evolved from file processing systems. Before the database was invented, data was stored in different data files and data administration was essential in every data processing organization. Data administration is a high level function that is responsible for the overall management of data resources in an organization. With databases developed on a mainframe legacy system, the role of data administration still remains popular. Organizations that employ separate data administration functions often have mainframe-based databases that have been developed using established systems development methodology, including development of logical and physical models.

As client/server database technology develops, the blend of the two roles between data administration and database administration is becoming a reality. These organizations emphasize the capacity to build a database quickly, tuning it for maximum performance and being able to restore it to production quickly when problems develop. These databases are more likely to be departmental, client/server databases that are developed quickly using newer developmental approaches such as prototyping, which allow changes to be made quickly. There are no universally accepted data administration and database administration structures. Organizations vary widely in their approaches to data administration. As business practices change, roles are also changing within organizations. In organizations where there is no separate data administration function, the DBA also assumes the responsibilities of the data administrator.

Model-Driven Master Data Components

Building on the notion of the preprogrammed add–modify instance widget, the leading data warehouse administrative package based on hyper generalization also provides an adaptable architectural component that development teams can easily incorporate in order to quickly establish robust master data management [MDM] for their company’s EDW applications.

Figure 15.22 shows how these packages enable EDW teams to generate key MDM elements from their business models and then draw upon an adaptable web-based master data administration tool to arrive at canonical records for key business entities such as customer, product, and location. Using the hyper generalized warehouse administration package, the EDW developers employ machine-driven development to create four components based on the business models they have created:

The database tables of the landing area

The ETL for processing the master data elements

The master data repository for validated records, and another for rejected records

The data transform module for processing master data elements automatically assembles candidate master data records from the landing area according to the logic provided by the developer’s business model. This process then decides whether to accept or reject each candidate record. Candidate records are evaluated using multiple tests, such as regular-expression parsing for acceptable formats, valid domains screening for legal values, and parent record lookup for implied foreign keys. Records passing these tests are sent to a staging area from which the data warehouse can load them into the EDW. The master data processing component also employs “fuzzy logic” to discern whether or not the candidate records are already in the master data repository. Fuzzy logic relies on cascading matching events to quantify how well candidate data resemble existing master data records. Candidates with many matching components will pass a threshold value that the master data managers have set for each entity, causing the MDM process to consider them already included in the master data, dropping them from any further processing.

The MDM process places records failing to meet the required threshold values for each master data element type into a work-in-progress area. The company’s data stewards and data administrators then collaborate on manually processing the rejected records using a web-based user interface provided with the master data management utility. The web-based interface automatically adapts to the structure of each master data element and allows developers to customize the processing workflow for each entity. Figure 15.23 illustrates some of the details of a typical workflow. Data stewards selected from the departmental business staff review the rejected records, searching for defects in formatting or semantics that they can correct in order to make a candidate record acceptable for the master data collection. Later, a data administrator reviews the corrected records and releases those that he or she accepts to a pending-records pool.

MDM tools provide considerable flexibility, supporting other approval workflows besides the example discussed here. Data steward approval can be required even for new records that pass the data quality tests. These records can be distributed to data stewards as they arrive from source systems or queued for bulk authorization. No matter the path records take among the data stewards, when the pending pool of approved records reaches a preset limit, such as a time-based event or a particular number of pending records, the MDM applications release the accepted records to the data warehouse. The model-driven ETL will then treat the released master data records as simply a trusted source for dimensional entities and load them into the data warehouse.

The end users can search and browse the approved master data using another adaptable web-based interface included in the hyper generalized EDW automation system, making the MDM repository an important component to the company’s BI data dictionary. Should end users spot records or values that they question, know must change, or believe are missing altogether, they can submit a change request via the MDM interface. The data stewards will process these requests, again using their web-based management tool, correcting the values or creating records as appropriate, all of which then flow to the master data administrator for approval and release to the warehouse.

Figure 15.24 provides a schematic notion of the management interface that the data stewards and administrators use. The middle of the top panel provides a summary of the candidate records waiting in the working area for the data stewards to correct. The right side shows the number of records now in pending status after data administrator review, as well as counts of records published for the data warehouse to incorporate in its subject areas.

The stewards and administrators can click into any one of the values displayed to enter a searchable list of the records for a given entity in any state within the master data repository. These users can open up any one of the values shown in the resulting list to view and edit any record in particular. The bottom panel of Figure 15.24 shows a single rejected record for a service customer. The two errors displayed reveal that [1] this record has defective value for county [it should have been expressed as “Los Angeles,” not “LA”] and [2] it also has a postal code not found among those known for the customer’s city. At this point, the data steward can click on both the county and the postal code to receive a searchable list of acceptable values for these fields. Once correct values have been selected, the error flags will be cleared and the record will be sent by the application’s workflow to the data administrator for approval and release to the warehouse.

With the addition of a machine-driven master data management processing and a few adaptable web interfaces, the hyper generalized data warehouse automation tool eliminates a large number of value-added loops that the EDW team would have had to construct in order to derive clean dimensional data for the company’s key business entities. By eliminating the need for so much programming, the MDM features represent a crucial element that EDW team leaders need to add to their reference architecture. They should consider the machine-driven master data facility as a preprocessing layer just before the data warehouse and add it to their reference architectural diagram. As shown in Figure 15.25, the MDM facility takes data from the landing area to a sublayer of published master data elements. Objects in the published sublayer will then be incorporated as trusted dimension tables into the integration layer and later the star schemas when warehouse data is projected into the presentation layer.

In this overall system, the hyper generalized data model allows the development team to deliver both master data management and regular data transforms using business-model-driven application generation and modification. Because it offers machine-assisted tools for both master data and subject area development, it is no surprise that teams opting for hyper generalization can achieve 3–10 times the delivery speed as teams using traditional EDW methods and technologies. The dollar value of the human toil that these tools eliminate alone will justify the purchase and implementation costs of the hyper generalized data warehouse automation package. Far surpassing those savings, however, will be the value of the additional business opportunities from which companies will be able to profit because their EDW teams can now deliver and adapt a data warehouse—including its master data—with an order of magnitude greater agility.

Corporate Architecture

What is needed is a form of corporate architecture. Most enterprises have not reached the potential that they could have. They have often used old methods and used process-oriented solutions. Instead, they should have used a more datacentric approach that took into consideration the strategic future of the organization.

What has happened in the small company that evolved is not unique. In fact, it is rampant in the world of business today. In 1979, Richard Nolan [1979] of the Harvard Business School wrote an article for the Harvard Business Review entitled “Managing the Crisis in Data Processing.” In it, he described stages of data and processing awareness in companies from his analysis of many major companies. While this assessment is 30+ years old, the problems still exist. The lesson has not been learned or understood as to how to maximize efficiency.

In Nolan’s article he defines six stages of growth, which we will examine in detail:

Stage 1: Initiation

Stage 2: Contagion

Stage 3: Control

Stage 4: Integration

Stage 5: Data administration

Stage 6: Maturity

These are covered in a little more detail following as to how they affect or reflect the growth aspect of the organization.

Stage 1: Initiation. The first few applications to handle the company’s data are developed. These are mostly cost-reducing applications such as payroll, accounting, order control, billing, and invoicing. As each application is implemented, users and operational management start identifying additional business need. The information technology department is small and is a job shop. Overall, information technology exerts no control during this stage.

Stage 2: Contagion. This is when the burgeoning requests for new applications that seem to spread by contact begin to move into swing. This stage is characterized by growth—big and fast. As the user demands for new applications increase, information technology finds itself unable to keep up with the growth. It soon degrades into a period of uncontrolled growth, with each application being built without reference to or consideration of the other applications. The result of this is the proliferation of redundant and replicated data and processes. There seems to be no control, and there is no common focus or planning. Integration is lost, and bridge systems and manual reconciliation units have to be created.

Stage 3: Control. Information technology at this point has recognized that it needs to introduce something to curb the runaway development. The lax controls of Stage 2 have had their impact. Users are frustrated and angry at their inability to get information. Management cannot get the information they need for decision support. There are application backlogs, and application maintenance costs are sky-high. Information technology attempts to again control by restructuring the existing applications, instituting a database management group, and formalizing planning and control by introducing development methodologies. Application development slows while the information technology is restructuring and rebuilding.

Stage 4: Integration. Existing applications are retrofitted. The use of models becomes the center of application development methodology. The users get more information out of access to the data and thereby increase their demands for more from information technology. Information technology expands to meet the demand, and costs spiral upward.

Redundant data and lack of company-wide data analysis frustrate the attempts for the information technology area to develop control and planning applications. Information technology becomes aware of how important the database is in the restructuring and retrofitting process. This represents a fundamental change in the way the applications are built. The change is from simply automating procedures to the examination and consolidation of data for processing. The integration of the data moves the company and information technology into Stage 5.

Stage 5: Data administration. This is the organizational artifact of the integration of the data and the applications. In this stage, organization-wide strategic planning is implemented, information resource management is emphasized. A top-down development methodology is defined that is datacentric and based on stable data models. The reporting data are spun off into reporting and decision support databases. After effort, final application retrofitting is completed on existing applications. Finally, as the company starts to approach Stage 6, applications start to emulate the organizational processes.

Stage 6: Maturity. In this stage, organization-wide data analysis and data modeling have been completed and implemented. Applications mirror the enterprise’s function, and the corporate structure has changed to allow for an architect approach to be fostered and followed.

6.2.2.8 NASA, USGS, NOAA

NASA is a research agency responsible for developing all civilian satellites for the US government. NASA operates many research satellites for both EO and space science application to validate the operational readiness of sensor and spacecraft platform improvements [Fig. 6.5]. Access to data from all US civilian satellites is free and open. NASA data can be searched for at //www.nasa.gov/. The EO [//earthobservatory.nasa.gov] as a part of the EOS Project Science Office at NASA Goddard Space Flight Center serves to provide leadership in organizing open data, convening partners, and demonstrating the power of Big Data analytics through inspiring projects.

The US Geological Survey [USGS] is an operational agency that controls the Landsat satellites and provides Landsat data products in addition to numerous other in situ, aerial, and satellite data within their land remote sensing database that is accessible through the Earth Explorer web interface at //earthexplorer.usgs.gov/.

The US NOAA is responsible for providing weather information derived from ground-based Doppler radar installations, aerial platforms, and satellite data taken over the US. NOAA controls all the polar-orbiting and geostationary weather satellites operating over the US. Access to NOAA data products can be searched for via //www.noaa.gov/.

D Business Intelligence

Like its supply chain solution, SAP initially allowed third-party vendors to supply decision support tools that allow organizations to analyze data from the core ERP system, but SAP recently developed a series of systems to perform these functions. Their offerings include the SAP Business Information Warehouse [BW], SAP Knowledge Management [KM], and SAP Strategic Enterprise Management [SEM].

SAP BW is data warehouse software that performs presentation, analysis, data storage and management, transformation and loading, data extraction, data administration, and system administration. The presentation function includes interfaces for standardized report generation, ad hoc queries, a catalog of available reports, Microsoft Excel extraction, Web distribution, and graphical data visualization. The analysis function contains an OLAP [online analytical processing] engine that enables slicing, drill down, statistical reporting, and other OLAP functions. The system allows users to drill into the operational transaction data in addition to accessing data warehouse contents. Additionally, the software provides functions for data storage and management [storing multidimensional views of data], extraction, transformation and loading [procedures for extracting, cleaning, and validating data], data administration [creating schema, cubes, mapping, etc.], and systems administration [scheduling, monitoring systems, security, capacity planning].

SAP divides its KM initiative into three categories, Knowledge Development, Knowledge Transfer, and SAP Content. Knowledge Development includes tools and consulting services to assist organizations in developing knowledge management programs. Consultants assist organizations in defining needs and planning content requirements. Authoring tools help users create [or convert] company information, training materials, documentation, system simulations, and performance tests into a knowledge repository. The Knowledge Transfer process enables web-based replication of information objects [e.g., documents, presentations] and indexing and retrieval of knowledge content. SAP Content extracts and synthesizes knowledge from the core ERP system in the form of business knowledge, product knowledge, training materials, and documentation. Supporting all functions in the KM initiative is the Knowledge Warehouse [Info DB V.4], which provides the repository and suite of tools to facilitate authoring, translation, distribution, delivery, and retrieval.

SAP SEM is a set of software that enables executives and senior managers to consolidate financial data, conduct corporate planning and simulate business impacts, monitor corporate performance, automate collection of business information, and maintain stakeholder relationships. The software contains five major components. Business Consolidation enables financial consolidation and value-based accounting. Business Planning and Simulation supports the creation of dynamic and linear business models, simulation of scenarios, analysis of scenario results, and rolling forecasts. Corporate Performance Monitor uses industry-specific and customer-developed performance indicators that are linked to a Balanced Scorecard or Management Cockpit to continually monitor performance all levels relative to strategic targets. Business Information Collection supports automated and semiautomated collection of business information from internal and external sources, including an automatic search of Internet information. Stakeholder Relationship Management facilitates the communications with stakeholders regarding business strategy. The module also collects structured feedback from stakeholders and integrates them with the Corporate Performance Monitor and Business Planning and Simulations modules.