The price of light is less than the cost of darkness. Arthur C. Nielsen
Integration and Data Load Summary
This article discusses the integration and data load step in our hypothetical project to reach compliance with FERC Order 881. This step can be completed as often as needed within the project to reach a state of “done,” but each project is different, so your experience may differ from our 9-month project plan. The output for this step will involve passing master approval for implemented data loading and interfacing.
System Configuration Review:
In the previous article, we discussed the system configuration, which involved providing configuration and implementation for all approved requirements related to the project. While there is a lot of flexibility in this stage, it is crucial to pay attention to details.
Integration and data load
During this phase, you may ask, “Are we there yet?” The answer could be yes, but it is also possible that you have just completed your first work package. Integration and data load can run parallel to the System Configuration stage. If so, the project may still have a few design issues and data clean-up. The main objective is to provide the definition, design, implementation, and approval of all data intelligence related to our FERC 881 requirements. It is time to provide structure for:
- Data Modeling and Data Mapping
- Data Customization, if required
- Data Extraction, Interpretation, and Conversion Rules
- Synchronization logic and data validation
- Scheduling, Trigger Notifications
Data Modeling and Data Mapping
First, the main goal of data modeling is to establish consistent data standards for your entire organization. It is used to turn the abstract into a visual representation. Data models aid in data governance, compliance, and data integrity.
Data Modeling can be broken down into physical, logical, and conceptual steps. Conceptual data models provide a high-level visualization of business processes and rules. They help align business stakeholders, system architects, and developers on project requirements and information.
A logical data model defines the data elements and relationships in a project. It shows the names and attributes of entities in the database, specifically manufacturers name plate data, including model numbers; in addition, the IPS® asset registry can display any custom properties needed in your organization.
The physical data model specifies how data will be stored, including storage needs, access speed, and redundancy details. It’s a technical design used by database analysts and developers to define data types and requirements.
5 Best Practices for Data Modeling:
1. Clearly define the scope and stakeholders.
2. Standardize your naming conventions
3. Organize data into tables to eliminate redundancies or anomalies.
4. Use Denormalization to optimize query performance.
5. Apply Indexing Strategies to improve data retrieval.
Data Mapping connects data fields from one source to another, reducing errors and preparing your data for data reporting.
Data Customization
Data Customization may be necessary when creating an alias for data items or sorting data output. For example, suppose a piece of equipment or an entire substation was acquired in the past, and the previous equipment owner named an ABC Breaker by a different naming convention than what you use with all your other breakers. In that case, you must create an alias for that breaker to match your naming convention.
Data Extraction, Interpretation, and Conversion
Clear and specific conversion requirements help ensure the data conversion process is accurate and efficient. Interpretation involves evaluating the source data’s quality, completeness, and compatibility. Careful planning and execution are key for successful data extraction. The extracted data should be validated to ensure it is complete, relevant, and accurate.
Data arriving from multiple sources can be disordered and perhaps even duplicated. Data extraction and ingestion should always include data clean up. However, this can happen in the source system, in transit (typically the most efficient) or in the destination system. The Data Interpretation’s goal is to standardize all data. For network model data, IPS® fully supports any CIM compliant data export.
Finally, data conversion rules ensure that converted data meets the quality and accuracy standards, allowing data to be shared across various applications.
Data Questions to ask:
- What migration scripts are available?
- Loading base Network Model
- Defining update processes
- Loading identified MLSE objects
This stage will involve obtaining master approval for implemented data loading and interfacing. It will also provide input for testing and training for the FERC 881 project.
Synchronization logic and data validation
Synchronization logic will establish consistency between the source and target data sources. Data validation provides the stakeholders with reliable information to make best-case decisions. It is accomplished by taking a sample containing all the required data. The source data is matched with the schema of the destination, and it is then checked for accuracy and completeness.
Scheduling, Trigger Notifications
Decisions on what scheduling and trigger notifications to implement are necessary and should be well-defined by the project. With the FERC 881 mandates you may require notifying users when the temperature has varied outside the projected targets. Setting up automatic reminders or just reporting the actual temperatures for an area could be helpful for compliance support.
The key to successful data migration lies in data-mapping. Once data has been defined and verified it becomes much easier to create data validation scripts and data migration scripts. Successful data mapping makes the difference between successfully automating data migration or falling back on manual input.