Purge Data

Purging Device Data

Note: The action to Purge Data is related to the Thin Data action but Purge Data removes all records within the specified time window.

This topic addresses purging of selected Node records for a specified time period. It is a distinct and separate action from deleting or purging Nodes, or purging expired device data from the database. Do not confuse the Purge Data function with either of the following operations:

Delete Nodes: Delete from a Point in Time or Purge from system - Deleting a Node from a point in time or purging a Node and its records from the system are administrative actions that are useful for appropriate circumstances. See Deleting Nodes and Purging Records for more information.
Purge expired device data from the database - Purging expired device data from the database is an optional part of a system command to Perform Database Maintenance on your CygNet Measurement system. See Perform Database Maintenance for more information.

Use the Purge Data button to access options to purge historical data records from the system for selected device Nodes, from a specified begin time (either going forward for all collected records, or until a specified end time). This administrative option can be used as an alternative action for circumstances where it is not appropriate to purge the entire Node and start again. When device data is purged using this action, the Node and its remaining data remains in the system.

To purge duplicate historical data records for selected device Nodes, while retaining master records for the specified time range, use the Thin Data action instead. See Thinning Device Data for more information.

Note: Be certain of any decision to purge data from your system. The Purge Data action cannot be undone.

CygNet Measurement supports sending a command to purge all device records, for a single Node or for multiple Nodes at a time, via the Administrative menu in FMS Explorer only. The action considers the defined normalization views. The selected data is removed from the system entirely, and an audit record of the purging action is maintained by CygNet Measurement. Once the data is purged, the Node's collection status will revert to "Not requested" for that time period, and it will appear to the system as though data for the device was never collected within the specified date/time range.

Note: The dates/times specified for the Purge Data action are always in Device time (D).

The command to purge device data can be executed when the following is true.

The data has not been closed.
For raw liquid periodic, batch, or ticket data, the data has not been exported.

Example

If it is discovered that a device has been receiving "bad" data as of a certain time period, an administrator may decide to discard all data following that time, while retaining the previous (and possibly subsequent) "good" data for the Node. Once the problem causing the problematic data has been resolved, data can again be collected for the device.

You must have proper security authorization to perform these actions. See FMS Security (and NODE security event) for information about configuring security access for purging device data.

Purging Data from Device Nodes

To purge data from a device Node record, open the Admin menu in FMS Explorer, and then click Purge Data to access the Purge Data dialog box and define the action details.

Purge Data

The following table describes options found on the Purge Data dialog box.

Option	Description
Node list
Group	Click to select the Group filter from the drop-down menu. Options are as follows. All Nodes General Station By Category Default value is All Nodes.
Node category	Click to select the Node category filter from the drop-down menu. Options are as follows. Device Gas Device Liquid Device Default value is Device.
Filter	Type a filter into the text box, and click the filter type option to use. Options are as follows. Name - Node name Desc - Node description ID - Node ID Defaults are * (multiple character wildcard) and Name.
Retrieve Nodes	Click to apply the filters selected in the Group , Node category, and Filter fields and view the results. Selection of multiple Node results is supported for the data purging action.
[Node Icon]	Lists the icons indicating Node type for each Node that meets the defined filter criteria.
Node Name	Lists the Node names that meet the defined filter criteria.
Node Desc	Lists the descriptions of the Nodes that meet the defined filter criteria.
Audit history	Click to view the Audit History of Purge Data actions for the selected Node. See Viewing Audit History for more information. Note: Audit history can only be shown for a single Node at a time. If multiple Nodes are selected when you request audit history, results will be shown for the first Node selected. When no Node is selected, the button is unavailable.
Options
Purge begin date (D)	Enter the date/time, in Device time (D), from which to purge the device Node data. All data will be purged from the date/time specified here, either going forward for all collected records (default) or, until the specified Purge end date, if the Purge all records beyond begin date check box is cleared. Note: For hourly devices, select Purge begin date values to encompass the full hour containing the beginning of the data you want to purge. For daily devices, select Purge begin date values to encompass the full day containing the beginning of the data you want to purge, e.g. set the time equal to 00:00:00. Select the date/time from which to remove the selected Node data, in the following ways. Click on the single down arrow to access the calendar and navigate to the desired start date, or select the part of the date you wish to change and type the desired value. Click to select the part of the start time you wish to change and navigate with the paired up/down arrows to change it.
Purge all records beyond begin date	Select the check box if you want to make the purge action persist from the Purge begin date (D), going forward for all collected records. Clear the check box to enable the Purge end date field and specify a time range if you want to make the purge action range between the Purge begin date (D) and the Purge end date (D) only.
Purge end date (D)	If Purge all records beyond begin date is cleared, enter the date/time, in Device time (D), up until which you want to purge the device Node data. All data will be purged from the Purge begin date specified, going forward in time until, but not including, the date/time specified here. Note: For hourly devices, select Purge end date values to encompass the full hour containing the end of the data you want to purge. For daily devices, select Purge end date values to encompass the full day containing the data you want to purge, e.g. set the time equal to 12:00:00 (midnight). Select the date/time until which you want to delete the selected Node data, in the following ways. Click on the single down arrow to access the calendar and navigate to the desired finish date, or select the part of the date you wish to change and type the desired value. Click to select the part of the finish time you wish to change and navigate with the paired up/down arrows to change it.
Records to purge	Select at least one of the following check boxes to include the selected record types in the data purge. Clear the check boxes to exclude selected record types from the data purge. Note: Be certain of your selection(s), as there is NO "undo" for this action. Alarm - Select the check box to include alarm records in the data purge. Batch - Select the check box to include batch records in the data purge. Note: Raw batch records that have been exported cannot be purged. Configuration, event, and gas analysis - Select the check box to include configuration, event, and gas analysis records in the data purge. Note: This purge action presents a Select Recalculation Process dialog box. Select a recalculation process to apply to associated history records. See Purging Configuration or Gas Analysis Data for more information. Periodic history - Select the check box to include periodic history records in the data purge. Note: Purging Periodic history data also impacts the normalized data that has been aggregated into station meter and station group data, and/or that has been used in normalization views sent to the SQL database or SCADA. Product - Select the check box to include product records in the data purge. Ticket - Select the check box to include ticket records in the data purge. Note: Raw ticket records that have been exported cannot be purged.
Purge	Note: Be certain of your action, as there is NO "undo" for this action. Click Purge to purge the selected device Node data from the system. This will completely remove the records from the start date/time forward, or until the specified end date/time, and will also historize the change. A confirmation message will appear, asking you to verify that you wish to purge the data. Click Yes to continue. A second cautionary message will appear, reminding you that purging data is permanent and complete; it cannot be retrieved. Click Yes to complete the purge action. Once the data purge is complete, a command dialog box will appear, displaying results of the Purge Data command. Results are displayed in table form in the top pane. Click on any data row to view detailed command results for the selected record in the bottom pane. Note: Make a plan to manually re-collect data as needed. The Request New Data command will not automatically be aware of changes due to a device data purge. Any purged data must be specifically re-collected manually.
Close	Click Close to close the dialog box.

Purging Data with Dependencies

Device Node data with dependencies can still be purged. When data next comes into the system to replace the purged data, associated values will be updated.

Purging Configuration or Gas Analysis Data

When you click Purge - and the Configuration, event, and gas analysis check box is selected and included in the purge action, but the Periodic history check box is not selected so periodic data is not being purged - the Select Recalculation Process dialog box appears so that you can select the recalculation behavior to apply to affected historical records.

To Select Recalculation Behavior

When the purge includes configuration or gas analysis data that could affect the calculation of historical records, the Select Recalculation Process dialog box appears after the confirmation and cautionary messages. Make a selection as follows.

Click a radio button to select the appropriate recalculation process from the available options.
- Recalculate flow - Select this process for most data purges. This process will recalculate the energy, volume, and/or mass values for any historical records affected by the current data purge.
- Recalculate VCF - Select this process to fix configuration value mismatches between FMS and the device. This process will recalculate only the VCF value, for any historical records affected by the current data purge.
- None - Select this process to purge data only. This process will not recalculate any historical records.
Click Save to save and apply the recalculation process selected, or click Cancel to abandon the change(s).

Auditing Purged Device Data

No record is kept of the specific data that is purged, but the FMS maintains an audit trail describing the purge data operation. If a purge action is unsuccessful, an error will be logged. Errors can also be viewed in the command log file, or via the service console window.

Logs can be viewed through the Commands menu View button, by selecting User/Purged Device Data for the Command type, or through the CygNet Audit Service.

See Viewing Command Logs in this section and Audit Service View in the Auditing section for more information.

Re-Normalizing Station Data

If any normalization views have been defined, each defined normalization view will be traversed as part of the data purge operation.