IWeb Administrator Guide

Deduplication Settings

The following settings pages are located in the Deduplication category on the Administration Main Menu page (Administration > Deduplication):

Manual Deduplication Ambiguous ID Master Deduplication by External Source Merge History Report Define Deduplication Process

Separate Bad Merges Run Deduplication Schedule Recurring Deduplication Stop Scheduled Deduplication

Manual Deduplication

This option allows an administrator to view duplicate patient records and make a decision whether to merge the two (or more) records into one, or leave them as separate if they are not similar enough. The goal is to group records belonging to the same patient so that a complete picture of the patient's vaccination history can be viewed.

This is a manual process where the administrator manually reviews both the incoming patient record and the existing record from the Reserve table, then makes the decision whether to create a new patient record or merge the patient records.

If the Administration > Properties > Deduplication > Prevent Merge of Different Birth File Numbers option is enabled, the import logic checks to see if the patients have different birth file numbers and prevents a merge if they are different. However, the logic does not apply to master merges (including user reported duplicates)

When records are imported and the record is a possible match with a queued deduplicate record, the new record is staged and not processed until the previous possible match is processed through Manual Deduplication. This can occur when records from a high-priority Organization (IRMS), such as Death Records, are staged rather than processed. By enabling the logic, the state can prioritize a particular Organization (IRMS)'s records in order to process them first.

Deduplications to the database can occur for various reasons. See Deduplication: Additional Information for more information.

Only Registry Client users with the System Administration permission can perform Manual Deduplication.

To access this option, click the Administration > Deduplication > Master Deduplication link. The Manual Deduplication page opens. The fields available on the page are as follows:

Click Continue to start the process. Depending on the number of matches located for the incoming record, one of three pages may appear:

• Page 1 - Indicates a matching reserve record
• Page 2 - Indicates multiple matching reserve records
• Page 3 - Indicates a match to multiple master records

The columns on these pages are as follows

Note that the vaccinations for the incoming record and system potential matches, if any, are shown in the Patient Vaccination Record section at the bottom of the page.

Click one of the buttons (and/or option) to continue:

• Return to Select Screen - Click this button to return to the previous page without making any decisions.
• Previous - Click this button to display the previous record in the group of displayed records, if there is one.
• Skip - Click this button to keep the record in the Possible Duplicate database table when it is unknown whether or not the record should be merged.
• Merge - Click this button when the patient displayed in the Potential Match in System column is determined to be the same patient as the Incoming Patient, to merge the data for the one patient.
• New Patient - Click this button when the two patches are unique and should be kept in separate records.
• Yes, Make Incoming Record Historical - Select this option (located toward the bottom of the first section on the page) to flag the incoming record as historical.
• Collapse - Select this option to collapse the records so that only the most important fields are displayed on the page.
• Merge (checkboxes) - Displayed when more than one matching record is found. Select the checkboxes for the patients that should be merged to create one record.
• Master Record (radio buttons) - Located under the merge checkboxes. Select one record to designate it as the Master patient record.

Ambiguous ID

This option is used to correct records when unique patient IDs from incoming Organizations (IRMSes) no longer appear to be unique. The administrator can decide if the record actually does represent the same person or if the submitting site has a corrupt ID system. These records are stored in the Patient Ambiguous ID table.

This is a deduplication operation. As a result, it locks deduplication while it processes (Incoming is Different, Bypass button clicked). Due to the locking of records, only one person should process ambiguous IDs at a given time. Additionally, if there are other deduplication operations running, ambiguous IDs cannot be processed.

If two records are coming in from the same Organization (IRMS), the newest record should always overwrite the existing record, except when the newest record has a NULL value. When a record has a NULL value, the original field date is kept instead of a blank field.

Ambiguous ID Process

The process is as follows:

1. The incoming record is staged into the h33_patient_pre_reserve database table.
2. The deduplication logic processes the incoming record.
3. The first deduplication step is to look for an existing record with the same irmsSysId/irmsPatId. If found, the deduplication runs for that pair (the existing record and the incoming record with the same irmsSysId/irmsPatId). If either a definite or a possible match is located, the record merges the new data and deduplication is finished for that patient. If a match is not located, the patient is staged to Ambiguous ID.
4. If the user reviews the records individually (i.e., the user clicks the Examines Records button), and
   • The Incoming is Different, Bypass button is clicked, then the existing irmsPatId is changed (this is the ###.BYPASS.### that can be seen at times), the incoming record is placed back into the h33_patient_pre_reserve table for reprocessing, and when deduplication runs, it does not find the matching patient. The incoming record is processed through deduplication.
   • The Incoming is Same, Update button is clicked, then it depends on the scenario. Scenario 1: Only one reserve record is the matching irmsSysId/irmsPatId: The existing patient is overwritten with new patient data; the old patient name, etc., is completely gone (except in the history table); and this is treated as an update. For example, the deduplication logic determined that the incoming irmsSysId/irmsPatId record matched the existing one exactly except the health plan number was different, so the old health plan number is overwritten and only exists in the history. Scenario 2: Patient has multiple reserve records: The matching record is separated (a registry-level property enabled/disabled with a script called Separate Patients in Ambiguous ID) first, if enabled, when the button is clicked. If disabled, the matching record is not separated and the associated reserve record is updated. The master updates (or not) based on the normal rules (ownership, historical, etc.) The script can be obtained by logging a help desk ticket. The existing patient data for the separated record is overwritten with the new patient data, and all remaining reserves stay with the original record.

To access this option, click the Administration > Deduplication > Ambiguous ID link. The Ambiguous ID page opens. The columns and fields available on the page are as follows:

The buttons available on this page are as follows:

• Refresh Records - Refreshes the page to show the updated list and numbers of records.
• Export - Export the data into a CSV (comma-separated value) file that can be modified using Excel or an equivalent spreadsheet application. To export all of the records, just click Export. To export only specific records, use the Jump to Record, Patient DOB or Organization (IRMS) Patient ID field to narrow the list, then click Export.
• Examine Records - Display the first record in the total group of records.
• Deactivate - Remove all data ever sent by the Organization (IRMS) from the registry. This is only used when there is a significant amount of bad data associated with a specific Organization (IRMS).
• Process - Send all of the records through for the specific Organization (IRMS). Be sure to review the various records before using this option. The newest record does not overwrite existing fields containing NULL values. Use this button when you have manually reviewed the Ambiguous ID records while processing all records.
• Back - Display the previous page.

After clicking Examine Records, the Ambiguous ID - Examine Records page opens. Use the Next and Previous buttons to easily navigate forward and backward among the records. The incoming patient's data is listed in the left column and the data for the previous record in the system is displayed in the right column. Review the two records, determine what to do with the record, and select one of the available buttons:

• Incoming is different, Bypass - Resets the older record's patient ID to another unique ID, allowing the incoming record to come in as a new record so that it does not overwrite the existing record. The existing record is no longer able to be updated from the Organization (IRMS). A confirmation dialog box opens; select OK to indicate that the patients are different or Cancel to abort the bypass and return to the page.
• Incoming is same, Update - Indicates the incoming patient record is the same person as the previous record in system (column on the right) and allows the two records to merge.
• Jump to Record - Enter a record number in the textbox, then click the button. Displays the specific record within the group of records.
• Return to Select Organization (IRMS) - Reopens the previous page.
• Delete Incoming Demographic, keep Imms - Deletes the incoming patient record from the staging table, but processes the incoming vaccinations and attaches them to the existing patient's record.
• Delete all incoming data - Deletes the incoming patient record and the related immunization records from the staging table.

Master Deduplication by External Source

This option allows an administrator to view duplicate patient records and make a decision on whether to merge the two records into one or leave them as separate if they are not similar enough. The goal is to group records belonging to the same patient so a complete picture of the patient's vaccination history can be viewed.

Users must have Registry Client access with the System Administration permission to use this option.

When using this option, the Master Patient IDs are required.

To access this option, click the Administration > Deduplication > Master Deduplication by External Source link. The Master Deduplication by External Source page opens. The fields available on the page are as follows:

Click Continue to open the Compare Potential Duplicates in the Master Table page. The first patient's master record data is displayed in the first column, the fields are displayed in the second column, and the second patient's master record data is displayed in the third column. If there is vaccination information available, it displays in the Patient Vaccination Record section toward the bottom of the page.

Compare all of the fields and click one of the buttons to continue. Note that if the two patient records are merged incorrectly, a patient's record may show vaccinations that (s)he did not receive and, therefore, the patient may be under-immunized. Conversely, if a patient's records are not merged when they should be, vaccinations they did receive may not appear and they may be over-immunized.  Shared vaccination records make is more likely that the records represent the same person. However, when in doubt, keep the records separate.

• Cancel - Return to the previous patient without making any changes.
• Merge First => Second - Merge the first patient's data into the second patient's master record.
• Merge Second => First - Merge the second patient's data into the first patient's master record.

Merge History Report

The Merge History Report lists automatic and manual merges that were performed in IWeb. The types of merges included are User-Reported Duplicates, Master merges (completed via Master Deduplication by External Source), Master Duplicate Scan (a nightly run job that scans for duplicates0, Third Record Source (duplicates that are found based on new incoming records), and Manual merges (merges that are reviewed on the Manual Deduplication page).

The report output includes the report parameters used to generate the report, as well as SIIS Patient IRMS ID and IRMS Patient Detail ID.

The report output omits one of the patient's SIIS Patient IDs when the merge is a Manual merge. The reason for this is because when a patient is queued for Manual Review, the SIIS Patient ID is not created until after the patient is reviewed.

The report is based on three tables:  H33_MERGE_AUDIT_HISTORY (contains data for user-reported duplicate merges), H33_MERGE_LIST (contains data for Master merges, Master Duplicate Scan, and Third Record), and H33_MANUAL_DEDUP_DECISION_LOG.

To run the report, click the Administration > Deduplication > Merge History Report link. The Merge History Report page opens. Enter the information into the report criteria fields as needed. The fields available on the page are as follows:

When all of the report information has been entered, click one of these buttons:

• Create Report - Runs the report and displays it in a new tab. Use the browser's print function to print the page.
• Export Report - Runs the report in a tab-delimited format and opens it in the default spreadsheet application.

Define Deduplication Process

This option is used to define the reasons a duplicate record could occur. Some examples include:

• First Name spelling
• Last Name spelling
• First Name hyphenation
• Last Name hyphenation

To define deduplication reasons, click the Administration > Deduplication > Define Deduplication Reasons link. The Define Deduplication Reasons [Search/Add] page opens. Enter the search criteria (optional) and click Search. The search criteria fields are as follows:

If a match is found with the entered search criteria, it displays in the Search Results list. Otherwise, the page re-displays with the entered information and an Add button. Click Add to add the new deduplication reason.

Search results display in the Search Results section toward the bottom of the page. Click the arrow button the Select column to edit it in on the Define Deduplication Reason [Edit] page. Make any changes and click Save.

Separate Bad Merges

This option is used to separate patient records that were erroneously merged. An indicator that a bad merge has occurred is if the record contains vaccinations that are known to be false.

To perform this option, click the Administration > Deduplication > Separate Bad Merges link. The Separate Bad Merges page opens. Enter the Master Patient ID number and click Get Reserve Mapping Records. The Contributing Reserve Records page opens.

There may be more than two Reserve records that can be separated. Select the checkbox to flag the reserve record(s) to separate and select one of the records to be the Master record. Click Separate Reserve Records as many times as necessary. The new Master Record ID appears at the top of the page.

Click the Back button to return to the Administration Main Menu page.

Run Deduplication

This option is used to start the deduplication process including concurrent multiple runs. The scheduler utilizes all five threads for deduplication. If deduplication is running for all Organizations (IRMSes), only one deduplication process can run at a time. If deduplication is running for one specific Organization (IRMS), multiple deduplication processes can run, but only for one Organization (IRMS) each. The progress tables add the Organization (IRMS) and the audit tables log the progress tables.

The deduplication logic checks for dedup processes for the specific Organization (IRMS). If there is no Organization (IRMS) specified, no other dedup processes are allowed to start.

To run deduplication, click the Administration > Deduplication > Run Deduplication link. The Run Deduplication page opens. The Deduplication Type defaults to Deterministic unless the Probabilistic option has been defined. (See Deduplication: Additional Information for more information.) Optionally select a specific Organization (IRMS) from the drop-down list and click Run to start the deduplication job process. The message A deduplication job has been scheduled appears at the top of the page.

Schedule Recurring Deduplication

This option allows an administrator to set up a schedule to automatically run the deduplication process. To do so, click the Administration > Deduplication > Schedule Recurring Deduplication link. The Schedule Recurring Deduplication page opens with the header Indicate when deduplication should occur, using the form below.

Enter the frequency, hour, and minute the job should run (all fields are required) and click Save.

Stop Scheduled Deduplication

This option allows an administrator to stop a scheduled deduplication process.  To do this, click the Administration > Deduplication > Stop Scheduled Deduplication link. The Stop Scheduled Deduplication page opens with the header Currently Running Procedure. If there is a procedure currently running, it appears at the top of the page. Click Refresh to refresh the page if necessary.

Click Run Stopall to immediately terminate any database procedures. Click the Back button to return to the Administration Main Menu page.

Related topics:

• Administration Settings
• Settings
• User Management
• Define Values
• Patient Management
• Vaccine Management
• HL7 Management
• School Nurse
• Arizona
• Child Care Settings
• Immunization Schedule Settings
• SIF Import Settings
• Countermeasure and Response
• Site Administration