Owner: Ben Guaraldi and Vladimer Shioshvili
This is an overview of the DATIM4U mechanism exchange process for transferring Mechanisms from the global DATIM FactsInfo system to the DATIM4U Node. The DATIM FactsInfo System is the source system for DATIM mechanisms.
The following system components are involved in the transaction:
- The Global PEPFAR System
- The FactsInfo system which is the source for PEPFAR Mechanisms
- The sync.datim.org server, running mechanism synchronization cron jobs
- The Global InfoMan system which provides the standards-based process for transferring mechanisms
- The 4U Node's InfoMan system which provides the standards-based process for transferring mechanisms
- The 4U Node's DHIS2 system which imports the mechanisms
The process will move mechanisms from the FactsInfo CSV file to the Node's DHIS2 system.
The steps are:
- Receive Mechanism CSV from FactsInfo. FactsInfo system deposits a CSV file containing latest mechanism data daily using a scheduled job to sync.datim.org server. This is a normal part of the DATIM system and not specific to DATIM4U.
A comma separated values file (CSV) is uploaded to sync.datim.org FTP server by FACTS Info each night, overwriting the previous night's file.
Cron jobs on sync.datim.org process the CSV file to communicate updates to DATIM Global and 4U systems. (The process for updating the DATIM Global's list of mechanisms is not in the scope of this document.)
The nightly file contains all Active applicable mechanisms within FACTS Info. DATIM4U uses this file to make additions and updates to what is currently in the DATIM4U system.
sync.datim.org archives each night's file.
- Populate the Global InfoMan Mechanisms CSD document (DATIM-FactsInfo).
A bash script running as a daily cron job on sync.datim.org sends the CSV to the create_mechanisms xquery on the ILR. It follows these steps:
- Check the FactsInfo CSV file to ensure that expected header is present;
- Convert FactsInfo CSV file (received in Windows 1251 encoding) to utf-8;
- Wrap the CSV file in XML tags to be legible to ILR;
- Clear the DATIM-FactsInfo document prior to sending latest version (using a curl command);
- Send the XML file containing embedded CSV content to the create_mechanisms xquery of the ILR via OpenHIM (using a curl command).
The InfoMan stored query "create_mechanisms" follows these steps:
- Convert the CSV content embedded in the incoming XML file to an XML format based on the CSV file headers;
- Loop through all the unique partners and get a list of partners as CSD organizations (entityID is based on the PrimePartnerID);
- Loop through all the unique agencies and get a list of agencies as CSD organizations (entityID is based on the FundingAgency);
- Loop through the entire list of mechanisms and get a list of CSD organizations with extension (entityID is based on FiscalYear, PlanningReportingCycle, HQMechanismID, and LegacyMechanismID);
- Go through the above three lists and add them to the given CSD document (which has a document name determined by the URL calling create_mechanisms—in this case, it is called DATIM-FactsInfo) or update existing organizations.
- This step is designed to be controlled by a job scheduled from the 4U OpenHIM. It is recommended that the system is configured to run this job nightly and the system is initially configured to support this recommendation. The job refreshes the 4U InfoMan CSD document titled DATIM-FactsInfo. During the cache update process, the additions and changes that were made to the global CSD DATIM-FactsInfo document are refreshed on the node, thus syncing the contents of the two documents.
- Put in the mediator name here what box is this being run on
- Import the updates into the 4U DHIS2 system. The Mechanism Importer is also a scheduled job run from the OpenHIM. Out of the box, it is configured to be run nightly right after Step 3. It uses the CSD to import the mechanisms into DATIM, taking the following actions on all mechanisms that are related to organisation units level 3 or lower on the DATIM4U system:
- Add new mechanisms as category options and category option combos;
- Add new partners and agencies as category option groups;
- Create user groups (users, all mechanisms, and user administrators) for each new mechanism;
- Update mechanism name, agency, partner, dates, or country, if any have changed.
How to manually run the process
Each step in the above process can be run manually. Here's how:
- This is controlled by the FactsInfo system, and manual triggering requires coordination with FactsInfo team.
- First, ssh into sync.datim.org as user facts. Then run /home/facts/facts2ILR.sh
- There are two options:
- Manually trigger the cache refresh through the DATIM4U OpenHIM.
- Log into the OpenHIM
- Select Channels -> AUTO Update Node ILR Facts-Info document from Global.
- Click the Manually Trigger button to initiate the refresh.
- Manually trigger the cache refresh through the InfoMan.
- Navigate to the Infoman's pollService/DATIM-FactsInfo
- Click on the link "Update local cache of DATIM-FactsInfo"
- First, ssh into the DATIM4U node OHIE box. Then run "sudo -i" then "cd /var/lib/mechanismimporter && /usr/bin/nodejs sync/csd" [Note from Ben: This will change soon, hopefully.]
Viewing and checking the process from the Node's OpenHIM perspective
The transaction is facilitated by the DATIM4U OpenHIM. In the 4U OpenHIM, there are two transactions to complete the cache refresh.
- The "AUTO Update Node ILR Facts-Info document from Global" channel is used to trigger the CSD "Update local cache of DATIM-FactsInfo" link.
- The Post to the 4U cache is received from the "Global ILR" channel.
In the Global OpenHIM, there is one transaction. It is executed through the "Global ILR" channel and queries the global DATIM-FactsInfo document for any changed or updated mechanisms.