The need for a better data exchange solution
Why data governance around data exchange with partners is a pain today? What can be done?
Every organisation exchanges a lot of data daily with its business partners, investors, auditors, regulators, etc. Most of these are shared via SFTP, emails (sometimes with password protection) and APIs.
Data sharing agreements between organisations usually spell out the data transfer purpose, schema, security, purging and other compliance clauses. However, it is tough to enforce them when the operations team uploads the data files to the SFTP folder or shares them via email. API-based data contracts may be better at enforcement, but they require custom implementation on both sides, which is expensive for most situations.
We need an open-source software solution that lets businesses interface with partners in a standardised manner. The solution should help build a decentralised network for data exchange between partner organisations with built-in data governance support, as described below.
Peer to Peer data transfer between organisations via Data Exchange (DX) Nodes
DX Node software runs on the IT infrastructure of each participating business
Partner Registry on DX Nodes with auto-discovery support
Data Contract schema definitions as per the signed agreement
Personally Identifiable Information (PII) Anonymization
Scripting support for data validations
Web Application Interface & API endpoints for uploading to & downloading from nodes
Support for broadcasting the data to multiple partners at the same time
Secure data-in-transit channel using PKI
Role based access controls
Detailed audit logs around each data transfer
Data archiving & purging support
Data synchronization fault tolerance
The solution should be simple, so all organisations can install the DX Node & benefit. With the open source utility as a base, data management vendors can offer commercial solutions to improve the data exchange experience of medium and large organisations. At the same time, small organisations can continue to use the open source solution.