Service Offerings
Egress Service
Egress Service
Egress Service
Key Requirement
- Scalable API extraction service to extract data from source system, such as Salesforce.
- Technology considerations
- Native API integrations to source systems like Salesforce.
- Needs development effort to build and maintain API integrations as product onboards different sources.
- API throttling, scalability, resilience, self healing capabilities for seamless operation
- Make API integration client agnostics, to be able to scale from one customer to another.
- Tech Stack:
- Python/Java based egress service orchestrated through ETL engines like Airflow, Luigi etc.
Service Layer
Egress Service
Egress Service
Key Requirement
- Serve all client requests to retrieve data effectively and securely
- Technology considerations
- Offering cache service to improve the performance of fetching the data and documents.
- Offering security on the service layer to allow authorized users to access the data from data vault.
- Offering access level security to incorporate data level security
Storage Layer
Egress Service
Transformation Service
Key Requirement
- Long term, persistent storage layer to store extracted data onto S3 Data lake/Dynamodb/mongoDb document store
- Capable of cold storage for archival and quick retrieval
- Technology considerations
- Offering cold storage capabilities for long term cold storage/archival of data from source system
- Offering fast retrieval capabilities for, high value/often used data elements
- Tech stack: S3/Glacier
Transformation Service
Transformation Service
Transformation Service
Key Requirement
- Transform and Parse data from Egress service to store to storage layer
- Technology considerations
- Schema on read capable service that is capable to handling changing payloads
- Transform varying payloads from the egress service to Normalized table structures, add partitioning capabilities for long term data vault storage
- Schema metadata collection service to collect changes to schema layer
API endpoint
Transformation Service
API endpoint
Key Requirement
- Service to support integration with downstream/target apps
- Service to export data in bulk
- Technology considerations
- Restful API with secured SSL and token authorization to extract data
- Capability to integrate with other application to extract data
- Capability to extract data in bulk to perform local analytics
- Capabilities to perform fast scans to support integrations to online systems.
Security
Transformation Service
API endpoint
Key Requirement
- Authorization and authentication to only approved data in data vault
- Supported by well defined Admin panel supporting standard business hierarchies and roles
- Reporting hierarchies, Department Hierarchies, Admins etc
- Technology considerations
- Capability to integrate with various enterprise SSO services
- Provide standard integration with 3p services such as google authe etc.