Services

Introduction

The SBE platform is comprised of many microservices, each responsible for its own resources and potentially multiple features. This document serves as a high level description of each microservice, what resources it manages, and in what capacity it is involved in different features offered by the digital thread.

To make the list more manageable, we will split the services into 3 categories

1. Platform Services

   • Services that are responsible for managing core SBE resources. These services must be deployed to be able to use SBE in any capacity. They power the model/mapping/ontology/channel management systems as well as the UIs SBE provides

2. Pipeline Services

   • Services that are responsible for publishing and syncing data in and out of SBE.

3. DMZ Services

   • Auxiliary services provided by SBE. They are not required for using SBE with external systems and adapters.

If you would like to know the flow of how core features work at a service level, please refer to this document: Diagrams

Platform Services

channel-manager

• Managed Resources

  • Channels, Data Sources, Data Source Types

• Key Features

  • Attach

• Description

  • Responsible for storing and retrieving channels, data sources and data source types. Validates channel properties and is responsible for the attach process. The SDK/adapters ping channel manager when checking if a project is already attached or not.

• Depends On

  • Lifecycle service - validates an entity set exists during attach.

clazz-service

• Managed Resources

  • Clazz Lifecycles, Link Dictionary Entries

• Key Features

  • Ontology Management

• Description

  • Manages all ontologies in SBE. Maintains compiled in-memory classes for faster retrieval. Computes diffs and applies changes using diff payloads. Creates and manages copies of clazzes when branching off of an clazz entity set.

• Depends On

  • Lifecycle service - all clazzes must exist in an entity set.

guardian

• Managed Resources

  • Permissions, Actors

• Key Features

  • RBAC Security

• Description

  • Responsible for security in SBE. Permissions are at the partition level, so any time any user wants to interact with a partition or a resource contained in a partition, guardian is used to validate and authorize the user.

• Depends On

  • Keycloak

indexer

• Managed Resources

  • Elasticsearch indices

• Key Features

  • Indexing SBE platform data

• Description

  • Responsible for listening for resource events over rabbit and sending them to elasticsearch. Indexer populates all SBE indexes.

• Depends On

  • Elasticsearch

lifecycle-service

• Managed Resources

  • Partitions, Entity Sets, Contexts, Subscriptions, Expressions

• Key Features

  • Configuration Management, Subscriptions, Entity Set Copies, Entity Set Configuration, Diff/Merge

• Description

  • Manages the configuration management aspects of SBE. Partitions, entity sets, subscriptions and context are all managed by this service. Any subscription actions first come to this service and then are routed to the model service. Likewise with making copies of ontologies and mappings, the lifecycle service will receive the request first from rest-gateway and then route the request to the corresponding service. Similarly with diff merge, the request first comes to lifecycle service and then is routed to the appropriate resource service (mapping/model/clazz service).

• Depends On

  • Mapping/Clazz/Model service for branching/subscription operations.

link-service

• Managed Resources

  • Links

• Key Features

  • Link Management, Link Projection, Thread Links

• Description

  • Links are their own resource in SBE. For example, when a model is retrieved from the database, a request is made to link service to find all inbound and outbound links for that model. For link projection, whenever an entity set’s context is updated, a link projection event is triggered to re-point any applicable links.

• Depends On

  • Lifecycle Service - Receives context update events for link projection. Also checks context when saving links.

mapping-service

• Managed Resources

  • Mapping Lifecycles

• Key Features

  • Mapping Management

• Description

  • Manages all mappings in SBE. Computes diffs and applies changes using diff payloads. Creates copies of mappings when branching off of an existing ontology entity set that contains mappings.

• Depends On

  • Lifecycle service - mappings must exist in an entity set.

  • Clazz service - the schema of a mapping is based off of a clazz. When saving a mapping the clazz is retrieved from clazz service and used for validation.

model-service

• Managed Resources

  • Model Lifecycles

• Key Features

  • Model Management

• Description

  • Manages of all models in SBE. Computes diffs and applies changes using diff payloads. Creates and manages copies of models when dealing with subscriptions.

• Depends On

  • Lifecycle service - models must exist in an entity set.

  • Link service - when saving a model, any links that it owns is managed through link service. When getting a model, the model is hydrated with links from link service.

  • Clazz service - the schema of a model is based off of the clazz it references in its origin reference. In the case of publishes, this is the clazz the model is mapped to.

rest-gateway

• Managed Resources

  • None

• Key Features

  • Hosting restful APIs

• Description

  • Hosts all SBE APIs that contain /api/v1. This service is responsible for routing rest call to the appropriate service, usually via grpc. The gateway allows us to aggregate multiple resources from different services into one payload.

• Depends On

  • channel manager

  • clazz service

  • lifecycle service

  • mapping servcie

  • model service

  • sychronizer

  • data trader

Pipeline Services

All pipeline services are used for every publish and refresh

change-recorder

• Managed Resources

  • Changelog

• Key Features

  • Calculating commit IDs

• Description

  • Calculates commit hashes for channels. This service is responsible for letting users know if their commit ID is behind and if they need to refresh before publishing.

• Depends On

  • Model service - is notified by model service that models and links are finished saving during a publish request.

  • Synchronizer - notifies synchronizer that the commit ID is finished being calculated on publishes.

data-trader

• Managed Resources

  • External Items (data lake), Item Prints, Audits

• Key Features

  • Storing all external items from publishes in the data lake, audit management, retrying publishes

• Description

  • Stores all external items in the data lake so publishes can be retried without having to use an adapter. Stores all audits generated by the system and serves them to the tracking UI.

• Depends On

  • Synchronizer - sends processed external items to synchronizer on publishes.

  • Interchange - sends processed external items to interchange on refreshes.

interchange

• Managed Resources

  • None

• Key Features

  • Entry point for all publish/refresh commands

• Description

  • All adapters/SDKs talk to interchange and no other pipeline services directly. All publish and refresh commands start at interchange, it is the gateway for adapters. For refreshes, it stores the external items to be given to the adapters in a mongo database until the adapter is ready to receive them.

• Depends On

  • Data trader - on publish is sends all external items received from the publish request to data trader. On refresh it receives all transformed items from data trader.

  • Channel manager - Verifies the commit ID on a publish is up to date.

synchronizer

• Managed Resources

  • Correlations

• Key Features

  • Correlates external items and SBE models

• Description

  • Manages correlations for models and links. A correlation ties an external locator to an SBE model’s FQN.

• Depends On

  • Model service - retrieves models for refresh requests and saves models and publish requests. Saves published models during a publish request. Archives models at the end of a publish request during the synchronized stage.

  • Transformer - sends transformation requests to transformer for both publishes and refreshes..

  • Lifecycle service - Get subscription information during publish and refresh requests.

transformer

• Managed Resources

  • None

• Key Features

  • Transforms external items to SBE models and vice versa

• Description

  • Uses the mappings from mapping service to transform an external item to an SBE model for publishes. Also transforms SBE models to external items for refreshes.

• Depends On

  • Mapping service - retrieves mappings for all transformation processes.

  • Lifecycle service - retrieves mapping context for the channel being transformed.

  • Synchronizer - Synchronizer sends transform requests to transformer and transformer sends the transformed results back to synchronizer.

DMZ Services

ontology-converter

• Managed Resources

  • None

• Key Features

  • Loading OWL files into SBE

• Description

  • Allows users to load OWL ontology files into SBE. This service converts the rdf/xml OWL file to SBE clazzes and then saves them to the clazz service.

• Depends On

  • Clazz service - saves the transformed OWL file as SBE clazzes after transformation.