Node syncing

Node syncing is the process of determining what data the origin node wishes to share, and the actual data syncing between the two nodes.

Node syncing consists of two parts:

structure sync determines the structures that the destination node can access,
data sync syncs the data from the origin node onto the destination node.

The two nodes must go through node pairing before they can perform any syncing.

Structure sync

Structure sync is the process of determining what structures the destination node is allowed to access. In our case, we specify modules and module fields.

The origin node has full control over what data the destination node is allowed to access. This can be as simple as allowing access to specific modules and as complex as allowing access to only specific module fields (configured in the Corteza Low Code administration panel, or via the API).

The origin node defines exposed structures: The origin node specifies what structures the destination node is allowed to access (modules and module fields). The origin node can also change or remove these structures after the initial configuration.
The destination node consumes shared structures: The destination node fetches available structures (modules and fields) from the origin node and defines a set of mappings to determine how (if at all) the data is stored. The destination node can also change or remove these mappings after the initial configuration.

Data sync

Data sync is the process of syncing the data from the origin node onto the destination node. In our case, we sync records.

The destination node has full control over how the data is stored (if at all) via field mappings.

The data syncing uses Low Code services and storage layer, which simplifies the overall architecture.

The system design allows Federation to be decoupled and moved elsewhere.

This part of the node sync is lengthy and complicated, so we provide two diagrams to help you visualize it.

The diagram outlines the entire data syncing process from the API request to field mapping and storage from the destination node perspective.

@startuml
skinparam responseMessageBelowArrow true
actor "Origin Node" as NodeOrigin

box "Destination Node" #f7f7f7

participant "Federation Service" as FederationService
participant "Compose Record Service" as ComposeRS
database Store
participant "Federation Data Service" as FDataService
participant Decoder

activate FederationService

NodeOrigin <-> FederationService: Get federated records.

Store <-> FederationService: Get module M properties and field mapping definitions.

note over Decoder
The decoder converts federated records
from a specific format into the internal
format used by Corteza.
end note
Decoder <-> FederationService: Decode federated records into the internal structure.

FDataService <-> FederationService: Apply field mapping to obtain the final set of records to create or update.

FederationService -> ComposeRS: Create or update records
activate ComposeRS
ComposeRS -> Store: Write data to the store layer.

ComposeRS -> FederationService: status
deactivate ComposeRS

FederationService -> Store: Update data sync status
deactivate FederationService
end box
@enduml

The diagram outlines the entire data syncing process from the API request to field mapping and storage from the origin node perspective.

@startuml
skinparam responseMessageBelowArrow true
actor "Destination Node" as NodeDestination

box "Origin Node" #f7f7f7

participant "Federation Rest Controller" as FRestController
participant "Federation Service" as FederationService
participant "Compose Record Service" as ComposeRS
database Store
participant "Federation Data Service" as FDataService
participant Encoder

activate NodeDestination
NodeDestination -> FRestController: Get federated records.
note over NodeDestination
This will only provide the data
that was updated after the last
successful data sync.
end note
activate FRestController

FRestController -> FederationService: Get federated records.
activate FederationService

FederationService <-> Store: Get module M properties and mapped fields.

FederationService -> ComposeRS: Get filtered Compose records.

activate ComposeRS
ComposeRS <-> Store: Get filtered Compose records.

ComposeRS -> FederationService: List of filtered Compose records.
deactivate ComposeRS

FederationService <-> FDataService: Prepare final list of records to share.
note over FDataService
This step removes fields
we do not wish to share.
end note


note over Encoder
The encoder converts records
from a specific format into the
requested format.
end note
FederationService <-> Encoder: Encode records into the requested format.


FederationService -> FRestController: Final list of federated records.
deactivate FederationService

FRestController -> NodeDestination: Final list of federated records.
deactivate FRestController
deactivate NodeDestination

end box
@enduml

The destination node requests changed data: The destination node requests any origin node data changes that have occurred since the last successful data sync. The changes are fetched for each structure (module) on a fixed endpoint.
The origin node provides changed data: The origin node determines what data (records) changed since the last successful data sync. The data is passed through access control where we remove any values we do not wish to share. Lastly, the data is encoded into the requested format and provided to the destination node.
The destination node stores the data: The destination node firstly decodes the data, then converts the data based on field mapping definition, and lastly creates or updates store entries.
The destination node updates the syncing status: The destination node updates the status of the data sync, taking note of the timestamps to enable future data syncing.