Node syncing

Node syncing is the process of determining what data the origin node wishes to share, and the actual data syncing between the two nodes.

Node syncing consists of two parts:

structure sync determines the structures that the destination node can access,
data sync syncs the data from the origin node onto the destination node.

The two nodes must go through node pairing before they can perform any syncing.

Structure sync

Structure sync is the process of determining what structures the destination node is allowed to access. In our case, we specify modules and module fields.

The origin node has full control over what data the destination node is allowed to access. This can be as simple as allowing access to specific modules and as complex as allowing access to only specific module fields (configured in the Corteza Low Code administration panel, or via the API).

The origin node defines exposed structures: The origin node specifies what structures the destination node is allowed to access (modules and module fields). The origin node can also change or remove these structures after the initial configuration.
The destination node consumes shared structures: The destination node fetches available structures (modules and fields) from the origin node and defines a set of mappings to determine how (if at all) the data is stored. The destination node can also change or remove these mappings after the initial configuration.

Data sync

Data sync is the process of syncing the data from the origin node onto the destination node. In our case, we sync records.

The destination node has full control over how the data is stored (if at all) via field mappings.

The data syncing uses Low Code services and storage layer, which simplifies the overall architecture.

The system design allows Federation to be decoupled and moved elsewhere.

This part of the node sync is lengthy and complicated, so we provide two diagrams to help you visualize it.

The diagram outlines the entire data syncing process from the API request to field mapping and storage from the destination node perspective.

Figure 1. The diagram outlines the entire data syncing process from the API request to field mapping and storage from the destination node perspective.

The diagram outlines the entire data syncing process from the API request to field mapping and storage from the origin node perspective.

Figure 2. The diagram outlines the entire data syncing process from the API request to field mapping and storage from the origin node perspective.

The destination node requests changed data: The destination node requests any origin node data changes that have occurred since the last successful data sync. The changes are fetched for each structure (module) on a fixed endpoint.
The origin node provides changed data: The origin node determines what data (records) changed since the last successful data sync. The data is passed through access control where we remove any values we do not wish to share. Lastly, the data is encoded into the requested format and provided to the destination node.
The destination node stores the data: The destination node firstly decodes the data, then converts the data based on field mapping definition, and lastly creates or updates store entries.
The destination node updates the syncing status: The destination node updates the status of the data sync, taking note of the timestamps to enable future data syncing.