You are reading the documentation for an outdated Corteza release. 2024.9 is the latest stable Corteza release.

Syncing structures

The process of determining what structures on the origin node, is exposed to it’s destination node. The origin node has full control over what data the destination node is allowed to access. This can be as simple as allowing access to only specific modules; and as complex as allowing access to only specific module fields.

The two nodes must be paired prior to this. See Node pairing.

Exposing structures

Firstly, in order to perform any data sharing, the origin node must define what structures (in our case modules and fields) the destination node can access. This can be performed via the Corteza Low-Code administration panel, or directly via the API.

Besides the module itself, the origin node must also specify what fields the destination node is allowed to access.

API

Expose module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X PUT "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID" \
  -H "Authorization: Bearer $JWT"
  -H "Content-Type: application/json" \
  --data "[{
      \"name\":\"LinkedIn\",
      \"label\":\"LinkedIn Url\",
      \"kind\":\"Url\",
      \"is_multi\":0
    }]";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "name": "LinkedIn",
                "label": "LinkedIn Url",
                "kind": "Url",
                "is_multi": 0
            }
        ]
    }
}

See exposed module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID
Example request
curl -X GET "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/exposed" \
  -H "Authorization: Bearer $JWT";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": false,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": false,
            }
        ]
    }
}

Remove exposed module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X DELETE "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID" \
  -H "authorization: Bearer $JWT";
Example response

Change exposed module fields

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X PUT "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/exposed" \
  -H "Authorization: Bearer $JWT"
  -H "Content-Type: application/json" \
  --data "[{
      \"name\":\"LinkedIn\",
      \"label\":\"LinkedIn\",
      \"kind\":\"Url\",
      \"is_multi\":1
    }, {
      \"name\":\"Phone\",
      \"label\":\"Phone\",
      \"kind\":\"String\",
      \"is_multi\":0
    }]";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": 0,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": 0,
            }
        ]
    }
}

Store

Exposed modules

Table 1. federation_module_exposed
Column Type Description

ID

uint64

federation module id

rel_node

uint64

node id (destination node id - who are we sharing to)

rel_compose_module

uint64

module id on source node

fields

json

list of fields

Syncing structures

When the two nodes are paired and the origin node has exposed some structures, the actual structure sync can occur.

Outline of structure sync process on the origin node.
@startuml
skinparam responseMessageBelowArrow true
hide footbox
actor "Destination Node" as NodeDestination
box "Origin Node" #f7f7f7
participant "Federation Rest Controller" as FRestController
participant "Federation Service" as FederationService
participant "Compose Module Service" as ComposeMS
database Store
participant "Federation Structure Service" as FStructureService
participant Encoder
NodeDestination -> FRestController: get structure
note left : only the structure\nthat was updated after\nlast successful sync
activate FRestController
FRestController -> FederationService: get structure for all modules
activate FederationService
FederationService -> Store: get module id list
activate Store
Store -> FederationService: federated modules id list
deactivate Store
FederationService -> ComposeMS: get filtered compose Modules
activate ComposeMS
ComposeMS -> Store: get filtered list of modules
activate Store
Store -> ComposeMS: list of compose Modules
deactivate Store
ComposeMS -> FederationService: list of compose Modules
deactivate ComposeMS
FederationService -> FStructureService: module list + federated module ID list + federated field ID list
note left : omit the fields we do not wish to share
activate FStructureService
FStructureService -> FederationService: module list with specific fields
deactivate FStructureService
FederationService -> Encoder: transform the list to specific structure
activate Encoder
Encoder -> FederationService: transformed list
deactivate Encoder
FederationService -> FRestController: transformed list of modules (structure)
deactivate FederationService
FRestController -> NodeDestination: list of modules
deactivate FRestController
end box
@enduml
Outline of structure sync process on the destination node.
@startuml
skinparam responseMessageBelowArrow true
hide footbox
actor "Origin Node" as NodeOrigin

box "Destination Node" #f7f7f7

participant "Federation Service" as FederationService
database Store
participant "Federation Structure Service" as FStructureService
participant Decoder

FederationService -> NodeOrigin: get structure for module
activate FederationService
activate NodeOrigin
NodeOrigin -> FederationService: structure for module
deactivate NodeOrigin
note left : federated Module structure from Origin

FederationService -> Decoder: structure for module in specific format
activate Decoder
Decoder -> FederationService: structure for module
deactivate Decoder

FederationService -> FStructureService: structure for module and fields
activate FStructureService
FStructureService -> FederationService: remapped module with fields
deactivate FStructureService

FederationService -> Store: write federated modules and fields info
FederationService -> Store: structure sync OK
deactivate FederationService
end box
@enduml
Destination node requests changed structures

Origin node provides a set of endpoints that the destination node can use to fetch shared structures.

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/federation/exposed/modules" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "filter": {
            "query": "after=1600109447",
            "page": 1,
            "perPage": 20,
            "count": 97,
        },
        "set": [
            {
                "type": "GET",
                "rel": "Account",
                "href": "$BASE_URL/federation/exposed/modules/$MODULE_ID?after=1600109447"
            },
            {
                "type": "GET",
                "rel": "Contact",
                "href": "$BASE_URL/federation/exposed/modules/$MODULE_ID?after=1600109447"
            }
        ]
    }
}
Destination node fetches updated structures

The destination node requests structure definitions based on the above provided set of endpoints. The origin node provides the structure definition with respect to the fields that the destination node is allowed to access.

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the origin node
$NODE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/federation/exposed/modules/$MODULE_ID" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": false,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": false,
            }
        ]
    }
}
Destination node updates its store

Once the structure sync is finished, the destination node writes the structures to the store and updates the nodes status.

This step doesn’t create any system resources, such as records, on the destination node. This is performed later in the data sync.

Store

Shared modules

Table 2. federation_module_shared
Column Type Description

ID

uint64

federation module id

handle

varchar(200)

Module handle

name

varchar(64)

Module name

rel_node

uint64

node id (source node id - who is sharing with us)

xref_module

uint64

federation module id on source node (id in federation_module_exposed)

fields

json

list of fields

Field mapping

Field mapping allows the destination node to determine what fields from the origin node should map into what fields on the destination node. This allows some flexibility when it comes down to datamodel definitions on both origin and destination nodes. This can be performed via the Corteza Low-Code administration panel, or directly via the API.

API

Set field mapping for a module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X PUT "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/mapped" \
  -H "Authorization: Bearer $JWT"
  -H "Content-Type: application/json" \
  --data "[{
      \"origin\":{
          \"name\":\"LinkedIn\",
          \"kind\":\"Url\",
          \"is_multi\":0
        },
        \"destination\":{
            \"name\":\"Social\",
            \"kind\":\"String\",
            \"is_multi\":0
        }
    }]";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "mapping": [
            {
                "origin": {
                    "name": "LinkedIn",
                    "kind": "Url",
                    "is_multi": 0
                },
                "destination": {
                    "name": "Social",
                    "kind": "String",
                    "is_multi": 0
                }
            }
        ]
    }
}

Get field mapping for a module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X GET "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/mapped" \
  -H "Authorization: Bearer $JWT";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "mapping": [
            {
                "origin": {
                    "name": "LinkedIn",
                    "kind": "Url",
                    "is_multi": 0
                },
                "destination": {
                    "name": "Social",
                    "kind": "String",
                    "is_multi": 0
                }
            }
        ]
    }
}

Remove field mapping for a module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X DELETE "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID" \
  -H "authorization: Bearer $JWT";
Example response

Store

Module mapping

Table 3. federation_module_mapping
Column Type Description

federation_module_id

uint64

id from federation_module_exposed

compose_module_id

uint64

existing module

field_mapping

json

json field mappings, ex: [{ source: 'node_A_module_field_7', dest: 'node_B_module_field_2', transform: 'string' }]

Example

Phase I - on Origin node

First phase is exposing the desired modules for a specific node (to the Destination), so the structure mapping on the Destination and then the data sync can be done.

compose_module

id handle name

161250629010849793

Account

Account

compose_module_field

id kind name label is_multi

161250629061509121

String

Phone

Phone

0

161250629027758081

Url

LinkedIn

LinkedIn

0

161250629044666369

String

Description

Description

0

federation_node

id name

1

Origin server

2

Destination server

federation_module_exposed

id rel_node rel_compose_module field_mapping

11

2

161250629010849793

[{"name":"Phone","kind":"String","is_multi":0}]

Phase II - on Destination node

There are 2 phases in the phase II. First the module info from Origin is saved. After that we can do the mapping. The modules on the Destination need to be created beforehand.

compose_module

id handle name

261250629010849755

Account_federated

Account (federated from Origin)

compose_module_field

id kind name label is_multi

161250629061509121

String

Mobile

Mobile

0

161250629044666369

String

Desc

Description

0

federation_node

id name

1

Our server

2

Misc server (some other server)

3

Origin server in this example (from phase I)

1. Fetch and save the module info

federation_module_shared

id handle name rel_node xref_module field_mapping

22

Account

Account

3

11

[{"name":"Phone","kind":"String","is_multi":0}]

2. Mapping finished, modules created

The sharing of modules info from Origin is added, that is enough information for us to handle mapping from UI. We can now pick the fields from the field_mapping column that we need and store it into the mapping table.

federation_module_mapping

federation_module_id compose_module_id fields

22

261250629010849755

[{"origin":{"name":"Phone","kind":"String","is_multi":0},"destination":{"name":"Mobile","kind":"String","is_multi":0}}]