You are reading the documentation for an outdated Corteza release. 2024.9 is the latest stable Corteza release.

Federation

An abstract overview over the entire federation system.
Figure 1. An abstract overview over the entire federation system.
Node pairing

The process of establishing a federated network between two nodes. After they are paired, they are able to securely exchange data. Refer to Pairing nodes and Authentication for details.

Structure syncing

The process of determining what structures on the origin node (the one that contains the data), is exposed to it’s destination node (the one that wishes to access the data). The origin node has full control over what data the destination node is allowed to access. This can be as simple as allowing access to only specific modules; and as complex as allowing access to only specific module fields. Refer to Syncing structures and Access control for details.

Data syncing

The process of updating data on the destination node, based on the updated data source. The destination node has the ability to determine what field from the origin node maps into what field in the destination node. Refer to Syncing data for details.

Glossary

Pairing

Before the two nodes are paired, we are unable to determine what node is the data source and what node is the destination. To avoid confusion, we use node A and node B when referring to the two nodes.
Node URI

The URI that identifies a federation node.

Node A

The node that wishes to include another node in it’s federated network — the node that initialized the pairing process.

Node B

The node that wishes to join the federated network.

NodeID A

Node ID of the federated node A.

NodeID B

Node ID of the federated node B.

Token A

The authentication token to access protected resources on node A.

Token B

The authentication token to access protected resources on node B.

Data exchange

Origin node

The node that shares the data to the other node. If node A requests data from node B, node B is origin, as it contains the data.

Destination node

The node that receives the data from the origin node. If node A requests data from node B, node A is destination as it receives the data.

Federated node

Node status

The current status of the federated node. Can be one of the following:

  • Pairing: the node is in the pairing process,

  • paired: the node has been paired successfully,

  • failed: the node failed to pair.

Structure sync status

The current status of the structure sync. Can be one of the following:

  • Syncing: the node is currently syncing structures from the origin node,

  • synced: the node has successfully finished syncing,

  • failed: there was an error while syncing.

Data sync status

The current status of the data sync. Can be one of the following:

  • Syncing: the node is currently syncing data from the origin node,

  • synced: the node has successfully finished syncing,

  • failed: there was an error while syncing.

Security

Authentication

Federated nodes leverage Corteza’s already established authentication facility that uses system user and JWT tokens (later referred as a token). During the pairing process between the two nodes, both nodes create a system user and securely exchange their authentication tokens.

  • Node A defines a system user and token A that allows node B to access protected resources,

  • node B defines a system user and token B that allows node A to access protected resources.

Whenever node A wishes to access protected resources on node B, it uses token B for authentication; vice-versa for node B accessing protected resources on node A.

System users and authentication tokens are unique for each node pair. When a new node is added to the federated network, a new pair of system users and tokens are generated.

Token A and token B are not the same.

Access control

The federation service uses Corteza’s already established RBAC access control facility that operates over user roles, system resources and operations over given resources.

Each node in a federated network has the ability to define what resources a paired node is allowed to access. This can be as simple as controlling access to records belonging to specific modules, or as detailed as controlling access to specific fields for specific modules.

When a destination node requests data from the origin node, the origin node:

  • Passes the request through the access control facility to check if we are allowed to access the resource,

  • collects the requested data, following the defined RBAC permissions, excluding data that the destination node is not allowed to access.

The federation service then performs some additional operations, see Syncing data, and sends it over to the destination node.

Logging

Corteza federation service keeps track of the important events (actions) that occurred between the two federated nodes. The logs are stored inside Corteza’s actionlog facility; it stores:

  • When the action ocurred,

  • the invoking user,

  • the accessed resource,

  • the performed operation,

  • the result of the operation, …​

Logged events

Node pairing

Pairing started

Logged when the nodes initiate in the pairing process.

Pairing failed

Logged when an error occurs during the pairing process.

Pairing finished

Logged when the nodes have been paired successfully.

Structure sync

Structure sync started

Logged when the nodes initiate in the structure sync process.

Structure sync failed

Logged when an error occurs during the structure sync process.

Structure sync finished

Logged when the nodes have successfully performed the structure sync process.

Data sync

Data sync started

Logged when the nodes initiate in the data sync process.

Data sync failed

Logged when an error occurs during the data sync process.

Data sync finished

Logged when the nodes have successfully performed the data sync process.

Pairing nodes

The process of establishing a federated network between two nodes with intent of securely sharing data. The process consists of two steps:

Node identification

The step identifies the two nodes so they know where to access the information.

Node handshake

The step exchanges the authentication tokens so the two nodes can access protected resources from each other.

Outline of the entire node pairing process.
Figure 2. Outline of the entire node pairing process.

Node identification

Outline of the node identification step of the node pairing process.
Figure 3. Outline of the node identification step of the node pairing process.

No authentication tokens are exchanged in during the identification step.
Node A administrator registers node B and generates it’s node URI

This step lets node A know about node B. The generated node URI identifies node A and is in the form of corteza://$NODE_ID_A:$OTT@$DOMAIN_A?name=$NAME.

$OTT is a generated OTT token that allows us to perform some authentication when performing the handshake.
Used variables
# Base URL of node A api
$API_A_BASE

# Main administrator JWT for node A
$MAIN_JWT_A

# Node A domain
$DOMAIN_A

# Node B domain
$DOMAIN_B

# Node name
$NODE_NAME

# Node A nodeID
$NODE_ID_A

# Node B nodeID
$NODE_ID_B

# Node URI
$NODE_URI
Example request
curl -X POST "$API_A_BASE/federation/nodes" \
  -H "authorization: Bearer $MAIN_JWT_A" \
  --header "Content-Type: application/json" \
  --data "{
    \"myDomain\": \"$DOMAIN_A\",
    \"domain\": \"$DOMAIN_B\",
    \"name\": \"$NODE_NAME\"
  }";
Example response
{
  "response": {
    "nodeID": "$NODE_ID_A",
    "sharedNodeID": "$NODE_ID_B",
    "name": "\"$NODE_NAME\"",
    "domain": "\"$DOMAIN_B\"",
    "status": "\"pending\"",
    "nodeURI": "\"$NODE_URI\""
  }
}
Node A administrator sends the node URI to the node B administrator

This step transports the node URI to the node that we wish to pair with.

This step is performed manually by the node A administrator. The two administrators should use a secure channel in order to exchange this information.
Node B administrator registers node A using the node URI

This step lets node B know about node A. Both nodes A and B are now identified and prepared to perform the Node handshake.

Used variables
# Base URL of node B api
$API_B_BASE

# Main administrator JWT for node B
$MAIN_JWT_B

# Node B domain
$DOMAIN_B

# Node A domain
$DOMAIN_A

# Node name
$NODE_NAME

# Node B nodeID
$NODE_ID_B

# Node A nodeID
$NODE_ID_A

# Node URI
$NODE_URI
Example request
curl -X POST "$API_B_BASE/federation/nodes" \
  -H "authorization: Bearer $MAIN_JWT_B" \
  --header "Content-Type: application/json" \
  --data "{
    \"myDomain\": \"$DOMAIN_B\",
    \"nodeURI\": \"$NODE_URI\"
  }";
Example response
{
  "response": {
    "nodeID": "$NODE_ID_B",
    "sharedNodeID": "$NODE_ID_A",
    "name": "\"$NODE_NAME\"",
    "domain": "\"$DOMAIN_A\"",
    "status": "\"pending\"",
    "nodeURI": "\"$NODE_URI\""
  }
}

In the above response, $NODE_URI contains the value provided by node A.

Node handshake

Outline of the node handshake step.
Figure 4. Outline of the node handshake step.
Node B administrator initializes the process with node A

This configures the state on node B and generates the $TOKEN_B; so node A will be able to access protected resources on node B.

Used variables
# Base URL of node B api
$API_B_BASE

# Main administrator JWT for node B
$MAIN_JWT_B

# Node B nodeID
$NODE_ID_B
Example request
curl -X POST "$API_B_BASE/federation/nodes/$NODE_ID_B/pair" \
  -H "authorization: Bearer $MAIN_JWT_B" \
  --header "Content-Type: application/json";
Example response
{}
Node B sends a handshake request to node A

This notifies the node A administrator that node B wishes to establish a federated network. This request must be manually confirmed by the node administrator.

This request is authenticated by the before generated $OTT token, outside of the standard authentication facility.
Used variables
# Base URL of node A api
$API_A_BASE

# Node A nodeID
$NODE_ID_A

# Node URI
$NODE_URI

# Node B auth token
$TOKEN_B

# Node B nodeID
$NODE_ID_B
Example request
curl -X POST "$API_A_BASE/federation/nodes/$NODE_ID_A/handshake" \
  --header "Content-Type: application/json" \
  --data "{
    \"nodeURI\": \"$NODE_URI\",
    \"token\": \"$TOKEN_B\",
    \"nodeIDB\": \"$NODE_ID_B\"
  }";
Example response
{}
Node A administrator confirms the handshake request

This configures the state on node A and generates the $TOKEN_A; so node B will be able to access protected resources on node A.

Used variables
# Base URL of node A api
$API_A_BASE

# Node A nodeID
$NODE_ID_A

# Main administrator JWT for node A
$MAIN_JWT_A
Example request
curl -X POST "$API_A_BASE/federation/nodes/$NODE_ID_A/handshake-confirm" \
  -H "authorization: Bearer $MAIN_JWT_A" \
  --header "Content-Type: application/json";
Example response
{}
Node A completes the handshake with node B

This sends the $TOKEN_A to node B so it will be able to access protected resources on node A.

Notice how node A uses $TOKEN_B to authenticate this request.
Used variables
# Base URL of node B api
$API_B_BASE

# Node B nodeID
$NODE_ID_B

# Node B auth token
$TOKEN_B

# Node A auth token
$TOKEN_A
Example request
curl -X POST "$API_B_BASE/federation/nodes/$NODE_ID_B/handshake-complete" \
  -H "authorization: Bearer $TOKEN_B" \
  --header "Content-Type: application/json" \
  --data "{
    \"token\": \"$TOKEN_A\"
  }";
Example response
{}

Syncing structures

The process of determining what structures on the origin node, is exposed to it’s destination node. The origin node has full control over what data the destination node is allowed to access. This can be as simple as allowing access to only specific modules; and as complex as allowing access to only specific module fields.

The two nodes must be paired prior to this. See Node pairing.

Exposing structures

Firstly, in order to perform any data sharing, the origin node must define what structures (in our case modules and fields) the destination node can access. This can be performed via the Corteza Low-Code administration panel, or directly via the API.

Besides the module itself, the origin node must also specify what fields the destination node is allowed to access.

API

Expose module
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X PUT "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID" \
  -H "Authorization: Bearer $JWT"
  -H "Content-Type: application/json" \
  --data "[{
      \"name\":\"LinkedIn\",
      \"label\":\"LinkedIn Url\",
      \"kind\":\"Url\",
      \"is_multi\":0
    }]";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "name": "LinkedIn",
                "label": "LinkedIn Url",
                "kind": "Url",
                "is_multi": 0
            }
        ]
    }
}
See exposed module
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID
Example request
curl -X GET "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/exposed" \
  -H "Authorization: Bearer $JWT";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": false,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": false,
            }
        ]
    }
}
Remove exposed module
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X DELETE "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID" \
  -H "authorization: Bearer $JWT";
Example response
Change exposed module fields
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X PUT "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/exposed" \
  -H "Authorization: Bearer $JWT"
  -H "Content-Type: application/json" \
  --data "[{
      \"name\":\"LinkedIn\",
      \"label\":\"LinkedIn\",
      \"kind\":\"Url\",
      \"is_multi\":1
    }, {
      \"name\":\"Phone\",
      \"label\":\"Phone\",
      \"kind\":\"String\",
      \"is_multi\":0
    }]";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": 0,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": 0,
            }
        ]
    }
}

Store

Exposed modules
Table 1. federation_module_exposed
Column Type Description

ID

uint64

federation module id

rel_node

uint64

node id (destination node id - who are we sharing to)

rel_compose_module

uint64

module id on source node

fields

json

list of fields

Syncing structures

When the two nodes are paired and the origin node has exposed some structures, the actual structure sync can occur.

Outline of structure sync process on the origin node.
Figure 5. Outline of structure sync process on the origin node.
Outline of structure sync process on the destination node.
Figure 6. Outline of structure sync process on the destination node.
Destination node requests changed structures

Origin node provides a set of endpoints that the destination node can use to fetch shared structures.

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/federation/exposed/modules" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "filter": {
            "query": "after=1600109447",
            "page": 1,
            "perPage": 20,
            "count": 97,
        },
        "set": [
            {
                "type": "GET",
                "rel": "Account",
                "href": "$BASE_URL/federation/exposed/modules/$MODULE_ID?after=1600109447"
            },
            {
                "type": "GET",
                "rel": "Contact",
                "href": "$BASE_URL/federation/exposed/modules/$MODULE_ID?after=1600109447"
            }
        ]
    }
}
Destination node fetches updated structures

The destination node requests structure definitions based on the above provided set of endpoints. The origin node provides the structure definition with respect to the fields that the destination node is allowed to access.

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the origin node
$NODE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/federation/exposed/modules/$MODULE_ID" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": false,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": false,
            }
        ]
    }
}
Destination node updates its store

Once the structure sync is finished, the destination node writes the structures to the store and updates the nodes status.

This step doesn’t create any system resources, such as records, on the destination node. This is performed later in the data sync.

Store

Shared modules
Table 2. federation_module_shared
Column Type Description

ID

uint64

federation module id

handle

varchar(200)

Module handle

name

varchar(64)

Module name

rel_node

uint64

node id (source node id - who is sharing with us)

xref_module

uint64

federation module id on source node (id in federation_module_exposed)

fields

json

list of fields

Field mapping

Field mapping allows the destination node to determine what fields from the origin node should map into what fields on the destination node. This allows some flexibility when it comes down to datamodel definitions on both origin and destination nodes. This can be performed via the Corteza Low-Code administration panel, or directly via the API.

API

Set field mapping for a module
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X PUT "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/mapped" \
  -H "Authorization: Bearer $JWT"
  -H "Content-Type: application/json" \
  --data "[{
      \"origin\":{
          \"name\":\"LinkedIn\",
          \"kind\":\"Url\",
          \"is_multi\":0
        },
        \"destination\":{
            \"name\":\"Social\",
            \"kind\":\"String\",
            \"is_multi\":0
        }
    }]";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "mapping": [
            {
                "origin": {
                    "name": "LinkedIn",
                    "kind": "Url",
                    "is_multi": 0
                },
                "destination": {
                    "name": "Social",
                    "kind": "String",
                    "is_multi": 0
                }
            }
        ]
    }
}
Get field mapping for a module
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X GET "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/mapped" \
  -H "Authorization: Bearer $JWT";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "mapping": [
            {
                "origin": {
                    "name": "LinkedIn",
                    "kind": "Url",
                    "is_multi": 0
                },
                "destination": {
                    "name": "Social",
                    "kind": "String",
                    "is_multi": 0
                }
            }
        ]
    }
}
Remove field mapping for a module
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X DELETE "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID" \
  -H "authorization: Bearer $JWT";
Example response

Store

Module mapping
Table 3. federation_module_mapping
Column Type Description

federation_module_id

uint64

id from federation_module_exposed

compose_module_id

uint64

existing module

field_mapping

json

json field mappings, ex: [{ source: 'node_A_module_field_7', dest: 'node_B_module_field_2', transform: 'string' }]

Example

Phase I - on Origin node

First phase is exposing the desired modules for a specific node (to the Destination), so the structure mapping on the Destination and then the data sync can be done.

compose_module

id handle name

161250629010849793

Account

Account

compose_module_field

id kind name label is_multi

161250629061509121

String

Phone

Phone

0

161250629027758081

Url

LinkedIn

LinkedIn

0

161250629044666369

String

Description

Description

0

federation_node

id name

1

Origin server

2

Destination server

federation_module_exposed

id rel_node rel_compose_module field_mapping

11

2

161250629010849793

[{"name":"Phone","kind":"String","is_multi":0}]

Phase II - on Destination node

There are 2 phases in the phase II. First the module info from Origin is saved. After that we can do the mapping. The modules on the Destination need to be created beforehand.

compose_module

id handle name

261250629010849755

Account_federated

Account (federated from Origin)

compose_module_field

id kind name label is_multi

161250629061509121

String

Mobile

Mobile

0

161250629044666369

String

Desc

Description

0

federation_node

id name

1

Our server

2

Misc server (some other server)

3

Origin server in this example (from phase I)
1. Fetch and save the module info

federation_module_shared

id handle name rel_node xref_module field_mapping

22

Account

Account

3

11

[{"name":"Phone","kind":"String","is_multi":0}]
2. Mapping finished, modules created

The sharing of modules info from Origin is added, that is enough information for us to handle mapping from UI. We can now pick the fields from the field_mapping column that we need and store it into the mapping table.

federation_module_mapping

federation_module_id compose_module_id fields

22

261250629010849755

[{"origin":{"name":"Phone","kind":"String","is_multi":0},"destination":{"name":"Mobile","kind":"String","is_multi":0}}]

Syncing data

The process of updating data on the destination node, based on the updated data source. The destination node has the ability to determine what field from the origin node maps into what field in the destination node.

Data sync operates directly on compose services and storage layers, but is designed to be decoupled and moved away from the main corteza service.

The two nodes must be paired prior to this. See Pairing nodes.

Syncing data

After the two nodes are paired and the structure syncing has finished, we can proceed with the data sync.

Data syncing uses already established compose services along with it’s storage layer. This removes the need for any additional storage layer modifications.
Outline of the data sync process on the destination node.
Figure 7. Outline of the data sync process on the destination node.
Outline of the data sync process on the origin node.
Figure 8. Outline of the data sync process on the origin node.
Destination node requests the information about any data changes

Origin node provides a set of endpoints that the destination node can use to fetch newly created, updated, and deleted data. The destination node provides some additional filtering parameters; such as last sync timestamp; to exclude any unchanged data.

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/federation/exposed/records?after=$AFTER_TIMESTAMP" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "filter": {
            "query": "after=1600109447",
            "page": 1,
            "perPage": 20,
            "count": 97,
            "deleted": 0
        },
        "set": [
            {
                "type": "GET",
                "rel": "Lead",
                "href": "$BASE_URL/federation/exposed/records/$MODULE_ID?after=1600109447"
            },
            {
                "type": "GET",
                "rel": "Contact",
                "href": "$BASE_URL/federation/exposed/records/$MODULE_ID?after=1600109447"
            }
        ]
    }
}
Destination node requests changed data

The destination node fetches the data on per-module basis from the above provided set of endpoints.

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/exposed/modules/$MODULE_ID/records?after=$AFTER_TIMESTAMP" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "filter": {
            "moduleID": "132954639472525355",
            "query": "",
            "sort": "createdAt DESC",
            "page": 1,
            "perPage": 20,
            "count": 97,
            "deleted": 0
        },
        "set": [
            {
                "recordID": "161135411379307175",
                "moduleID": "132954639472525355",
                "values": [
                    {
                        "name": "name",
                        "value": "John"
                    },
                    {
                        "name": "surname",
                        "value": "Doe"
                    }
                ],
                "createdAt": "2020-09-08T19:56:14Z",
                "updatedAt": "2020-09-09T18:05:33Z"
            },
            {
                "recordID": "161134990657061543",
                "moduleID": "132954639472525355",
                "values": [
                    {
                        "name": "name",
                        "value": "Walter"
                    },
                    {
                        "name": "surname",
                        "value": "White"
                    }
                ],
                "createdAt": "2020-09-08T19:52:03Z",
                "updatedAt": "2020-09-11T19:44:03Z"
            }
        ]
    }
}
Origin node provides changed data for the requested structure

The origin node provides a set of changes based on base filter parameters such as timestamp, and requested structure to determine what data the destination node would like to receive. The filtered data, along with federated structure definitions are then passed into internal data manipulation systems to transform the data into the desired output, such as Activity Pub, and JSON (this also removes any fields that are not exposed by the origin node). The response also includes any additional filtering and pagination related parameters so the data can be fetched in chunks, or re-fetched if it any issues occurred.

Destination node transforms the provided data into internal resource structures

The destination node uses data manipulation systems, along with module mapping definitions (see Field mapping) to transform the provided data set into internal resource structures. These are then used to update the destination node’s storage.

Destination node updates the nodes status

The node’s status is updated to indicate when the last successful data sync has occurred.

API

Node pair

Create a federated node from a series of parameters

Used variables
# Base URL of node A api
$API_A_BASE

# Main administrator JWT for node A
$MAIN_JWT_A

# Node A domain
$DOMAIN_A

# Node B domain
$DOMAIN_B

# Node name
$NODE_NAME

# Node A nodeID
$NODE_ID_A

# Node B nodeID
$NODE_ID_B

# Node URI
$NODE_URI
Example request
curl -X POST "$API_A_BASE/federation/nodes" \
  -H "authorization: Bearer $MAIN_JWT_A" \
  --header "Content-Type: application/json" \
  --data "{
    \"myDomain\": \"$DOMAIN_A\",
    \"domain\": \"$DOMAIN_B\",
    \"name\": \"$NODE_NAME\"
  }";
Example response
{
  "response": {
    "nodeID": "$NODE_ID_A",
    "sharedNodeID": "$NODE_ID_B",
    "name": "\"$NODE_NAME\"",
    "domain": "\"$DOMAIN_B\"",
    "status": "\"pending\"",
    "nodeURI": "\"$NODE_URI\""
  }
}

Create a federated node from a node URI

Used variables
# Base URL of node B api
$API_B_BASE

# Main administrator JWT for node B
$MAIN_JWT_B

# Node B domain
$DOMAIN_B

# Node A domain
$DOMAIN_A

# Node name
$NODE_NAME

# Node B nodeID
$NODE_ID_B

# Node A nodeID
$NODE_ID_A

# Node URI
$NODE_URI
Example request
curl -X POST "$API_B_BASE/federation/nodes" \
  -H "authorization: Bearer $MAIN_JWT_B" \
  --header "Content-Type: application/json" \
  --data "{
    \"myDomain\": \"$DOMAIN_B\",
    \"nodeURI\": \"$NODE_URI\"
  }";
Example response
{
  "response": {
    "nodeID": "$NODE_ID_B",
    "sharedNodeID": "$NODE_ID_A",
    "name": "\"$NODE_NAME\"",
    "domain": "\"$DOMAIN_A\"",
    "status": "\"pending\"",
    "nodeURI": "\"$NODE_URI\""
  }
}

Initialize the handshake

Used variables
# Base URL of node B api
$API_B_BASE

# Main administrator JWT for node B
$MAIN_JWT_B

# Node B nodeID
$NODE_ID_B
Example request
curl -X POST "$API_B_BASE/federation/nodes/$NODE_ID_B/pair" \
  -H "authorization: Bearer $MAIN_JWT_B" \
  --header "Content-Type: application/json";
Example response
{}

Request the handshake with node A

Used variables
# Base URL of node A api
$API_A_BASE

# Node A nodeID
$NODE_ID_A

# Node URI
$NODE_URI

# Node B auth token
$TOKEN_B

# Node B nodeID
$NODE_ID_B
Example request
curl -X POST "$API_A_BASE/federation/nodes/$NODE_ID_A/handshake" \
  --header "Content-Type: application/json" \
  --data "{
    \"nodeURI\": \"$NODE_URI\",
    \"token\": \"$TOKEN_B\",
    \"nodeIDB\": \"$NODE_ID_B\"
  }";
Example response
{}

Confirm the requested handshake

Used variables
# Base URL of node A api
$API_A_BASE

# Node A nodeID
$NODE_ID_A

# Main administrator JWT for node A
$MAIN_JWT_A
Example request
curl -X POST "$API_A_BASE/federation/nodes/$NODE_ID_A/handshake-confirm" \
  -H "authorization: Bearer $MAIN_JWT_A" \
  --header "Content-Type: application/json";
Example response
{}

Complete the handshake

Used variables
# Base URL of node B api
$API_B_BASE

# Node B nodeID
$NODE_ID_B

# Node B auth token
$TOKEN_B

# Node A auth token
$TOKEN_A
Example request
curl -X POST "$API_B_BASE/federation/nodes/$NODE_ID_B/handshake-complete" \
  -H "authorization: Bearer $TOKEN_B" \
  --header "Content-Type: application/json" \
  --data "{
    \"token\": \"$TOKEN_A\"
  }";
Example response
{}

Origin structures

Add module to federation

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X PUT "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID" \
  -H "Authorization: Bearer $JWT"
  -H "Content-Type: application/json" \
  --data "[{
      \"name\":\"LinkedIn\",
      \"label\":\"LinkedIn Url\",
      \"kind\":\"Url\",
      \"is_multi\":0
    }]";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "name": "LinkedIn",
                "label": "LinkedIn Url",
                "kind": "Url",
                "is_multi": 0
            }
        ]
    }
}

Change sharing fields

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X PUT "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/exposed" \
  -H "Authorization: Bearer $JWT"
  -H "Content-Type: application/json" \
  --data "[{
      \"name\":\"LinkedIn\",
      \"label\":\"LinkedIn\",
      \"kind\":\"Url\",
      \"is_multi\":1
    }, {
      \"name\":\"Phone\",
      \"label\":\"Phone\",
      \"kind\":\"String\",
      \"is_multi\":0
    }]";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": 0,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": 0,
            }
        ]
    }
}

Info about exposed federated module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID
Example request
curl -X GET "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/exposed" \
  -H "Authorization: Bearer $JWT";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": false,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": false,
            }
        ]
    }
}

Remove module from federation

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X DELETE "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID" \
  -H "authorization: Bearer $JWT";
Example response

Destination structures

Info about shared federated module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the origin node
$NODE_ID
Example request
curl -X GET "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/shared" \
  -H "Authorization: Bearer $JWT";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": false,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": false,
            }
        ]
    }
}

List shared modules

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID
Example request
curl -X GET "$BASE_URL/federation/nodes/$NODE_ID/modules" \
  -H "Authorization: Bearer $JWT";
Example response
{
    "response": {
        "filter": {
            "query": "",
            "handle": "",
            "name": "",
            "sort": "name ASC",
            "count": 1
        },
        "set": [
            {
                "moduleID": "122709113267335170",
                "handle": "Account",
                "name": "Account",
                "createdAt": "2019-12-18T17:45:15Z",
                "updatedAt": "2020-05-26T13:29:36Z",
                "fields": [
                    {
                        "kind": "Url",
                        "name": "LinkedIn",
                        "label": "LinkedIn",
                        "isMulti": false,
                    },
                    {
                        "kind": "String",
                        "name": "Phone",
                        "label": "Phone",
                        "isMulti": false,
                    }
                ]
            }
        ]
    }
}

Set module mapping for a module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X PUT "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/mapped" \
  -H "Authorization: Bearer $JWT"
  -H "Content-Type: application/json" \
  --data "[{
      \"origin\":{
          \"name\":\"LinkedIn\",
          \"kind\":\"Url\",
          \"is_multi\":0
        },
        \"destination\":{
            \"name\":\"Social\",
            \"kind\":\"String\",
            \"is_multi\":0
        }
    }]";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "mapping": [
            {
                "origin": {
                    "name": "LinkedIn",
                    "kind": "Url",
                    "is_multi": 0
                },
                "destination": {
                    "name": "Social",
                    "kind": "String",
                    "is_multi": 0
                }
            }
        ]
    }
}

Get module mapping for a module

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X GET "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID/mapped" \
  -H "Authorization: Bearer $JWT";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "mapping": [
            {
                "origin": {
                    "name": "LinkedIn",
                    "kind": "Url",
                    "is_multi": 0
                },
                "destination": {
                    "name": "Social",
                    "kind": "String",
                    "is_multi": 0
                }
            }
        ]
    }
}

Remove module mapping from federation

Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID
Example request
curl -X DELETE "$BASE_URL/federation/nodes/$NODE_ID/modules/$MODULE_ID" \
  -H "authorization: Bearer $JWT";
Example response

Structure sync

Get master changes

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node
$NODE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/federation/exposed/modules" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "filter": {
            "query": "after=1600109447",
            "page": 1,
            "perPage": 20,
            "count": 97,
        },
        "set": [
            {
                "type": "GET",
                "rel": "Account",
                "href": "$BASE_URL/federation/exposed/modules/$MODULE_ID?after=1600109447"
            },
            {
                "type": "GET",
                "rel": "Contact",
                "href": "$BASE_URL/federation/exposed/modules/$MODULE_ID?after=1600109447"
            }
        ]
    }
}

Sync shared module structure

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the origin node
$NODE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/federation/exposed/modules/$MODULE_ID" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "moduleID": "122709113267335170",
        "handle": "Account",
        "name": "Account",
        "createdAt": "2019-12-18T17:45:15Z",
        "updatedAt": "2020-05-26T13:29:36Z",
        "fields": [
            {
                "kind": "Url",
                "name": "LinkedIn",
                "label": "LinkedIn",
                "isMulti": false,
            },
            {
                "kind": "String",
                "name": "Phone",
                "label": "Phone",
                "isMulti": false,
            }
        ]
    }
}

Fetch exposed data changes

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/federation/exposed/records?after=$AFTER_TIMESTAMP" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "filter": {
            "query": "after=1600109447",
            "page": 1,
            "perPage": 20,
            "count": 97,
            "deleted": 0
        },
        "set": [
            {
                "type": "GET",
                "rel": "Lead",
                "href": "$BASE_URL/federation/exposed/records/$MODULE_ID?after=1600109447"
            },
            {
                "type": "GET",
                "rel": "Contact",
                "href": "$BASE_URL/federation/exposed/records/$MODULE_ID?after=1600109447"
            }
        ]
    }
}

Fetch exposed data

$TOKEN_B is the token that was generated during the handshake and is used to authenticate the user on the Origin node (the one who shares the data) by the Destination node.
Used variables
# Base url for the federation api
$BASE_URL

# JWT of the user
$JWT

# Node id of the destination node (?exposed) or the origin node (?shared)
$NODE_ID

# Federation module id
$MODULE_ID

# Node B auth token
$TOKEN_B
Example request
curl -X GET "$BASE_URL/exposed/modules/$MODULE_ID/records?after=$AFTER_TIMESTAMP" \
  -H "Authorization: Bearer $TOKEN_B";
Example response
{
    "response": {
        "filter": {
            "moduleID": "132954639472525355",
            "query": "",
            "sort": "createdAt DESC",
            "page": 1,
            "perPage": 20,
            "count": 97,
            "deleted": 0
        },
        "set": [
            {
                "recordID": "161135411379307175",
                "moduleID": "132954639472525355",
                "values": [
                    {
                        "name": "name",
                        "value": "John"
                    },
                    {
                        "name": "surname",
                        "value": "Doe"
                    }
                ],
                "createdAt": "2020-09-08T19:56:14Z",
                "updatedAt": "2020-09-09T18:05:33Z"
            },
            {
                "recordID": "161134990657061543",
                "moduleID": "132954639472525355",
                "values": [
                    {
                        "name": "name",
                        "value": "Walter"
                    },
                    {
                        "name": "surname",
                        "value": "White"
                    }
                ],
                "createdAt": "2020-09-08T19:52:03Z",
                "updatedAt": "2020-09-11T19:44:03Z"
            }
        ]
    }
}

Store

Federation node

Table 4. federation_node
Column Type Description

ID

uint64

Node ID

status

string

Node status

structure_status

string

Structure sync status

structure_synced_at

Timestamp

Last structure sync

data_status

string

Data sync status

data_synced_at

Timestamp

Last data sync

Exposed federation module

Table 5. federation_module_exposed
Column Type Description

ID

uint64

federation module id

rel_node

uint64

node id (destination node id - who are we sharing to)

rel_compose_module

uint64

module id on source node

fields

json

list of fields

Shared federation module

Table 6. federation_module_shared
Column Type Description

ID

uint64

federation module id

handle

varchar(200)

Module handle

name

varchar(64)

Module name

rel_node

uint64

node id (source node id - who is sharing with us)

xref_module

uint64

federation module id on source node (id in federation_module_exposed)

fields

json

list of fields

Federation module mapping

Table 7. federation_module_mapping
Column Type Description

federation_module_id

uint64

id from federation_module_exposed

compose_module_id

uint64

existing module

field_mapping

json

json field mappings, ex: [{ source: 'node_A_module_field_7', dest: 'node_B_module_field_2', transform: 'string' }]