Extensions

This section covers the system behind extensions. If you’re interested in extension development, refer to the integrator guide.

Overview

Components

Corteza Corredor Server

A Node.js service written in TypeScript. It is the heart of the system as it parses and serves automation scripts. It also executes server-scripts.

Corteza Server

The Corteza server is responsible for automation script execution. On its own, Corredor server is unable to do any code execution; it must be invoked either via Corteza server or manually via gRCP protocol.

Corredor Client

Each Corteza webapp that allows client-script execution implements this. The Corredor Client is responsible for client-script registration and their execution. Automation scripts are received from the Corteza Server.

System diagram

Automation system

Corredor Server

Corredor server is mainly responsible for:

  • Loading, processing and serving automation scripts,

  • server-script execution.

Server-scripts file structure

/server-scripts (1)
    /... (2)
1 Root folder for all server scripts (under each search path).
2 Undefined file structure; can be defined as needed.

Client-scripts file structure

/client-scripts (1)
    /auth (2)
        /... (7)

    /admin (3)
        /... (7)

    /compose (4)
        /... (7)

    /messaging (5)
        /... (7)

    /shared (6)
        /... (7)
1 Root folder for all client scripts (under each search path).
2 Defines a bundle for Corteza Auth.
3 Defines a bundle for Corteza Admin.
4 Defines a bundle for Corteza Low Code.
5 Defines a bundle for Corteza Messaging.
6 Reserved directory for any shared logic, such as custom libraries, assets, …​
7 Undefined file structure; can be defined as needed.

Automation script processing

Flow outline:

Scan extensions defined in the .env

Go over all extensions defined in the CORREDOR_EXT_SEARCH_PATHS .env variable. Extensions are separated by :.

Note that all of the following steps are performed for each of the extensions.

Dependency management

Load all requested dependencies by the given extension.

Dependencies are scoped to the extension that has requested them. If two extensions wish to use the same dependency (axios for example) they both should define it.

Parse server-scripts

Go over each script file defined inside the server-scripts sub-tree. Parse the script and extract all the important bits such as dependencies, triggers, iterators and the exec function itself.

At the moment server scripts are not bundled.

Parse client-scripts

Go over each script file defined inside the client-scripts sub-tree. Parse the script and extract all the important bits such as dependencies, triggers, iterators and the exec function itself. Create a per-app bundle containing relevant automation scripts, dependencies and other relevant bits. Refer to Client-script bundling for details about creating bundles.

Bundling is done using webpack.

Prepare a list of available scripts

This allows us to inspect what scripts were loaded from inside web-apps. It also simplifies the process of determining available automation scripts from inside client-script bundles.

Failing automation scripts are also included in this list but provide a descriptive error. This allows easier debugging.

Automation script naming

Corredor determines automation scripts name on-the-fly from the extension and it’s relative path. This simplifies the script’s definition as we don’t need to worry about unique names and possible conflicts between extensions. Script name is defined as:

/${path to script}/${file name}:${export name}(1)(2)(3)
1 Path to the script file, excluding the search path.
2 The script’s file name.
3 Used export name; this will normally be default.

The base search path of the extension is excluded from the script’s name. This allows easier reusability across different systems that don’t define the same file hierarchy.

Client-script bundling

Client-scripts are bundled using webpack. Bundles help us assure consistent code execution between different user-agents and allow the use of external node dependencies.

The bundling process creates multiple bundles, one for each available application (Auth, Admin, Low Code, …​). This simplifies the process of registering client-scripts on the client-side.

Bundle name is determined from the extensions file-structure:

<search-path>/client-scripts/<bundle>/<path-to-script>/*.js

The bundling process creates a boot loader file for each available application. The file consists of { name, triggers, security } JSON objects for each available automation script. The Webpack is then executed over each boot loader file to create a bundle for the given application. If the application doesn’t define any automation scripts the bundling process is omitted.

gRPC server

Corredor server allows communication via the gRPC server via the gRPC protocol via the following services:

  • Server scripts with list and exec procedures,

  • client scripts with list and bundle procedures.

Refer to protobuf service definition for the details on the definitions.

You can use BloomRPC or similar when working solely on the Corredor server to remove the need for the Corteza server.

Server-script execution

Corredor server is and should not be able to perform any code execution on its own. Any execution is and should be invoked either by the Corteza server or manually via gRPC protocol.

Watching for changes

When the Corredor server is running in development mode or is configured so via the .env variables it uses file watchers to detect changes to the extensions and dependency definition files. When a change occurs, the server wil restart the entire Automation script processing flow.

Corteza Server

Corteza server is mainly responsible for:

  • Providing endpoints for fetching scripts and script execution,

  • scheduling deferred automation scripts,

  • requesting script execution on the Corredor server.

The following automation scripts are executed directly by the Corteza server:

  • Implicit server-scripts,

  • explicit server-scripts,

  • deferred scripts,

  • sink scripts, and

  • iterator scripts.

Security context

Script endpoints

[APPLICATION NAME]/automation/list

Provides a list of available automation scripts for the given Corteza application.

[APPLICATION NAME]/automation/{bundle}-{type}.{ext}

Provides the client-script automation bundle for the given Corteza application.

If the given application doesn’t have it’s own bundle, the endpoint returns a 404 not found.

[APPLICATION NAME]/automation/trigger

Requests explicit server-script execution on the Corredor server.

system/sink

Provides an endpoint that server-scripts are able to listen on and perform operations for.

Event bus

Any executable automation script is registered on the event bus. Whenever any part of the system might require a script execution, it dispatches an event on the event bus and any registered script that conforms to the requirements is executed on the Corredor server.

The script registration flow is as follows:

Fetch available scripts from the Corredor server

This provides a full automation script list, including invalid scripts (those that didn’t compile successfully) — these are excluded.

Prepare a lightweight script representation

Define a lightweight handler struct for easier execution determination and script identification. Execution is performed by the handler function that allows a bit more flexibility on how to handle each script type.

Note that the iterators handler function is a bit more complex than the rest.

Register the script on the event bus

Register the above simplified automation script on the event buss.

Now when it comes to script execution, this is invoked by dispatching an event on the event bus. To outline the flow:

Dispatch the event on the event bus

An event is a generic interface that allows the event bus to determine what scripts should be executed.

Execute the scripts

Iterate and execute any automation script that conforms to the dispatched event. This can be done either synchronously or asynchronously.

Server side script execution

If the result of an automation script is an Aborted error, the current script should not effect the system’s state, but it also should not terminate any further execution.

Explicit

Explicit scripts are invoked by the front-end applications or manually via the API. When invoked, an event is constructed and dispatched via the event bus.

This is also true for sink script execution.

Implicit

These are triggered as a side-effect of another operation, such as record creation, user registration, password change, …​ Each part of the system that performs some operation that may invoke script execution generates an event and dispatches it on the event bus.

The same is also true for the Scheduler.

Iterator

Iterator script execution is invoked like any other deferred script. The important difference is that a new instance is executed for every resource that matches the trigger’s constraints.

If 10 records match the provided filter, the script will be executed 10 times.

System sink

System sink allows the implementation of custom endpoints on the Corteza server. This can be used for things like webhooks.

This works like so:
  1. intercept any HTTP request on the /sink endpoint,

  2. basic request validation (provided required parameters),

  3. validate request against the sink’s signature,

  4. process request based on:

    • Content-Type or any other HTTP header,

    • remote address (IP),

    • request method and path,

    • username and password (HTTP basic auth),

    • GET (query string) parameters,

    • POST parameters,

  5. generate an event and dispatch it via the event bus.

Scheduler

The scheduler system is responsible for triggering deferred scripts. At it’s core, scheduler is a "simple" ticker that dispatches periodical event on the event bus.

The interval is "hard-coded" to 1minute but that can be made configurable in the future.

There is no mechanism in place that would prevent the automation scripts to overlap.

For example:
  • Script A runs every minute and the execution takes 1.5min,

  • on first tick, script A is executed,

  • on second tick, script A is executed again along side the old instance.

This could be improved in the future.

Corredor Client

Corredor client is mainly responsible for:

  • Script registration on the event bus,

  • registering available UI hooks,

  • script execution; either locally or on the server.

Security context

Event bus

Any executable automation script is registered on the event bus. Whenever any part of the system might require a script execution, it dispatches an event on the event bus and any registered script that conforms to the requirements is executed on the Corredor server.

The event bus concept follows the same pattern as the Corteza server does.

Fetch available script bundle from the Corteza server

This provides available client-script automation scripts defined for the given application. This allows us to register available client-scripts.

Fetch available scripts from the Corteza server

This provides a full automation script list, including invalid scripts (those that didn’t compile successfully) — these are excluded. This allows us to register explicit server-scripts.

Prepare a lightweight script representation

Define a lightweight Handler object for easier execution determination and script identification. Execution is performed by the handler function that allows a bit more flexibility on how to handle each script type.

Register the script on the event bus

Register the above simplified automation script on the event buss.

Now when it comes to script execution, this is invoked by dispatching an event on the event bus. To outline the flow:

Dispatch the event on the event bus

An event is an object that allows the event bus to determine what automation scripts should be executed and their execution context/arguments.

Execute the scripts

Iterate and execute any automation script that conforms to the dispatched event. This can be done either synchronously or asynchronously.

A big difference between server-side and client-side execution is that client-side arguments are provided as references and not as copied objects.

Any change done to the original object is reflected to the original object.

UI hooks

A UI hook defines a component that represents an explicit automation script — either a client-script or a server-script. UI hooks mainly define the component properties — what element to render, styling, what script should be executed, additional context, …​

It is up to the web-app to render these components and dispatch events on the event bus when needed.

Script execution

Corredor client is able to interact with some UI elements such as window.alert, system notifications, …​

Explicit

Explicit scripts are invoked by the user by explicitly invoking the above mentioned UI hooks. The web-app creates an event and dispatches it on the event bus. The handler function requests the script’s execution on the Corteza server.

An event caused by an explicit script will provide the script name that should be executed. Implicit events do not.

Implicit

These are triggered as a side-effect of another operation, such as form submission, view render, validation error, …​ Each part of the system that performs some operation that may invoke script execution generates an event and dispatches it on the event bus.