Automation System Facility

The goal of this section is to provide a detailed, low-level overview of the automation system. Detailed understanding is crucial for maintaining existing features and development of additional features. It’s also a plus when performing complex debugging.

If you are interested solely in automation script development, feel free to skip to the [ext-dev] section. Any important bit from this section will be included in the following sections.

Corredor

Corredor is the main component in the entire automation system. It’s a Node.js system written in TypeScript responsible for parsing, serving and executing automation scripts.

Execution can also be performed inside the client web application (see Web Application).

Corredor is unable to perform any execution without an explicit invocation from the Corteza server. Corredor and Corteza servers are connected and communicate via the gRPC protocol with the following services (see protobuf service definition for details):

server scripts with list and exec procedures,
client scripts with list and bundle procedures.

Main responsibilities

Scanning configured search paths for scripts and dependency definition files,
loading, inspecting found scripts; parsing triggers, recording compile errors and preparing a final list of automation scripts,
running dependency management to load extension’s dependencies,
running file change watcher to rebuild the automation scripts and reload dependencies,
bundling all front-end scripts with WebPack,
running gRPC server and exposing the above listed services for script listing and execution.

Invalid automation scripts (invalid or missing trigger, syntax/logic errors, …) are included in the full list including all discovered errors.

When the Corredor server is ran in production mode, it does not perform any file watching and reloading. This can be changed by either running it in development mode or setting environment variables. Refer to the readme for details.

Script Parsing

Script parsing is an important bit of the system and it introduces a few gotchas that you should be aware of. The flow for automation script processing is as follows:

determine available source files and present them in a light-weight File interface (referred to as file) (see Determining script sources),
for each file:
1. determine script name (see Determining script name),
2. convert the source to an AST tree,
3. find and parse the default exported object. If it’s not found or invalid, the parsing for this file is aborted with a descriptive error,
4. parse the exported object into the Script object:
  - determine meta data (label, description); if provided,
  - evaluate security context; if provided,
  - evaluate triggers; if provided,
  - evaluate iterators; if provided,
  - determine exec function.

Triggers and iterators can not be used in the same script. If both are provided, the iterator is given priority.

The core of the automation script is evaluated outside of the source file, meaning any imports and other symbols defined outside of the exported object are ignored. This prevents the use of imports, constants, functions and other bits defined outside of the exported object inside the script’s meta definition (trigger, iterator, label, …). They can however still be used inside the automation script at runtime.

Determining script sources

Automation system supports multiple extensions and so we need a way to configure multiple sources (search paths).

These are defined by the Corredor’s CORREDOR_EXT_SEARCH_PATHS .env variable. The value must conform to the pattern of: CORREDOR_EXT_SEARCH_PATHS=ABS_PATH_1:ABS_PATH_2:…:ABS_PATH_N

Example search path for multiple extensions:

CORREDOR_EXT_SEARCH_PATHS=/opt/extension/crm:/opt/extension/brm

For more details and additional configuration refer to the README.

Determining script name

Script names provide a unique identification that can be used to reference scripts through the entire system. Script name is defined as:

/${path to script}/${file name}:${export name}(1)(2)(3)

1	Path to the script file, excluding the search path.
2	The script’s file name.
3	Used export name; this will normally be `default`.

Example automation script name:

/client-scripts/compose/crm/UpdateLeadSource:default

As stated above, the base path (the search path defined in the .env) is excluded from the automation script name. This assures that the name can be used over multiple systems (eg. development, staging and production). This also allows us to overwrite different automation scripts with our own Overwriting automation scripts.

The ability to overwrite scripts opens the possibility of unwanted behavior, such as name collisions. To avoid this, we recommend that the name of the extension is repeated in the path under scripts (note how the crm is repeated in the below example).

Example:

/opt/extension/crm/server-scripts/crm/Case/SetLabel.js
/opt/extension/crm/client-scripts/compose/crm/Account/SetAddress.js

Overwriting automation scripts

The ability to overwrite automation scripts can come in handy, when we want to modify or all together replace a specific automation script. For example, we would like to replace the /crm/Contact/SetLabel:default automation script. If we create a file under /opt/extensions/crm/server-scripts/crm/Contact/SetLabel.js, Corredor would prefer our definition instead of the default definition.

Corredor will load resources from the provided list of search paths in the order they were defined.

Dependencies

Dependencies are an important part of any larger JavaScript code and so the system allows the use of external dependencies defined inside the standard package.json and yarn.lock files.

The dependencies are loaded for each extension and are scoped to it alone; meaning that if two extensions use the same package, both should define it in their own package.json. The dependencies can then be used as elsewhere.

Bundling

Due to some limitations, server scripts are not bundled. This will be resolved at some later point in time.

Client scripts

All client automation scripts are bundled into multiple bundles for each available service (Auth, Admin, Low Code, Messaging and One). Bundling enables the use of external dependencies (see [ext-facility-csdeps] section) and increases consistency across different web browsers.

Script bundling consists of the following steps:

load all valid client automation scripts grouped by available services,
for each service:
1. create a boot loader
2. use Webpack to create a bundle based on the boot loader,

Boot loading creates a file containing { name, triggers, security } JSON objects for each automation script. On Webpack bundle, the boot loader file is used to create the final bundle for the given service. See web-app Event bus section for details on trigger registration and script execution.

Corteza Server

The second important bit of the automation system is the Corteza server. On it’s own, Corredor server is unable to perform any script execution without an explicit invocation (either by the Corteza server or manually via BloomRPC or similar).

Main responsibilities

Load and register automation scripts on the event bus,
run scheduler for deferred automation script execution,
setup http handler for sink routes,
setup http handler for explicit server script invocation,
setup http handler for automation script listing,
setup http handler for automation script bundles,
request script execution based on the provided event.

Event bus

In order to implement a powerful and flexible automation system, we have decided to base it around an event bus. With the help of an event bus, we are able to trivially handle any combination of automation scripts in a unified, robust manner.

Server side event bus and client side event bus are not the same, so both should receive equal attention when reading.

Server script trigger registration

Firstly we need to register implicit, deferred, sink and iterator server scripts on the event bus, so they can listen and respond to dispatched events.

There is a slight difference between iterator automation scripts and other types of automation scripts (excluding explicit scripts) when it comes to their execution. Iterator scripts are executed for each resource in the matched set, so their handler functions differ.

If the script is an iterator: When an iterator script is provided, the handler function invokes the automation script for each resource in the given set,
If the script is any other (except explicit scripts): When any other script is provided (excluding explicit scripts), the handler function is a generic function that executes for the given event on the given resource.

Other than that, the flow is as follows:

fetch available automation scripts from Corredor server,
for each trigger of every valid automation script (excluding explicit scripts):
1. prepare a handler function with respect to the above comment,
2. register the trigger with the handler function on the event bus.

Event handling and script execution

Finally we discuss event dispatching, handling and script execution. Events are dispatched over the event bus either synchronously or asynchronously. When an event is dispatched, the following happens:

find all scripts that can be executed for the given event,
execute every automation script in the filtered set,
if an error is thrown (other then 'Aborted'), stop the execution.

Scheduler

The scheduler system is responsible for deferred automation script execution. The system ticks every minute, at the minute, and dispatches events via the event bus.

There is no mechanism in place that would prevent the automation scripts to overlap. Either simplify/optimize the script or in the case of complex operations, move to batch processing.

For example:

Script A runs every minute and the execution takes 1.5min,
on first tick, script A is executed,
on second tick, script A is executed again along side the old instance.

Sink

The sink system allows us to implement custom endpoints on the Corteza server. System’s flow:

intercept any HTTP request on the /sink endpoint,
basic request validation (provided required parameters),
validate request against the sink’s signature,
process request based on:
- Content-Type or any other HTTP header,
- remote address (IP),
- request method and path,
- username and password (HTTP basic auth)
- GET (query string) parameters
- POST parameters
dispatch the event via event bus.

Web Application

The last important bit of the automation system is the web application. The web application implements a small subset of Corteza server’s automation system features that allow script listing and their execution. The main purpose of this system is to provide the ability to interact with the user from inside automation scripts.

Since client scripts are executed in the browser, they are considered less secure. These scripts should not work with any sensitive data such as authentication tokens, API credentials, personal data filtering, …

Main responsibilities

Fetching and registration of automation scripts provided by Corredor server via Corteza server,
registration of automation triggers on the system event bus,
execution of automation scripts based on implicit/explicit events.

Event bus

Client side event bus and server side event bus are not the same, so both should receive equal attention when reading.

Client script trigger registration

Firstly we need to register client scripts on the event bus, so they can listen and respond to dispatched events. The flow is as follows:

fetch the bundled automation scripts for the given service,
for each trigger of every automation script:
1. prepare a generic execution function,
2. register the trigger with the execution function on the event bus.

Explicit server script trigger registration

Secondly we need to register explicit server scripts (manual server scripts). The flow is as follows:

fetch available automation scripts, excluding non explicit and non server script entries,
for each trigger of every automation script:
1. prepare an execution function that invokes the Corteza server via API client libraries,
2. register the trigger with the execution function on the event bus.

Event handling and script execution

Finally we discuss event dispatching, handling and script execution. Events are dispatched over the event bus and executed synchronously.

There is a slight difference between event validation when it comes to implicit and explicit scripts.

If the script name is provided: when the script name is provided, it is assumed, that we are invoking an explicit script. In this case, the event type must be defined as manual ('onManual'),
If the script name is not provided: when the script name is not provided, it is assumed, that we are invoking implicit script(s). In this case, the event type must not be defined as manual ('onManual').

Other than that, the flow is as follows:

validate the event,
filter triggers based on the provided event,
for each available trigger:
1. prepare context and payload,
2. execute the automation with the above generated meta,
3. handle possible errors.

When it comes to automation script execution, there is a small deviation between Corredor server and web application. When the automation script is executed inside the web application, it’s resources are provided by reference and therefore any changes made to the resource is reflected through the rest of the pipeline without the need of explicit return statements. For some resources, such as records, the changes are also reflected on the UI.

UI hooks

A UI hook represents a UI component (usually a button), that is able to invoke an explicit automation script, being a client or server script. UI hooks solely provide component definition (script name, element type, design, labels, …) and it’s up to the user interface to render the component and dispatch the event when needed.

Triggers

Automation script triggers determine when, where, execution context; input and output of the automation script.

Triggers consist of 4 components:

Event type: Defines the event that should invoke the script’s execution. See [ext-resevt] for available event types.
Resource type: Defines the resource that the automation script is compatible with. This also defines the execution context, input and output of the automation script. See [ext-resevt] for available resource types and their meta,
Constraints: Defines a set of constraints that must be truthly in order for this script to be available. Refer to [ext-dev-triggerAnatomy] for details,
user interface properties (uiProps): Used only for manual triggers. Defines a set of parameters that define where and how the UI hook component will be rendered.

Iterators

Automation script iterators determine when, the execution context; the input and output of the automation script.

Iterators consist of 4 components:

Event type: Defines the event that should invoke the script’s execution. See [ext-resevt] for available event types.
Resource type: Defines the resource that the automation script is compatible with. This also defines the execution context, input and output of the automation script. See [ext-resevt] for available resource types and their meta.
Action: Defines the action that the automation script would like to perform.
Filter: Defines the filter that determines the resources that should be processed by the automation script. Refer to [ext-dev-iteratorConstraints] for details.

Client Scripts

File structure

For increased consistency across different types of user agents (different browsers, such as Google Chrome, Internet Explorer, …) all client scripts are processed and bundled.

For a more intuitive bundle definition, size optimization and consistency, we define the following file structure:

/client-scripts (1)
    /auth (2)
        /... (7)

    /admin (3)
        /... (7)

    /compose (4)
        /... (7)

    /messaging (5)
        /... (7)

    /shared (6)
        /... (7)

1	Root folder for all client scripts (under each search path).
2	Defines a bundle for Corteza Auth.
3	Defines a bundle for Corteza Admin.
4	Defines a bundle for Corteza Low Code.
5	Defines a bundle for Corteza Messaging.
6	Reserved directory for any shared logic, such as custom libraries, assets, …
7	Undefined file structure; can be defined as needed.

Client-script path structure

<search-path>/client-scripts/<bundle>/<path-to-script>/*.js

Server Scripts

File structure

Since server scripts are executed in the background on a capable machine, there is no need (at least not yet) for any inelegant grouping, so there is no need for any complicated file structuring. We define the following file structure:

/server-scripts (1)
    /... (2)

1	Root folder for all server scripts (under each search path).
2	Undefined file structure; can be defined as needed.

Server-script path structure

<search-path>/server-scripts/<path-to-script>/*.js