Extending and Customizing Corteza

Automation

Corteza provides a core with a minimal set of features required for functionality, reliability and security of the system. Automation system allows us to build upon the core functionalities and develop new use case specific features, such as implementing business logic, custom user interface, task automation and much more.

What can automations do

With the help of the automation system we are able to:

Automate common/repetitive tasks: Automate repetitive, time consuming tasks and focus on the important bits. Such tasks can be calculating values, sending emails and generating documents,
Implement business logic: Implement business logic specific to your use case. This can be as simple as sending emails to your clients or implementing a full blown CRM system,
Data processing: Run complex statistic operations on your data to uncover how you should tackle your market,
Data validation: Validate user input to assure data validity and thus assuring a robust system.

There is a lot more that we can achieve with the automation system. For additional examples and ideas refer to the Extension Examples section or get in touch with us.

Example use cases

Some of the more common use cases include:

sending a welcome email to a newly registered user,
inserting calculated price fields,
validating user input,
performing additional data processing before saving data,
implementing external integrations such as DocVerify and Twilio.

Prerequisite

Before we can dive into the fun bits, we need to cover some basics and setup the system. Automation scripts are written in JavaScript and executed either inside the user’s web browser or inside the specialized Corredor server.

Installation and setup

In order for Corteza system and automation scripts to function properly, you are required to firstly setup Corteza server and all relevant front-end web applications. After that you have to setup the Corredor server. Refer to [maint-index] for details on the detailed setup process.

Modern JavaScript

In order to tackle automation script development we strongly recommend becoming familiar with modern JavaScript, since all following examples heavily rely on it. If your knowledge of JavaScript is up to date, you can safely skip this subsection. The rest of the subsection provides a quick overview of the important bits, and points you to other resources for further reading.

About async/await and Promises

In short: Promises solve the problem of asynchronous code and remove the need for cumbersome callbacks via function parameters. They provide a simple and readable syntax and error handling via .catch().

This MDN article goes into more details: about using promises.

About use of await/async syntactic sugar

The await keyword allows us to implement async code in a iterative like fashion by blocking execution until the promise resolves. It’s important to note that the await keyword can only be used within async functions.

This MDN article goes into more details: about await.

The arrow function expression (⇒)

Arrow functions are used as a compact alternative to a regular function expression. Most of the time, they can easily be used inside .then(), .catch() and .finally().

This MDN article goes into more details and exposes some limitations and why they can not completely replace regular functions: about arrow functions

Terminology

Extension: A collection of automation scripts, corresponding libraries and other assets,
Script or Automation Script: Combination of automation triggers, meta data and the code itself,
Client scripts: Automation scripts executed within a client’s browser,
Server scripts: Automation scripts executed inside the Corredor server,
Triggers or Automation Triggers: A set of rules, constraints and options to define when, where and how an automation script is executed,
Resource Type: Defines the resource that this automation script is compatible with. It also defines the input and output of the automation script,
Event Type: Defines what event causes this automation script to execute. For a full list of event types and availability based on the defined resource see @todo…
Trigger constraints: A set of constraints that must be truthy in order for the automation script to be available and executable,
Trigger weight: Allows us to specify the order of automation script execution for the given event,
User interface properties or UIProps: Determine front-end aspects of the automation script invocation component; such as placement, label and color (see UIHooks),
Explicitly triggered or manual scripts: Automation scripts that are executed explicitly, such as a button press or an API endpoint call,
Implicitly triggered scripts: Automation scripts that are executed implicitly as a collateral to an event, such as record creation and form submission,
Deferred or scheduled scripts: Automation scripts that are executed at a specified time or on a configured interval,
Batch or iterator scripts: Automation scripts that are executed over a larger set of resources at the same time (used for batch processing).
Sink scripts: Automation scripts that are executed on HTTP requests and allow implementation of new endpoints,
Event bus: Event handling system inside client web applications and on the server,
UIHooks: Define a front-end components that is used to execute explicit (manual) automation scripts (see UIProps).

Automation System Overview

Automation system diagram

This will probably become a UML diagram; this one is for development (faster updates). @todo add some annotations and explanations on the diagram.

Figure 1. Abstract representation of the automation system.

Automation system components

Corredor server

When Corredor server starts, it

scans all search paths from configuration (.env) and load all scripts,
scans all package.json files in all search paths, and install additional packages,
runs file-change watchers for automation script reloading,
runs file-change watchers for package.json files for dependency reloading.

Corteza server

When Corteza server starts, it

connects to Corredor server and loads all server and client scripts,
creates event handlers from server script triggers and registers them in the event bus,
configures the scheduler system to trigger both onTimestamp and onInterval events once per minute at the minute.

When scheduling system on Corteza server ticks (once per minute), it

dispatches event through the event bus,
checks for registered events with compatible constraints,
requests Corredor server to execute the relevant scripts.

When user indirectly runs automation script via some action on the server (implicitly triggered server scripts)

raises a specific event (onSomething, beforeSomethingElse, afterSomethingElse) and dispatches it through the event bus,
checks for registered events with compatible constraints,
requests Corredor server to execute the relevant scripts.

When request is sent to /sink endpoint

raises a onRequest event and dispatches it through the event bus,
checks for registered events with compatible constraints,
requests Corredor server to execute the relevant scripts.

Corteza web application

When Corteza web application loads in a browser, it

fetches appropriate automation scripts from the Corteza server in the form of a bundle,
registers client scripts from the bundle in the event bus and the ui hooks.

When user clicks on an automation button (explicitly triggered client script), it

raises an event from the given configuration and the invocation context (related resource, user, custom arguments, …),
checks for registered events with compatible constraints,
requests the script’s execution inside the browser context (client-side execution).

When user clicks on an automation button (explicitly triggered server script), it

raises an event from the given configuration and the invocation context (related resource, user, custom arguments, …),
checks for registered events with compatible constraints,
requests Corredor server to execute the relevant scripts via the Corteza server.

When user indirectly runs automation script via some action in the client app (implicitly triggered client scripts), it

raises a specific event (onSomething, beforeSomethingElse, afterSomethingElse) and dispatches it through the event bus,
checks for registered events with compatible constraints,
requests the script’s execution inside the browser context (client-side execution).

Security

As with any system, automation scripts can cause unwanted complications if misused or left open for modification by users who might do harm, willingly or otherwise.

You (or whoever is responsible for the system) is responsible for making sure that the automation scripts are audited and written in a way that they are not harmful to the system or the data.

Automation script execution

Automation script execution is not jailed or virtualized and it is ran in the same context as the automation system; either Corredor in the case of server scripts or any front-end application in case of client scripts.

It is recommended to make use of security context feature described in the following subsection, and limit the permissions each automation script can use.

Security context

Automation scripts are no exception to Corteza permission system; any operation performed by an automation script (such as record creation) must pass all permission checks required by the resource. When not defined otherwise, manual and implicit automation triggers use the current user’s permission scope for the automation scripts security context.

But there are cases where this is not sufficient, and the execution requires permissions that the invoker does not have. For such cases we can specify a custom security context that the automation script will run in.

An automation script requires system user creation, which requires a specific set of privileges that user, running the script might not have. Instead of giving our users elevated privileges, we can define the security context of the automation script.

The same also applies to deferred, interval and sink automation scripts, since they are executed by the system without any user interaction (at least we can’t be sure who exactly executed it). In this case, the security context is required.

Due to privacy and security concerns, security context definition is not available for client side scripts.

When using custom permission context, we strongly recommend using a system user with the minimal required set of permissions to execute the required operations.

Role-based access control

Role-based access control allows us to define what user can access what automation scripts based on their permissions.

This only applies to explicit and client side automation scripts; server side scripts (except explicit scripts) ignore this check.

Resources and Events

Compose events

system

List of events on system

onManual
onInterval
onTimestamp

Table 1. Event arguments for `system`
Name	Type	Immutable
`invoker`	`auth.Identifiable`	false

system:sink

List of events on system:sink

onRequest

Table 2. Event arguments for `system:sink`
Name	Type	Immutable
`response`	`*types.SinkResponse`	false
`request`	`*types.SinkRequest`	true
`invoker`	`auth.Identifiable`	false

system:auth

List of events on system:auth

beforeLogin
beforeSignup
afterLogin
afterSignup

Table 3. Event arguments for `system:auth`
Name	Type	Immutable
`user`	`*types.User`	false
`provider`	`*types.AuthProvider`	false
`invoker`	`auth.Identifiable`	false

system:user

List of events on system:user

onManual
beforeCreate
beforeUpdate
beforeDelete
afterCreate
afterUpdate
afterDelete

Table 4. Event arguments for `system:user`
Name	Type	Immutable
`user`	`*types.User`	false
`oldUser`	`*types.User`	true
`invoker`	`auth.Identifiable`	false

system:role

List of events on system:role

onManual
beforeCreate
beforeUpdate
beforeDelete
afterCreate
afterUpdate
afterDelete

Table 5. Event arguments for `system:role`
Name	Type	Immutable
`role`	`*types.Role`	false
`oldRole`	`*types.Role`	true
`invoker`	`auth.Identifiable`	false

system:role:member

List of events on system:role:member

beforeAdd
beforeRemove
afterAdd
afterRemove

Table 6. Event arguments for `system:role:member`
Name	Type	Immutable
`user`	`*types.User`	false
`role`	`*types.Role`	false
`invoker`	`auth.Identifiable`	false

system:application

List of events on system:application

onManual
beforeCreate
beforeUpdate
beforeDelete
afterCreate
afterUpdate
afterDelete

Table 7. Event arguments for `system:application`
Name	Type	Immutable
`application`	`*types.Application`	false
`oldApplication`	`*types.Application`	true
`invoker`	`auth.Identifiable`	false

Compose events

compose

List of events on compose

onManual
onInterval
onTimestamp

Table 8. Event arguments for `compose`
Name	Type	Immutable
`invoker`	`auth.Identifiable`	false

compose:namespace

List of events on compose:namespace

onManual
beforeCreate
beforeUpdate
beforeDelete
afterCreate
afterUpdate
afterDelete

Table 9. Event arguments for `compose:namespace`
Name	Type	Immutable
`namespace`	`*types.Namespace`	false
`oldNamespace`	`*types.Namespace`	true
`invoker`	`auth.Identifiable`	false

compose:page

List of events on compose:page

onManual
beforeCreate
beforeUpdate
beforeDelete
afterCreate
afterUpdate
afterDelete

Table 10. Event arguments for `compose:page`
Name	Type	Immutable
`page`	`*types.Page`	false
`oldPage`	`*types.Page`	true
`namespace`	`*types.Namespace`	true
`invoker`	`auth.Identifiable`	false

compose:module

List of events on compose:module

onManual
beforeCreate
beforeUpdate
beforeDelete
afterCreate
afterUpdate
afterDelete

Table 11. Event arguments for `compose:module`
Name	Type	Immutable
`module`	`*types.Module`	false
`oldModule`	`*types.Module`	true
`namespace`	`*types.Namespace`	true
`invoker`	`auth.Identifiable`	false

compose:record

List of events on compose:record

onManual
onIteration
beforeCreate
beforeUpdate
beforeDelete
afterCreate
afterUpdate
afterDelete

Table 12. Event arguments for `compose:record`
Name	Type	Immutable
`record`	`*types.Record`	false
`oldRecord`	`*types.Record`	true
`module`	`*types.Module`	true
`namespace`	`*types.Namespace`	true
`recordValueErrors`	`*types.RecordValueErrorSet`	false
`invoker`	`auth.Identifiable`	false

Compose events

messaging

List of events on messaging

onManual
onInterval
onTimestamp

Table 13. Event arguments for `messaging`
Name	Type	Immutable
`invoker`	`auth.Identifiable`	false

messaging:message

List of events on messaging:message

onManual
beforeCreate
beforeUpdate
beforeDelete
afterCreate
afterUpdate
afterDelete

Table 14. Event arguments for `messaging:message`
Name	Type	Immutable
`message`	`*types.Message`	false
`oldMessage`	`*types.Message`	true
`channel`	`*types.Channel`	false
`invoker`	`auth.Identifiable`	false

messaging:command

List of events on messaging:command

onInvoke

Table 15. Event arguments for `messaging:command`
Name	Type	Immutable
`command`	`*types.Command`	true
`channel`	`*types.Channel`	true
`invoker`	`auth.Identifiable`	false

messaging:channel

List of events on messaging:channel

onManual
beforeCreate
beforeUpdate
beforeDelete
afterCreate
afterUpdate
afterDelete

Table 16. Event arguments for `messaging:channel`
Name	Type	Immutable
`channel`	`*types.Channel`	false
`oldChannel`	`*types.Channel`	true
`invoker`	`auth.Identifiable`	false

messaging:channel:member

List of events on messaging:channel:member

beforeJoin
beforePart
beforeAdd
beforeRemove
afterJoin
afterPart
afterAdd
afterRemove

Table 17. Event arguments for `messaging:channel:member`
Name	Type	Immutable
`member`	`*types.ChannelMember`	false
`channel`	`*types.Channel`	false
`invoker`	`auth.Identifiable`	false

Roadmap

Git repository as an extension source: Upgrade allows easier extension management by using various git repositories,
Support basic git operations: Upgrade allows easier extension management, such as changing extension version,
Automatic automation script testing: Upgrade enables easier and faster development, since local testing environment is no longer required,
Secret vault: Upgrade enables consistent and secure storage for sensitive data such as API credentials,
Custom UI components and style assets: Upgrade enables the customization of the entire user interface from the ground up,
File upload via gRPC service: Upgrade enables us to handle files directly from the automation scripts,
Support TypeScript for automation scripts: Upgrade enables us to use the more robust TypeScript as a language for automation scripts.
Integrated Development Environment: To allow developers to setup local Corredor environment with editor, overview and diagnostics and debug tools.
Monitoring: Performance metrics to allow developers improve performance of their scripts

Automation System Facility

The goal of this section is to provide a detailed, low-level overview of the automation system. Detailed understanding is crucial for maintaining existing features and development of additional features. It’s also a plus when performing complex debugging.

If you are interested solely in automation script development, feel free to skip to the Extension Development section. Any important bit from this section will be included in the following sections.

Corredor

Corredor is the main component in the entire automation system. It’s a Node.js system written in TypeScript responsible for parsing, serving and executing automation scripts.

Execution can also be performed inside the client web application (see Web Application).

Corredor is unable to perform any execution without an explicit invocation from the Corteza server. Corredor and Corteza servers are connected and communicate via the gRPC protocol with the following services (see protobuf service definition for details):

server scripts with list and exec procedures,
client scripts with list and bundle procedures.

Main responsibilities

Scanning configured search paths for scripts and dependency definition files,
loading, inspecting found scripts; parsing triggers, recording compile errors and preparing a final list of automation scripts,
running dependency management to load extension’s dependencies,
running file change watcher to rebuild the automation scripts and reload dependencies,
bundling all front-end scripts with WebPack,
running gRPC server and exposing the above listed services for script listing and execution.

Invalid automation scripts (invalid or missing trigger, syntax/logic errors, …) are included in the full list including all discovered errors.

When the Corredor server is ran in production mode, it does not perform any file watching and reloading. This can be changed by either running it in development mode or setting environment variables. Refer to the readme for details.

Script Parsing

Script parsing is an important bit of the system and it introduces a few gotchas that you should be aware of. The flow for automation script processing is as follows:

determine available source files and present them in a light-weight File interface (referred to as file) (see Determining script sources),
for each file:
1. determine script name (see Determining script name),
2. convert the source to an AST tree,
3. find and parse the default exported object. If it’s not found or invalid, the parsing for this file is aborted with a descriptive error,
4. parse the exported object into the Script object:
  - determine meta data (label, description); if provided,
  - evaluate security context; if provided,
  - evaluate triggers; if provided,
  - evaluate iterators; if provided,
  - determine exec function.

Triggers and iterators can not be used in the same script. If both are provided, the iterator is given priority.

The core of the automation script is evaluated outside of the source file, meaning any imports and other symbols defined outside of the exported object are ignored. This prevents the use of imports, constants, functions and other bits defined outside of the exported object inside the script’s meta definition (trigger, iterator, label, …). They can however still be used inside the automation script at runtime.

Determining script sources

Automation system supports multiple extensions and so we need a way to configure multiple sources (search paths).

These are defined by the Corredor’s CORREDOR_EXT_SEARCH_PATHS .env variable. The value must conform to the pattern of: CORREDOR_EXT_SEARCH_PATHS=ABS_PATH_1:ABS_PATH_2:…:ABS_PATH_N

Example search path for multiple extensions:

CORREDOR_EXT_SEARCH_PATHS=/opt/extension/crm:/opt/extension/brm

For more details and additional configuration refer to the README.

Determining script name

Script names provide a unique identification that can be used to reference scripts through the entire system. Script name is defined as:

/${path to script}/${file name}:${export name}(1)(2)(3)

1	Path to the script file, excluding the search path.
2	The script’s file name.
3	Used export name; this will normally be `default`.

Example automation script name:

/client-scripts/compose/crm/UpdateLeadSource:default

As stated above, the base path (the search path defined in the .env) is excluded from the automation script name. This assures that the name can be used over multiple systems (eg. development, staging and production). This also allows us to overwrite different automation scripts with our own Overwriting automation scripts.

The ability to overwrite scripts opens the possibility of unwanted behavior, such as name collisions. To avoid this, we recommend that the name of the extension is repeated in the path under scripts (note how the crm is repeated in the below example).

Example:

/opt/extension/crm/server-scripts/crm/Case/SetLabel.js
/opt/extension/crm/client-scripts/compose/crm/Account/SetAddress.js

Overwriting automation scripts

The ability to overwrite automation scripts can come in handy, when we want to modify or all together replace a specific automation script. For example, we would like to replace the /crm/Contact/SetLabel:default automation script. If we create a file under /opt/extensions/crm/server-scripts/crm/Contact/SetLabel.js, Corredor would prefer our definition instead of the default definition.

Corredor will load resources from the provided list of search paths in the order they were defined.

Dependencies

Dependencies are an important part of any larger JavaScript code and so the system allows the use of external dependencies defined inside the standard package.json and yarn.lock files.

The dependencies are loaded for each extension and are scoped to it alone; meaning that if two extensions use the same package, both should define it in their own package.json. The dependencies can then be used as elsewhere.

Bundling

Due to some limitations, server scripts are not bundled. This will be resolved at some later point in time.

Client scripts

All client automation scripts are bundled into multiple bundles for each available service (Auth, Admin, Low Code, Messaging and One). Bundling enables the use of external dependencies (see [ext-facility-csdeps] section) and increases consistency across different web browsers.

Script bundling consists of the following steps:

load all valid client automation scripts grouped by available services,
for each service:
1. create a boot loader
2. use Webpack to create a bundle based on the boot loader,

Boot loading creates a file containing { name, triggers, security } JSON objects for each automation script. On Webpack bundle, the boot loader file is used to create the final bundle for the given service. See web-app Event bus section for details on trigger registration and script execution.

Corteza Server

The second important bit of the automation system is the Corteza server. On it’s own, Corredor server is unable to perform any script execution without an explicit invocation (either by the Corteza server or manually via BloomRPC or similar).

Main responsibilities

Load and register automation scripts on the event bus,
run scheduler for deferred automation script execution,
setup http handler for sink routes,
setup http handler for explicit server script invocation,
setup http handler for automation script listing,
setup http handler for automation script bundles,
request script execution based on the provided event.

Event bus

In order to implement a powerful and flexible automation system, we have decided to base it around an event bus. With the help of an event bus, we are able to trivially handle any combination of automation scripts in a unified, robust manner.

Server side event bus and client side event bus are not the same, so both should receive equal attention when reading.

Server script trigger registration

Firstly we need to register implicit, deferred, sink and iterator server scripts on the event bus, so they can listen and respond to dispatched events.

There is a slight difference between iterator automation scripts and other types of automation scripts (excluding explicit scripts) when it comes to their execution. Iterator scripts are executed for each resource in the matched set, so their handler functions differ.

If the script is an iterator: When an iterator script is provided, the handler function invokes the automation script for each resource in the given set,
If the script is any other (except explicit scripts): When any other script is provided (excluding explicit scripts), the handler function is a generic function that executes for the given event on the given resource.

Other than that, the flow is as follows:

fetch available automation scripts from Corredor server,
for each trigger of every valid automation script (excluding explicit scripts):
1. prepare a handler function with respect to the above comment,
2. register the trigger with the handler function on the event bus.

Event handling and script execution

Finally we discuss event dispatching, handling and script execution. Events are dispatched over the event bus either synchronously or asynchronously. When an event is dispatched, the following happens:

find all scripts that can be executed for the given event,
execute every automation script in the filtered set,
if an error is thrown (other then 'Aborted'), stop the execution.

Scheduler

The scheduler system is responsible for deferred automation script execution. The system ticks every minute, at the minute, and dispatches events via the event bus.

There is no mechanism in place that would prevent the automation scripts to overlap. Either simplify/optimize the script or in the case of complex operations, move to batch processing.

For example:

Script A runs every minute and the execution takes 1.5min,
on first tick, script A is executed,
on second tick, script A is executed again along side the old instance.

Sink

The sink system allows us to implement custom endpoints on the Corteza server. System’s flow:

intercept any HTTP request on the /sink endpoint,
basic request validation (provided required parameters),
validate request against the sink’s signature,
process request based on:
- Content-Type or any other HTTP header,
- remote address (IP),
- request method and path,
- username and password (HTTP basic auth)
- GET (query string) parameters
- POST parameters
dispatch the event via event bus.

Web Application

The last important bit of the automation system is the web application. The web application implements a small subset of Corteza server’s automation system features that allow script listing and their execution. The main purpose of this system is to provide the ability to interact with the user from inside automation scripts.

Since client scripts are executed in the browser, they are considered less secure. These scripts should not work with any sensitive data such as authentication tokens, API credentials, personal data filtering, …

Main responsibilities

Fetching and registration of automation scripts provided by Corredor server via Corteza server,
registration of automation triggers on the system event bus,
execution of automation scripts based on implicit/explicit events.

Event bus

Client side event bus and server side event bus are not the same, so both should receive equal attention when reading.

Client script trigger registration

Firstly we need to register client scripts on the event bus, so they can listen and respond to dispatched events. The flow is as follows:

fetch the bundled automation scripts for the given service,
for each trigger of every automation script:
1. prepare a generic execution function,
2. register the trigger with the execution function on the event bus.

Explicit server script trigger registration

Secondly we need to register explicit server scripts (manual server scripts). The flow is as follows:

fetch available automation scripts, excluding non explicit and non server script entries,
for each trigger of every automation script:
1. prepare an execution function that invokes the Corteza server via API client libraries,
2. register the trigger with the execution function on the event bus.

Event handling and script execution

Finally we discuss event dispatching, handling and script execution. Events are dispatched over the event bus and executed synchronously.

There is a slight difference between event validation when it comes to implicit and explicit scripts.

If the script name is provided: when the script name is provided, it is assumed, that we are invoking an explicit script. In this case, the event type must be defined as manual ('onManual'),
If the script name is not provided: when the script name is not provided, it is assumed, that we are invoking implicit script(s). In this case, the event type must not be defined as manual ('onManual').

Other than that, the flow is as follows:

validate the event,
filter triggers based on the provided event,
for each available trigger:
1. prepare context and payload,
2. execute the automation with the above generated meta,
3. handle possible errors.

When it comes to automation script execution, there is a small deviation between Corredor server and web application. When the automation script is executed inside the web application, it’s resources are provided by reference and therefore any changes made to the resource is reflected through the rest of the pipeline without the need of explicit return statements. For some resources, such as records, the changes are also reflected on the UI.

UI hooks

A UI hook represents a UI component (usually a button), that is able to invoke an explicit automation script, being a client or server script. UI hooks solely provide component definition (script name, element type, design, labels, …) and it’s up to the user interface to render the component and dispatch the event when needed.

Triggers

Automation script triggers determine when, where, execution context; input and output of the automation script.

Triggers consist of 4 components:

Event type: Defines the event that should invoke the script’s execution. See Resources and Events for available event types.
Resource type: Defines the resource that the automation script is compatible with. This also defines the execution context, input and output of the automation script. See Resources and Events for available resource types and their meta,
Constraints: Defines a set of constraints that must be truthly in order for this script to be available. Refer to Constraints for details,
user interface properties (uiProps): Used only for manual triggers. Defines a set of parameters that define where and how the UI hook component will be rendered.

Iterators

Automation script iterators determine when, the execution context; the input and output of the automation script.

Iterators consist of 4 components:

Event type: Defines the event that should invoke the script’s execution. See Resources and Events for available event types.
Resource type: Defines the resource that the automation script is compatible with. This also defines the execution context, input and output of the automation script. See Resources and Events for available resource types and their meta.
Action: Defines the action that the automation script would like to perform.
Filter: Defines the filter that determines the resources that should be processed by the automation script. Refer to Filter for details.

Client Scripts

File structure

For increased consistency across different types of user agents (different browsers, such as Google Chrome, Internet Explorer, …) all client scripts are processed and bundled.

For a more intuitive bundle definition, size optimization and consistency, we define the following file structure:

/client-scripts (1)
    /auth (2)
        /... (7)

    /admin (3)
        /... (7)

    /compose (4)
        /... (7)

    /messaging (5)
        /... (7)

    /shared (6)
        /... (7)

1	Root folder for all client scripts (under each search path).
2	Defines a bundle for Corteza Auth.
3	Defines a bundle for Corteza Admin.
4	Defines a bundle for Corteza Low Code.
5	Defines a bundle for Corteza Messaging.
6	Reserved directory for any shared logic, such as custom libraries, assets, …
7	Undefined file structure; can be defined as needed.

Client-script path structure

<search-path>/client-scripts/<bundle>/<path-to-script>/*.js

Server Scripts

File structure

Since server scripts are executed in the background on a capable machine, there is no need (at least not yet) for any inelegant grouping, so there is no need for any complicated file structuring. We define the following file structure:

/server-scripts (1)
    /... (2)

1	Root folder for all server scripts (under each search path).
2	Undefined file structure; can be defined as needed.

Server-script path structure

<search-path>/server-scripts/<path-to-script>/*.js

Extension Development

The goal of this section is to provide a detailed overview over the automation script development process, while omitting low-level details about the system itself.

The section also provides some tips and tricks for faster, better and more secure development process.

This section is stand alone, meaning it’s independent of the previous section Automation System Facility.

Naming Conventions

We define the following set of naming conventions for a more consistent experience across different extensions:

@wip

File names: File names should be written in pascal case, ending with a .js extension (eg. SetLabel.js),
conv1: thing…

Helper Classes

Helper classes consist of methods that greatly reduce the needed effort and the amount of required code to perform a specific operation, so it’s recommended to use them as much as possible. These classes are also context aware, meaning that the classes are able to determine some base parameters on their own, such as the current module, record and namespace.

Example of creating a new record with the help of Compose helper class (note how we don’t need to specify the namespace):

Compose.makeRecord({ Title: 'Lead title' }, 'Lead')
  .then(myLead => Compose.saveRecord(myLead))
  .then(mySavedLead => {
    mySavedLead.recordID // saved recordID
  })

All helper classes are available in the automation script’s execution context:

ctx.Compose: Provides functions to work with Corteza Low Code, such as accessing, creating and deleting records. See the definition for details,
ctx.ComposeUI: Provides functions to work with user interface for Corteza Low Code, such as opening record pages and showing notifications. See the definition for details,
ctx.Messaging: Provides functions to work with Corteza Messaging, such as creating channels and creating messages. See the definition for details,
ctx.System: Provides functions to work with Corteza core system, such as finding and creating users and managing role membership. See the definition for details.

Automation Script Anatomy

To better understand the process of implementing custom functionalities with the automation system, we define and describe the anatomy of automation scripts, and establish a few rules. An automation script is considered valid if:

a given .js file is structured under client-scripts or server-scripts directory,
a given .js file exports exactly one JS object with the export default keyword,
a given script defines at least one valid trigger or iterator definition,
a given script defines the security context if the script is a deferred or sink script,
a given JS object defines an exec (args, ctx) function.

Invalid automation scripts are still processed by the Corredor, but are excluded from further operations. Detected errors are visible in the Corredor logs. This allows for easier debugging.

Overview of script object properties and function

label: Automation script’s label, used to provide a user-friendly identification of the automation script,
description: A more verbose automation script description,
security: Defines automation script’s security context and role based filtering,
triggers: Defines automation script’s triggers as propery or method. Not compatible with iterator,
iterator: Defines automation script’s iterator as property or method Not compatible with triggers,
exec: Execution function that is invoked when the automation script is triggered.

Security

Script’s security property allows definition of user who’s credentials will be used when executing the script. It also allows access control by defining list of allowed and denied roles that can execute it.

security.runAs: Define security context for the automation script,
security.allow: Explicitly define what roles have access to the automation script,
security.deny: Explicitly define what roles do not have access to the automation script,

security.runAs parameter is only valid for server scripts.

Exec function

The execution function defines the actual code for the automation script, and conforms to the following interface:

interface ScriptFn {
  (args: exec.Args, ctx?: exec.Ctx): unknown;
}

Execution arguments

Execution arguments (first argument) contain the arguments that the automation script can work with, such as a newly created record, updated module, deleted page, … Arguments depend on the automation script’s event and resource (see Resources and Events).

Execution context

Execution context (second argument) contains contextual information about the execution, such as security context, helper class instances, API clients, loggers, …

ctx.console: console object that can be used for logging. When running in the UA, this will be native window.console. When running in Corredor, this will be a Pino instance,
ctx.log: Shortcut for ctx.console.log,
ctx.$authUser: User object corresponding to the security context,
ctx.SystemAPI: Full blown API for Corteza System interaction,
ctx.ComposeAPI: Full blown API for Corteza Compose interaction,
ctx.MessagingAPI: Full blown API for Corteza Messaging interaction,
ctx.System: Helper class instance for the Corteza System,
ctx.Compose: Helper class instance for the Corteza Compose,
ctx.ComposeUI: Helper class instance for the Corteza Compose user interface,
ctx.Messaging: Helper class instance for the Corteza Messaging,
ctx.frontendBaseURL: Base URL used by front-end web applications. This is useful when generating URL’s inside server scripts.

Automation script execution result

When the script’s execution is complete, it should provide one of the following results:

Unknown Error: An error aborts the script’s execution chain (the current script and all following scripts).
Aborted Error: An error with the value of 'Aborted' stops the execution of the current automation script. Any further scripts down the chain are executed as usual.
false: A return value of false stops the execution of the current automation script. Any further scripts down the chain are executed as usual.
unknown: Any other return value specifies that the execution was successful and the following script (if any) can execute.

If the resource defines a before event variant, the return value will be used as the updated version of that resource. For example: if the automation script is defined with the resource type of compose:record and returns the updated version of the args.$record value, the existing version is replaced by the updated version.

Server script vs. client script

There is a small deviation when it comes to client and server script executions. One of the differences is that, when it comes to client scripts, the arguments are provided as a reference, meaning that any changes to the resource are reflected to the original object. Because of that the resource does not need to be returned in order for the changes to take effect.

For example, running the following code in the client script’s exec function will reflect the values without the need of returning the updated record.

{
  ...
  async exec ({ $record }) {
    $record.values.Field1 = 'value1'
    $record.values.Field2 = 10
  },
}

If your script should be able to revert the changes, in case of an error, you should use an intermediate object for the new values. For example:

{
  ...
  async exec ({ $record }) {
    const v = { ...$record.values }
    v.Field1 = 'value1'
    v.Field2 = 10

    try {
      await apiOperation($record)
    } catch (e) {
      throw new Error(e)
    }

    $record.setValues(v)
  },
}

Automation Trigger Anatomy

To better understand the power of automation triggers and their definition, we define and describe the anatomy of automation triggers.

Automation triggers are constructed with the following methods:

t.on: Defines the event type, such as 'manual', 'request' (see Resources and Events).
t.for: Defines the resource type, such as 'compose:record' (see Resources and Events),
t.where: Defines the constraints that must match in order for the automation script to be executed (eg. t.where('module', 'Lead')). Refer to Constraints for details,
t.before: Defines that the automation script is executed before a specific operation occurs,
t.after: Defines that the automation script is executed after a specific operation occurs,
t.at: Defines that the automation script is executed at a specific time,
t.every: Defines the interval in which the automation script is executed,
t.uiProp: Defines the visual representation of the manual automation trigger, such as its color and label.

Trigger and iterator are not compatible; you can only use one or the other within the same automation script.

Triggers will be evaluated in an isolated context outside of the actual automation script file. This prevents use of any constants or other symbols defiled outside of the trigger.

For example, the following trigger will cause:

const modName = 'Contact'
export default {
  triggers ({ on }) {
    return on('manual')
      .for(modName)
  },
  exec (args, ctx) {...}
}

When defining the automation script, you can use object destructuring to remove a few bits of code.

triggers (t) {
    return [ t.on(...)... ]
}

// can be replaced by

triggers ({ on }) { // any of the above mentioned methods
    return [ on(...)... ]
}

If the automation script contains just a single trigger, you can omit the array and just return the one.

triggers ({ on }) {
    return [ on(...)... ]
}

// can be replaced by

triggers ({ on }) {
    return on(...)...
}

Constraints

A constraint consists of 3 components:

resource attribute name (see [ext-constraintResources]),
operation,
value.

Constraint operations

Equal: check for equality between the two values. Available operators:
- eq
- =
- ==
- ===

Equality check is default operator. When using only 2 parameters for constraint, "equal" operator is used

Not equal: negated check for equality between the two values. Available operators:
- not eq
- ne
- !=
- !==
Like: check for string equality, with support for wildcards. Available operators:
- like
Not like: negated check for string equality, with support for wildcards. Available operators:
- not like

Available wildcards:

one or more characters: %, *,
one character: _, ?.

Match: check for string equality, defined as a regular expression. Available operators:
- ~
Not match: negated check for string equality, defined as a regular expression. Available operators:
- !~

Automation Iterator Anatomy

To better understand the power of automation iterators and their definition, we define and describe their anatomy.

Automation iterator are constructed with the following methods:

i: Defines the resource, action and the filter used for resource fetching. Passed object must conform to the interface:

interface {
  resourceType: string; (1)
  eventType: 'onManual' | 'onInterval' | 'onTimestamp' = 'onManual'; (2)
  action: 'update' | 'delete' | 'clone' | '' = ''; (3)
  filter: IteratorFilter; (4)
}

1	define the resource type; see Resources and Events for details.
2	define the event type; see Resources and Events for details.
3	define what action the script performs; see Iterator actions for details,
4	define the filter to use; see Filter for details. `i.at` Defines that the automation script is executed at a specific time, `i.every` Defines the interval in which the automation script is executed,

Iterator actions

update: The result of the automation script will be used to update the original resource,
delete: The result of the automation script will be used to delete the original resource,
clone: The result of the automation script will be used to create a new resource with updated values; original resource is left unchanged,

Iterator script doesn’t perform any operation on the resource except if explicitly specified by the action property.

Iterator and trigger are not compatible; you can only use one or the other within the same automation script.

Filter

Iterator filter provides ability to write query (conditions), sorting, limit and offset rules It closely resembles Corteza’s filter object and conforms to the following interface:

interface {
  query: string; (1)
  sort: string; (2)
  limit: number | string; (3)
  offset: number | string; (4)
  [_: string]: number | string; (5)
}

1	SQL like query filter to use (`WHERE <query>`).
2	SQL like sort to use (`ORDER BY <sort>`).
3	Maximum number of resource.
4	Offset from first record.
5	additional non-standard resource specific parameters.

Client Scripts

Client scripts are automation scripts executed in the client’s browser (user agent; UA). Client scripts should be used when:

interaction with the user is required,
response latency should be minimal,
execution reliability is not important,
we are not working with sensitive information such as authentication tokens or api credentials,
automation script produces a light load on system resources.

Example use cases

prompt user to confirm form submission,
validate or modify form’s data before submission,
redirect the user to another page; either in Corteza Low Code or other,
open an external resource inside a popup window.

Trigger types

Client scripts support two types of triggers; explicit and implicit.

Explicit

Explicit triggers execute on a specific user invocation, such as a button press. These include:

Manual: Manual automation triggers are defined as buttons inside the user interface. They most commonly appear inside Low Code automation page blocks or inside Low Code record list toolbars. They are invoked and executed inside the web application.

Manual automation triggers don’t need to be defined as client scripts. When defined as a server script it will be invoked from the UA and executed from inside Corredor.

Implicit

Implicit triggers execute as a collateral to another system event such as form submission. These triggers include before/after events. For a full list of available events refer to Resources and Events.

Server Scripts

Server scripts are processed, served and executed by Corredor server.

Server scripts should be used when:

interaction with the user is not required,
response latency is not as important,
execution consistency is important,
automation script produces a heavy load on system resources

Example use case

insert additional record fields based on an external data source,
send an email when a new user signs up,
run statistic operations once a month for reporting purposes.

Trigger types

Server scripts support a few different trigger types; explicit, implicit, deferred, iterators and sink.

Explicit

Explicit triggers execute on a specific user invocation, such as a button press. These include:

Manual: Manual automation triggers are defined as buttons inside the user interface. They most commonly appear inside Low Code automation page blocks or inside Low Code record list toolbars. They are invoked in UA and executed inside Corredor.

Implicit

Implicit triggers execute as a collateral to another system event such as record creation. These triggers include before/after events. For a full list of available events refer to Resources and Events.

Deferred

Deferred automation triggers are executed at a specific point in time; once or multiple times, such as a reminder. These include:

Scheduled: scheduled triggers are executed at an exact time specified by a time stamp in ISO 8601 format (YYYY-MM-DDTHH:mm:ss.sssZ). This trigger is executed exactly once,
Interval: interval triggers are executed periodically in an interval defined by a cron expression (see robfig/cron package for details).

Deferred automation triggers are executed at most once every minute, so you should not define an interval or timestamp that uses higher precision (seconds or milliseconds).

Iterators

Iterators are executed similarly to deferred automation triggers (by a schedule or an interval). The main difference is, that the automation script is executed for each resource that matches the provided filter.

Sink

Sink automation triggers are executed on a specific http request. They can be used to implement new routes on the API, such as a web hook to some external service.

@todo signatures

Record Validation

Automation scripts provide the ability to validate records before they are created or updated. This helps us define a system that is more resilient to the dangers of malformed user input.

Record validation from a script can be performed by scripts that are executed on:

server scripts:

beforeCreate,
beforeUpdate.

client scripts:

beforeFormSubmit,
onFormSubmitError.

When we wish to warn the user about a value error, the flow differs from client and server scripts, but the result is the same — a set of validator.ValidatorError objects contained inside a validator.Validated object (refer to [coredev-compose-recordValidation] for details).

interface ValidatorError {
  kind: string;
  message: string;
  meta: { [key: string]: unknown };
}

interface Validated {
  set: ValidatorError[];
}

Server scripts

When we wish to provide value errors from server scripts, we simply throw an instance of validator.ValidatorError. For example:

import { validator } from '@cortezaproject/corteza-js'

export default {
  ...
  async exec ({ $record }) {
    if ($record.value.Field !== 'Super Specific Value') {
      throw new validator.ValidatorError({
        kind: 'invalidValue',
        message: 'You didn\'t inter the super specific value',
        meta: {
          field: 'Field',
          recordID: $record.recordID,
        },
      })
    }
  },
}

Client Scripts

When executing client scripts, two extra parameters are present:

ctx.validator: The parameter contains the compose.RecordValidator object, that can be used to validate the record this script is executing for.
ctx.errors: The parameter contains the validator.Validated object, that contains current errors, and provides a place to store new errors.

When we wish to provide value errors from inside client scripts, we either use the ctx.validator or construct a validator.ValidatorError object manually. The validator.ValidatorErrors should then be pushed into ctx.errors. For example:

import { validator } from '@cortezaproject/corteza-js'

export default {
  ...
  async exec ({ $record }, { errors, validator }) {
    const errs = new validator.Validated()
    if ($record.values.Field !== 'Super Specific Value') {
      errs.push(new validator.ValidatorError({
        kind: 'invalidValue',
        message: 'You didn\'t inter the super specific value',
        meta: {
          field: 'Field',
          recordID: $record.recordID,
        },
      }))
    }

    $record.values.FullName = `${$record.values.FirstName} ${$record.values.LastName}`
    errs.push(...validator.run($record).set)
    errors.push(...errs)
  },
}

When the validator is unable to find any errors, the returned validator.Validated contains an empty set. Since the value is always present, there is no need for if statements, such as

if (errs && !errs.valid()) {
  errors.push(...errs.set)
}