Corteza Server

Corteza server is the back-end of the Corteza ecosystem. The core logic is written in GO, using go-chi for the routing.

Communication between the Corteza server and web applications is done using the REST API and web sockets. Communication between back-end services (Corteza server and Corredor) is done using gRPC.

Non-binary data is handled by the store layer and stored inside a database — currently PostgreSQL and MySQL. We will add support for additional databases in the future.

Binary data is handled using minIO and stored either locally or on a supported cloud provider.

Development setup

Corteza server runs on GO 1.16. This guide assumes that your GO development environment is already set up on the correct version.

Fork the Corteza server

Core contributors should skip this part.

go to the https://github.com/cortezaproject/corteza-server repository
click on the Fork button in the top right corner and follow the instructions on the screen.

Clone the repository

Using the CLI:

open the CLI
navigate to your working folder
run the git clone git@github.com:$YOUR_USERNAME_HERE/corteza-server.git command

Refer to the repository if you wish to use an alternative method.

Configure the environment

Inside your corteza-server directory:

run cp .env.example .env
open the .env file using VIM, Nano, or any other editor and modify the configuration as you see fit

Setup data storage

If your work requires persistent storage, make sure that you’ve set up a supported database engine. You can run the database either locally or inside a Docker container. Make sure to change the DB_DSN= .env variable.

If your work does not require persistent storage (you will work solely using tests), you can skip this step, and the testing framework will use the in-memory SQLite server. Make sure to remove the DB_DSN= .env variable.

Setup SMTP

For local development, you can use MailHog to log the emails sent from the server. Run the make mailhog.up command to start MailHog. Alternatively, you can use an actual SMTP provider like Gmail or MailGun. Make sure to change the SMTP_*= .env variables.

You can skip this part of your workflow does not require emails. Make sure to remove the SMTP_*= .env variables.

Serve the web applications

Corteza server allows you to serve all of the Corteza web applications for cases where you need to quickly test something on a user interface. Refer to the corteza-server/webapp/README.adoc file for details on how to set it up.

Make sure you’ve properly set the HTTP_ADDR .env variable to match your local environment variable, for example HTTP_ADDR=localhost:80. The default HTTP_ADDR value is HTTP_ADDR=localhost:80.

Port 80 is not available on Windows.

The authentication server uses the HTTP_ADDR and DOMAIN variables when generating cookies. The authentication flow will hang if it’s not defined.

If the login screen doesn’t do anything when you try to login, check if the cookie has the correct domain set.

Run the server

Run the make watch command to start the server. The server will automatically restart when changes to the code are detected.

The watcher can be a bit wonky at times. Manually restart the server if the changes are not getting picked up.

Run the tests

There are a handful of commands to run your tests:

test.integration to run integration tests (test files inside /tests)
test.store to run store tests (test files inside /store/tests)
test.all to run all tests
test.unit to run unit tests
test.pkg to run test files for the /pkg directory

All commands, but the test.all command, accept another .$WHAT_TO_TEST. For example, to only run Compose integration tests, you would run the make test.integration.compose command

Integration testing gotcha:

To prevent your tests from interacting with your development database, you can define an integration test specific .env file inside the /tests directory. There, you can configure a different database or remove it to use the SQLite in-memory database.

Store testing gotcha:

The same test suite runs over all supported databases. If you wish to test for a particular database, you must define the corresponding DSN in the root .env file. Refer to the /store/tests/main_test.go for details.

See the setup containerized database section if you wish to run your test databases in a container.

Setup containerized database

MySQL

We usually use the Percona fork of the MySQL database.

Use the following commands to set up the environment variables; make sure to replace the values if needed:

export DOCKER_NAME=percona;
export ROOT_PWD=root;
export MYSQL_PORT=3306;

To setup the database, perform the following steps:

docker run --name $DOCKER_NAME -e MYSQL_ROOT_PASSWORD=$ROOT_PWD -d -p $MYSQL_PORT:3306 percona:8.0;
docker exec -it $DOCKER_NAME mysql -uroot -p$ROOT_PWD;
CREATE DATABASE corteza;
CREATE USER 'corteza'@'172.17.0.1' IDENTIFIED BY 'corteza';
GRANT ALL PRIVILEGES ON corteza.* TO 'corteza'@'172.17.0.1';
FLUSH PRIVILEGES;

Use the following template to construct the DB_DSN .env variable:

DB_DSN="corteza:corteza@tcp(localhost:$MYSQL_PORT)/corteza?collation=utf8mb4_general_ci"

PostgreSQL

We usually use the official PostgreSQL image.

Use the following commands to set up the environment variables; make sure to replace the values if needed:

export DOCKER_NAME=pgsql2;
export ROOT_PWD=root;
export PGSQL_PORT=5432;

To setup the database, perform the following steps:

docker run --name $DOCKER_NAME -e POSTGRES_PASSWORD=$ROOT_PWD -d -p $PGSQL_PORT:5432 postgres:13;
docker exec -it $DOCKER_NAME psql -U postgres;
CREATE DATABASE corteza;
CREATE USER corteza WITH PASSWORD 'corteza';
GRANT ALL PRIVILEGES ON DATABASE corteza TO corteza;

Use the following template to construct the DB_DSN .env variable:

DB_DSN="postgres://corteza:corteza@localhost:$PGSQL_PORT/corteza?sslmode=disable"

SQLServer

This is a draft so these instructions don’t get lost; will refine at a later stage.

Use one of the following images:

SQLServer 2022: mcr.microsoft.com/mssql/server:2022-latest
SQLServer 2017: mcr.microsoft.com/mssql/server:2017-latest
SQLServer 2019: mcr.microsoft.com/mssql/server:2019-CU10-ubuntu-20.04

Example docker-compose for all three images:

  mssql_2022:
    image: mcr.microsoft.com/mssql/server:2022-latest
    restart: on-failure
    ports: [ "127.0.0.1:1433:1433" ]
    volumes:
      - "dbdata_mssql_2022:/var/opt/mssql"
    environment:
        - ACCEPT_EULA=Y
        - SA_PASSWORD=SUPERSECRET123

  mssql_2017:
    image: mcr.microsoft.com/mssql/server:2017-latest
    restart: on-failure
    ports: [ "127.0.0.1:1433:1433" ]
    volumes:
      - "dbdata_mssql_2017:/var/opt/mssql"
    environment:
        - ACCEPT_EULA=Y
        - SA_PASSWORD=SUPERSECRET123

  mssql_2019:
    image: mcr.microsoft.com/mssql/server:2019-CU10-ubuntu-20.04
    restart: on-failure
    ports: [ "127.0.0.1:1433:1433" ]
    volumes:
      - "dbdata_mssql_2019:/var/opt/mssql"
    environment:
        - ACCEPT_EULA=Y
        - SA_PASSWORD=SUPERSECRET123

To setup the database, perform the following steps:

docker compose exec -it mssql_2022 sh
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P "SUPERSECRET123"
CREATE DATABASE corteza;
GO

Example DB DSN: sqlserver://sa:SUPERSECRET123@localhost:1433?database=corteza :leveloffset: -1

Setup procedure

The initial procedure is located in the root app package which defines orchestration structures to configure, initialize, and start different parts of the system.

CLI commands run boot procedures from the first (setup) to the level required by the command.

Setup procedure levels

Setup

Configures and initializes low-level components, structures required through the entire lifecycle.

More notable components:

Logging: root logging facility which supports human (formatting, colors) or machine (JSON) readable formats,
health check: scheduler, email, and Corredor checks,
sentry: sentry can be used as an error reporting service (critical routines may use their own error/panic reporting functionalities),
email: an SMTP service for relaying sent email messages,
monitoring: a primitive resource usage logging,
scheduler: a facility that controls interval automation execution,
Corredor: a server client for the Corredor server.

When SMTP is disabled (no SMTP host is configured):

User signup no longer requires email verification,
password reset is disabled,
email sending via automation is disabled.

Store initialization

Establishes a connection to the configured database. The post connection procedures run database migrations to conform the schema to the current version.

Service initialization

Initializes all of the services as well as RBAC, resource translations, and (if enabled) federation.

Provision

The provision step assures that the initial resources (users, roles, namespaces, …) are imported from the provision files.

Activate

All of the previously initialized and configured services are started/activated including their watchers.