Import Low Code Records

2021.9.5 patch release requires you to provide module field definitions along with record import definitions (note the module fields entries in the defs.yaml — snippet below). If the field definitions are not provided, the import process errors out with missing field errors. This requirement will be lifted in a future patch release.

Before importing larger datasets (especially to production), it is advised to make a database backup and to test the configuration locally or on a staging server.

To import Low Code records you need to:

Prepare the source files — the CSV or JSONL files and the definition file(s),
run the command.

Source Files

Prepare a directory where all of your source files reside in.

/import:

/import
  /aa.csv
  /bb.csv
  /cc.csv
  /defs.yaml

The .csv files contain the records we wish to import, and the defs.yaml provides all of the import definitions.

You can use multiple definition files, for example one per module.

You can provide import definition inside your provision YAML files.

defs.yaml:

namespace: my-namespace
modules:
  aa:
    fields:
      f1:
        kind: String
        label: f1
    records: (1)
      source: aa.csv (2)
      key: id (3)
      mapping: (4)
        id: / (5)
        f1: (6)
          field: f1 (7)
  bb:
    fields:
      ref_a:
        kind: Record
        label: ref_a
        options:
          module: aa
      f1:
        kind: String
        label: f1
    records:
      source: bb.csv
      key: id
      mapping:
        id: /
        ref_a:
          field: ref_a
        f1:
          field: f1
  cc:
    fields:
      ref_a:
        kind: Record
        label: ref_a
        options:
          module: aa
      ref_b:
        kind: Record
        label: ref_b
        options:
          module: bb
      f1:
        kind: String
        label: f1
    records:
      source: cc.csv
      key: id
      mapping:
        id: /
        ref_a:
          field: ref_a
        ref_b:
          field: ref_b
        f1:
          field: f1

1	The `records` key contains the record import definitions.
2	The `source` key specifies what source file should be used (the name of the file).
3	The `key` key specifies what column should be used as the identifier for the record when matching references.
4	The `mapping` key specifies how the sources columns should be mapped to the module fields.
5	When you use the `/` instead of a module field, the column is omitted.
6	The key of the map defines the column of the source.
7	The field defines the module field the column value should be mapped to.

aa.csv:

id,f1
aa_1,aa_v_1
aa_2,aa_v_2
aa_3,aa_v_3

bb.csv:

id,ref_a,f1
bb_1,aa_1,bb_v_1
bb_2,aa_1,bb_v_2
bb_3,aa_1,bb_v_3
bb_4,aa_2,bb_v_4
bb_5,aa_2,bb_v_5
bb_6,aa_3,bb_v_6
bb_7,aa_3,bb_v_7

Notice how the ref_a column in the bb.csv file references the id column in the aa.csv file.

cc.csv:

id,ref_a,ref_b,f1
cc_1,aa_1,bb_1,cc_v_1
cc_2,aa_1,bb_1,cc_v_2
cc_3,aa_3,bb_1,cc_v_3
cc_4,aa_3,bb_1,cc_v_4
cc_5,aa_3,bb_6,cc_v_5
cc_6,aa_3,bb_7,cc_v_6

Running the Import

Use the import CLI command to import your records.

The argument to the CLI command needs to be the location of the source files, in our above case this is /import.

An example of running the CLI command for our case:

corteza-server import --merge-left-existing /import

Updating Records

If you wish to update your records using the import CLI command, you must use the record ID that Corteza assigned to the original. The identification process is subject to change in future versions.

Below is an example of updating records of three modules.

The values provided under the id column will be different for your example.

aa.csv:

id,f1
262499403762254961,aa_v_1 (EDITED aa_v_1)
262499403779032177,aa_v_2 (EDITED aa_v_2)
262499403779097713,aa_v_3 (EDITED aa_v_3)

bb.csv:

id,ref_a,f1
262499403779163249,262499403762254961,bb_v_1 (EDITED bb_v_1)
262499403779228785,262499403762254961,bb_v_2 (EDITED bb_v_2)
262499403795809393,262499403762254961,bb_v_3 (EDITED bb_v_3)
262499403795874929,262499403779032177,bb_v_4 (EDITED bb_v_4)
262499403795940465,262499403779032177,bb_v_5 (EDITED bb_v_5)
262499403796006001,262499403779097713,bb_v_6 (EDITED bb_v_6)
262499403796071537,262499403779097713,bb_v_7 (EDITED bb_v_7)

cc.csv:

id,ref_a,ref_b,f1
262499403796137073,262499403762254961,262499403779163249,cc_v_1 (EDITED cc_v_1)
262499403812586609,262499403762254961,262499403779163249,cc_v_2 (EDITED cc_v_2)
262499403812652145,262499403779097713,262499403779163249,cc_v_3 (EDITED cc_v_3)
262499403812717681,262499403779097713,262499403779163249,cc_v_4 (EDITED cc_v_4)
262499403812783217,262499403779097713,262499403796006001,cc_v_5 (EDITED cc_v_5)
262499403812848753,262499403779097713,262499403796071537,cc_v_6 (EDITED cc_v_6)