Skip to content

RDSC RDSC-3626 - adding opcode examples #1647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
Title: Add the opcode to the Redis output
Title: Using the operation code
alwaysopen: false
categories:
- docs
Expand All @@ -8,43 +8,90 @@ categories:
- rdi
description: null
group: di
linkTitle: Add the opcode to the Redis output
linkTitle: Using the operation code
summary: Redis Data Integration keeps Redis in sync with the primary database in near
real time.
type: integration
weight: 100
---

In the example below, the data is captured from the source table named `employee` and is written to the Redis database as a JSON document. When you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`.
The operation code (`opcode`) is a metadata field that indicates the type of operation that generated the change in the source database. It can be useful for tracking changes and understanding the context of the data being processed.

The opcode is only available in the [full row format]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format#full" >}}), and can be accessed in the `transform` and `output` sections of the job file.

It has one of the following values:

- r - Read (applies to only snapshots)
- c - Create
- u - Update
- d - Delete
- t = Truncate (PostgreSQL specific)
- m = Message (PostgreSQL specific)


You can add the value of the operation code to the output, and also use it in a conditional expression to modify the behavior of the job. The following examples demonstrate the different use-cases.

### Adding the operation code to the output

Use the `add_field` transformation to add a new field that contains the value of the `opcode` field from the source data. Note that the fields must be prefixed with `after` to be included in the output.

Here, the result will be Redis JSON documents with fields captured from the source table
(`employeeid`, `firstname`, `lastname`) and also with
an extra field `my_opcode` added using the `merge` update strategy (see the
[JSON job example]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example" >}})
for more information). The `opcode` expression refers to the operation code captured from
the source. This is a database-specific value that indicates which type of operation generated
the change (insert, update, etc).

```yaml
source:
schema: public
table: employee
row_format: full

transform:
# add the operation code to the data
- uses: add_field
with:
field: after.my_opcode
field: after.operation_code
expression: opcode
language: jmespath
output:
- uses: redis.write
```


### Filtering operation by output code.

In some cases you may want to ignore certain operations (for example, you may not be interested in deletions). Use the `filter` transformation to filter out any operations you don't need to process.

```yaml
source:
schema: public
table: employee
row_format: full

transform:
- uses: filter
with:
expression: opcode != 'd'
language: jmespath
```

### Modifying the output based on the operation code

The previous example filters out specific operations, but you can also modify the output based on the operation code. For example, you can add a new field that tracks the status of the record based on the operation code.

Note that when a source record is deleted, you must modify the value of the `opcode` field if you want to prevent the corresponding record in the target database from being removed automatically.

```yaml
source:
schema: public
table: employee
row_format: full

transform:
- uses: add_field
with:
data_type: json
mapping:
- employeeid
- firstname
- lastname
- my_opcode
connection: target
on_update: merge
```
fields:
# Here you set the value of the field based on the value of the opcode field
- field: after.status
expression: opcode == 'd' && 'inactive' || 'active'
language: jmespath

# You have to change the value of the opcode field to prevent deletion
- field: opcode
expression: opcode == 'd' && 'u' || opcode
language: jmespath
```
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ With `row_format: full` the input value is a JSON object with the following stru
- `key` - An object containing the attributes of the primary key. For example, `key.id` will give you the value of the `id` column as long as it is part of the primary key.
- `before` - An object containing the previous value of the row.
- `after` - An object containing the current value of the row.
- `opcode` - The operation code. Different databases use different values for the operation code. See [operation code values]({{< relref "#operation-codes" >}}) below for more information.
- `opcode` - The operation code. See [Using the operation code]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example" >}}) for more information about the possible opcode values and how to use them.
- `db` - The database name.
- `table` - The table name.
- `schema` - The schema name.
Expand Down Expand Up @@ -102,12 +102,3 @@ output:
expression: concat(['addresses-full', '#', values(key)[0]])
language: jmespath
```

## Operation code values {#operation-codes}

- r - Read (applies to only snapshots)
- c - Create
- u - Update
- d - Delete
- t = truncate (PostgreSQL specific)
- m = message (PostgreSQL specific)
Loading