From 7d14d68bfa33bc018ef44f9462af747f2da407aa Mon Sep 17 00:00:00 2001 From: Ilian Iliev Date: Fri, 6 Jun 2025 13:18:24 +0300 Subject: [PATCH 1/2] Moving time formatting examples to separate page Updating examples of how to set keys TTL --- .../formatting-date-and-time-values.md | 122 ++++++++++++++++++ .../redis-expiration-example.md | 100 ++------------ 2 files changed, 133 insertions(+), 89 deletions(-) create mode 100644 content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md new file mode 100644 index 0000000000..5032481fa7 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md @@ -0,0 +1,122 @@ +--- +Title: Formatting date and time values +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: null +group: di +linkTitle: Formatting date and time values +summary: Redis Data Integration keeps Redis in sync with a primary database in near + real time. +type: integration +weight: 40 +--- + +How to format date and time values depends on the source database and the data type of the field, and how they are represented in the incoming record. Below are examples for different databases and data types. + +## Oracle + +Oracle supports the following date and time data types: + +- `DATE` - represented by debezium as a 64-bit integer representing the milliseconds since epoch + ```yaml + transform: + - uses: add_field + with: + fields: + - field: formatted_date + language: sql + # Date is stored as a Unix timestamp in milliseconds so you need to + # divide it by 1000 to convert it to seconds. + expression: STRFTIME('%Y-%m-%d %H:%M:%S', DATE / 1000, 'unixepoch') + # Example: 1749047572000 is transformed to 2025-06-04 14:32:52 + ``` +- `TIMESTAMP` - the value is represented by Debezium as a 64-bit integer and depends on the number of decimal places of precision of the column, representing fractions of a second. For example, if the column is defined as `TIMESTAMP(6)`, there are six decimal places and so the value is represented as microseconds since epoch (since there are 10^6 microseconds in each second). +You can format it similarly to `DATE`, but you need to divide the value by the appropriate factor based on the precision. + +- `TIMESTAMP WITH TIME ZONE` - the value is represented as string representation of the timestamp with time zone information. + +- `TIMESTAMP WITH LOCAL TIME ZONE` - the value is represented as string representation of the timestamp with local time zone information. + + Both `TIMESTAMP WITH TIME ZONE` and `TIMESTAMP WITH LOCAL TIME ZONE` are supported by SQLite and can be formatted using the `STRFTIME` function. + + ```yaml + transform: + - uses: add_field + with: + fields: + - field: seconds_since_epoch + language: sql + # Convert the timestamp with local time zone to seconds since epoch. + expression: STRFTIME('%s', TIMESTAMP_FIELD) + + - field: date_from_timestamp + language: sql + # Convert the timestamp with local time zone to date and time. + expression: STRFTIME('%Y-%m-%d %H:%M:%S', TIMESTAMP_FIELD) + ``` + +---- + +## SQL Server +SQL Server supports the following date and time data types: + +- `date` - represented by Debezium as number of days since epoch (1970-01-01). You can multiply the value by 86400 (the number of seconds in a day) to convert it to seconds since epoch and then use the `STRFTIME` or `DATE` functions to format it. + ```yaml + transform: + - uses: add_field + with: + fields: + - field: with_default_date_format + language: sql + # Uses the default DATE format + expression: DATE(event_date * 86400, 'unixepoch') + + - field: with_custom_date_format + language: sql + # Uses the default DATE format + expression: STRFTIME('%Y/%m/%d', event_date * 86400, 'unixepoch') + ``` + +- `datetime`, `smalldatetime` - represented by Debezium as number of milliseconds since epoch. You have to divide the value by 1000 to convert it to seconds since epoch and then use the `STRFTIME` function to format it. + ```yaml + transform: + - uses: add_field + with: + fields: + - field: formatted_datetime + language: sql + expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetime / 1000, 'unixepoch') + ``` + +- `datetime2` - similar to `datetime` but with higher precision. For `datetime2(0-3)` the representation is the same as for `datetime`. For `datetime2(4-6)` it is the number of microseconds since epoch. and for `datetime2(7)` it is the number of nanoseconds since epoch. You can use the same approach as for `datetime` but you need to divide by 1000, 1000000 or 1000000000 depending on the precision. + +- `time` - the time of milliseconds since midnight. + ```yaml + transform: + - uses: add_field + with: + fields: + - field: formatted_time + language: sql + expression: TIME(event_time, 'unixepoch', 'utc') + ``` + +- `datetimeoffset` - represented as a timestamp with timezone information e.g. `2025-05-27T15:21:42.864Z` and `2025-01-02T14:45:30.123+05:00`. + ```yaml + transform: + - uses: add_field + with: + fields: + - field: formatted_datetimeoffset + language: sql + expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetimeoffset) + ``` + + + + + diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-expiration-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-expiration-example.md index b63e19a00e..c522e29628 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-expiration-example.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-expiration-example.md @@ -53,112 +53,34 @@ output: In some cases, you can also set the expiration time based on a field that contains a date, datetime, or timestamp value, but it depends on the source database and the data types it supports. See the examples below for your specific source database and data type. -### Oracle examples +There are two main approaches you can use to set the expiration time based on a date, datetime, or timestamp field: -The transformation depends on the data type of the field in the source database: +- For values representing a passed amount of time e.g. milliseconds since epoch start, you have to convert the value to seconds since epoch and subtracting the current time in seconds since epoch from it. -- `DATE` - represented by debezium as a 64-bit integer representing the milliseconds since epoch - ```yaml - output: - - uses: redis.write - with: - data_type: hash - expire: - # To set the expiration time to a date field, convert the value to seconds and subtract the current time in seconds since epoch - expression: EXPIRES_DATE / 1000 - STRFTIME('%s', 'now') - language: sql - ``` -- `TIMESTAMP` - the value is represented by Debezium as a 64-bit integer and depends on the number of decimal places of precision of the column, representing fractions of a second. For example, if the column is defined as `TIMESTAMP(6)`, there are six decimal places and so the value is represented as microseconds since epoch (since there are 10^6 microseconds in each second). - ```yaml - output: - - uses: redis.write - with: - data_type: hash - expire: - # To set the expiration time to a date field, convert the value to seconds (divider differs based on the fractional second precision) and subtract the current time in seconds since epoch. Example below is for 6 digits of precision. - expression: EXPIRES_TIMESTAMP / 1000000 - STRFTIME('%s', 'now') - language: sql - ``` -- `TIMESTAMP WITH TIME ZONE` - the value is represented as string representation of the timestamp with time zone information. -- `TIMESTAMP WITH LOCAL TIME ZONE` - the value is represented as string representation of the timestamp with local time zone information. - - For both `TIMESTAMP WITH TIME ZONE` and `TIMESTAMP WITH LOCAL TIME ZONE`, a two-step approach is needed. First, calculate the difference between the given time and now in seconds and then invert the value. ```yaml - transform: - - uses: add_field - with: - fields: - - field: expire_seconds - language: jmespath - expression: time_delta_seconds(EXPIRES_TS_TZ) - output: + output: - uses: redis.write with: data_type: hash expire: - # `time_delta_seconds` Returns the number of seconds between a given dt and now. - # A negative value means that the given dt is in the future, so we need to invert it. - # A positive value means that the given dt is in the past, so set the expiration to -1 (expire immediately). - expression: CASE WHEN expire_seconds < 0 THEN -expire_seconds ELSE -1 END + # To set the expiration time to a date field, convert the value to + # seconds (e.g. divide it by 1000 if the fields has milliseconds precision) + # and subtract the current time in seconds since epoch. + expression: EXPIRES_TIMESTAMP / 1000 - STRFTIME('%s', 'now') language: sql ``` ----- - -### SQL Server examples -SQL Server supports the following date and time data types: +- For values matching the subset of ISO 8601 supported by SQLite e.g. `2023-10-01T12:00:00`, `2023-10-01T12:00:00Z` or `2025-06-05T13:40:14.784000+02:00` you can use the `STRFTIME` function to convert the value to seconds since epoch and subtract the current time in seconds since epoch from it. -- `date` - represented in Debezium as number of days since epoch (1970-01-01). Please note that due to the lack of time information, this method is not very accurate. - ```yaml - output: - - uses: redis.write - with: - data_type: hash - expire: - # Calculate the number of seconds equivalent to the number of days and subtract the current time in seconds since epoch. - expression: (event_date * 86400) - strftime('%s', 'now') - language: sql - ``` - -- `datetime`, `smalldatetime` - represented in Debezium as number of milliseconds since epoch. - ```yaml - output: - - uses: redis.write - with: - data_type: hash - expire: - # Since event_datetime is in miliseconds, you must divide it by 1000 to convert it to seconds. - expression: event_datetime / 1000 - strftime('%s', 'now') - language: sql - ``` -- `datetime2` - similar to `datetime` but with higher precision. For `datetime2(0-3)` the representation is the same as for `datetime`. For `datetime2(4-6)` it is the number of microseconds since epoch. and for `datetime2(7)` it is the number of nanoseconds since epoch. You can use the same approach as for `datetime` but you need to divide by 1000, 1000000 or 1000000000 depending on the precision. - -- `time` - the time of milliseconds since midnight. - ```yaml - output: - - uses: redis.write - with: - data_type: hash - expire: - # Convert the time to seconds and subtract the current time in seconds since midnight. - expression: (event_time / 1000.0) - - ( - CAST(strftime('%H', 'now') AS INTEGER) * 3600 + - CAST(strftime('%M', 'now') AS INTEGER) * 60 + - CAST(strftime('%S', 'now') AS INTEGER) - ) - language: sql - ``` -- `datetimeoffset` - represented as a timestamp with timezone information, where the timezone is GMT ```yaml output: - uses: redis.write with: data_type: hash expire: - # Convert the time to seconds and subtract the current time in seconds since epoch. - expression: strftime('%s', event_datetimeoffset) - strftime('%s', 'now') language: sql + expression: STRFTIME('%s', EXPIRATION_TS) - STRFTIME('%s', 'now') ``` - +For more examples on how to manipulate date and time values, see the [Formatting date and time values]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values/">}}) page. + From 27a1286d3c0ce9d112b198d3a52bbf50ad29a799 Mon Sep 17 00:00:00 2001 From: Ilian Iliev Date: Fri, 6 Jun 2025 14:48:18 +0300 Subject: [PATCH 2/2] Adding PostgreSQL examples --- .../formatting-date-and-time-values.md | 67 +++++++++++++++++++ .../transform-examples/map-example.md | 2 +- 2 files changed, 68 insertions(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md index 5032481fa7..c3cbf12eea 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md @@ -120,3 +120,70 @@ SQL Server supports the following date and time data types: + + + +---- + +## PostgreSQL + +PostgreSQL supports the following date and time data types: + +- `date` - represented by Debezium as number of days since epoch (1970-01-01). You can multiply the value by 86400 (the number of seconds in a day) to convert it to seconds since epoch and then use the `STRFTIME` or `DATE` functions to format it. + ```yaml + transform: + - uses: add_field + with: + fields: + - field: with_default_date_format + language: sql + # Uses the default DATE format + expression: DATE(event_date * 86400, 'unixepoch') + ``` + +- `time` - the time of microseconds since midnight. + ```yaml + transform: + - uses: add_field + with: + fields: + - field: formatted_time + language: sql + # Divide by 1000000 to convert microseconds to seconds + expression: TIME(event_time / 1000000, 'unixepoch', 'utc') + ``` + +- `time with time zone` - a string representation of the time with timezone information, where the timezone is GMT, example `07:15:00Z`. + ```yaml + transform: + - uses: add_field + with: + fields: + - field: formatted_time_with_tz + language: sql + expression: STRFTIME('%H:%M:%S', event_time_with_time_zone) + ``` + +- `timestamp` - represented by Debezium as a 64-bit integer representing the microseconds since epoch. You can use the `STRFTIME` function to format it. + ```yaml + transform: + - uses: add_field + with: + fields: + - field: formatted_timestamp + language: sql + # Divide by 1000000 to convert microseconds to seconds + expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_timestamp / 1000000, 'unixepoch') + ``` + +- `timestamp with time zone` - represented by Debezium as a string representation of the timestamp with time zone information, where the timezone is GMT, e.g. `2025-06-07T10:15:00.000000Z` + ```yaml + transform: + - uses: add_field + with: + fields: + - field: formatted_timestamp_with_tz + language: sql + # Divide by 1000000 to convert microseconds to seconds + expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_timestamp_with_time_zone) + ``` diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md index 296297828c..6eb25b841d 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md @@ -167,4 +167,4 @@ like the following: 6) "3:35.0" 7) "storagesize" 8) "6.71MB" -``` \ No newline at end of file +```