ledger-api-test-tool - Add conformance test for parallel command dedu…

…plication using CommandSubmissionService [KVL-1099] (digital-asset#10869)

* Extract deduplication "features" into a configuration to be used around the tests.
Better naming for assertions that support sync and async deduplication

CHANGELOG_BEGIN

CHANGELOG_END

* Fix broken test and use consistency for tests

* ledger-api-test-tool - Add conformance test for parallel command deduplication

CHANGELOG_BEGIN
CHANGELOG_END

* Add import for 2.12 compat

* Add silencer plugin

* Split parallel command deduplication scenario into it's own test suite

* Add the parallel command deduplication test to append only ledgers

* Run parallel command deduplication tests for append only ledgers

* Apply suggestions from code review

Co-authored-by: fabiotudone-da <[email protected]>

* Code review renames

* Add compat import

* Run the test concurrently

Force JSON API to refresh packages for GET requests to /v1/query (dig…

…ital-asset#10835)

* Add failing test that covers the bug we found in digital-asset#10823

* Fix /v1/query endpoint bug

changelog_begin
- [JSON API] Fixed a bug that prevented the JSON API to be aware of
  packages uploaded directly via the Ledger API.
changelog_end

DPP-368 clean up flags (digital-asset#10711)

* Make batching parallelism configurable

changelog_begin
changelog_end

* Fail if incompatible cli flags are used

Make Index DB enable multiple party additions [DPP-546] (digital-asse…

…t#10623)

Preparation, small fixes

* Remove unnecessary TODO
* Fix FieldStrategy.idempotentInsert type parameters

Changing the DB-Schema

This change adapts the DB schema for all supported backends. After this change we only populate the party_entries table, and on the query side we reconstruct the state from this.

* Drop the party table
* Add indexing to party_entries

Adapting StorageBackend: ingestion

Since we only ingest party_entries, the party population needs to be removed.

* Drop the party table in ingestion code
* Fixes test

Adapting StorageBackend: queries

Queries needs to be adapted to construct the state from the read side.

* Rewrite queries.
* Fixes reset implementations.

Adapting JdbcLedgerDao

Since underlying storage changed, JdbcLedgerDao can be simplified: no special treatment needed with duplicate errors, since these errors are impossible to happen.
Removing JdbcLedgerDao tests, and adding a new test, testing the behavior of the new event-source party model. Please note: this database refactoring only applies to the append-only schema, so for the mutating schema the test is disabled.
During implementation a bug surfaced: it was not possible anymore to store the is_local information via JdbcLedgerDao.storePartyEntry. Although this bug is a minor issue, since that method is only used from single participant environment, still a fix was also implemented for this, by passing a magic participantId upon non-local party storage.

* Simplify storePartyEntry.
* Fixes bug introduced by append-only.
* adds/adapts tests

Refactoring: remove not used duplicateKeyError from StorageBackend

Changes to JdbcLedgerDao rendered this duplicateKeyError unused.

* Removes unused duplicateKeyError

Adapting sandbox-classic

In sandbox-classic it is not allowed to have updates for parties. Essentially the updates concerning already existent parties were dropped silently with logging without effect.
Here I started by pinning down this behaviour in the SqlLedgerSpec and SqlLedgerSpecAppendOnly. These tests were implemented with the original code in mind.
Then adapted the SqlLedger method: making sure of uniqueness by first trying to lookup the to-be-persisted party.

* Added tests grabbing a hold on original behavior
* Adapted implementation to ensure same behavior

Switching to correct is_local derivation for party queries as per review

* Adapting implementation: switching to aggregated table and a query on that
* Introducing QueryStrategy.booleanOrAggregationFunction to support Oracle
* Moving party related queries to PartyStorageBackendTemplate
* Fixes JdbcLedgerDaoPartiesSpec tests, and add another test case

Also:

* Align Update interface documentation
* Switching to explicit optionality in party query implementation asa per review

Co-authored-by: Simon Meier <[email protected]>

CHANGELOG_BEGIN
CHANGELOG_END

test: test for platform independent dars (digital-asset#10535)

We add a CI test to check that dars don't depend on the underlying
operating system where the dar is build.

CHANGELOG_BEGIN
CHANGELOG_END

Fix sdk version in script export tests (digital-asset#10559)

changelog_begin
changelog_end

[JSON-API] Query fix & additional DB tests to cover the fix (digital-…

…asset#10495)

* [JSON-API] Move database independent tests into a seperate abstract test (digital-asset#10482)

* [JSON-API] Move database independent tests into a seperate abstract test

The DatabaseStartupOps tests are now also tested against Oracle.

changelog_begin
- [JSON-API] Fixed that the schema version handling did not work correctly when using OracleDB
changelog_end

* Add missing copyright headers

* Adjusting the version query slightly to fix the oracle db integration tests

* Rewrite the version query of oracle to fix it (hopefully)

* Test the prefix collision the other way around

* Put the table prefix also infront of the ensure_json constraint in the oracle queries

* Convert the table name of the jsonApSchemaVersion table to uppercase so it can be found in the list of the created tables in Oracle.

* Fix scala 2.12 collection compatibility compiler error by using :+

* Update ledger-service/db-backend/src/main/scala/com/digitalasset/http/dbbackend/Queries.scala

Co-authored-by: Moritz Kiefer <[email protected]>

* Use flatTraverse instead of flatMap to fix the compile error in Queries.scala

* Process the startup mode also in the tests & error if it failed

* Add collections compat import to fix scala 2.12 build failure

* Be confused about the build error prior, revert the change

* Move dropAllTablesIfExist a bit down to have a better declaration order

* Extract the tables vector combined with the version table into a seperate val

* Remove debug in Queries.scala logging

* Make the initDatabaseDdlsAndVersionTable val lazy, so we don't get a nullpointer exception

Co-authored-by: Moritz Kiefer <[email protected]>

* Remove some code which is unrelated to the fix and the db tests

* Fix the oracle tests, I forgot to adapt them prior

Co-authored-by: Moritz Kiefer <[email protected]>

[JSON-API] Query fix & additional DB tests to cover the fix (digital-…

…asset#10495)

* [JSON-API] Move database independent tests into a seperate abstract test (digital-asset#10482)

* [JSON-API] Move database independent tests into a seperate abstract test

The DatabaseStartupOps tests are now also tested against Oracle.

changelog_begin
- [JSON-API] Fixed that the schema version handling did not work correctly when using OracleDB
changelog_end

* Add missing copyright headers

* Adjusting the version query slightly to fix the oracle db integration tests

* Rewrite the version query of oracle to fix it (hopefully)

* Test the prefix collision the other way around

* Put the table prefix also infront of the ensure_json constraint in the oracle queries

* Convert the table name of the jsonApSchemaVersion table to uppercase so it can be found in the list of the created tables in Oracle.

* Fix scala 2.12 collection compatibility compiler error by using :+

* Update ledger-service/db-backend/src/main/scala/com/digitalasset/http/dbbackend/Queries.scala

Co-authored-by: Moritz Kiefer <[email protected]>

* Use flatTraverse instead of flatMap to fix the compile error in Queries.scala

* Process the startup mode also in the tests & error if it failed

* Add collections compat import to fix scala 2.12 build failure

* Be confused about the build error prior, revert the change

* Move dropAllTablesIfExist a bit down to have a better declaration order

* Extract the tables vector combined with the version table into a seperate val

* Remove debug in Queries.scala logging

* Make the initDatabaseDdlsAndVersionTable val lazy, so we don't get a nullpointer exception

Co-authored-by: Moritz Kiefer <[email protected]>

* Remove some code which is unrelated to the fix and the db tests

* Fix the oracle tests, I forgot to adapt them prior

Co-authored-by: Moritz Kiefer <[email protected]>

KV: Fix KV's TransactionCommitter's rejection trace logs [KVL-1023] (d…

…igital-asset#10454)

* Fix KV TransactionCommitter's rejection trace logs

CHANGELOG_BEGIN
CHANGELOG_END

* Remove unused logging context in Rejections.buildRejectionEntry

* fmt

* Log transaction rejection during post-execution as well

* buildRejectionStep -> reject

* immediateRejectionStep -> reject

* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/validator/preexecution/WriteSetSelector.scala

Co-authored-by: Samir Talwar <[email protected]>

* Hide buildRejectionEntry and wrap it as preExecutionOutOfTimeBoundsRejectionEntry

* Fix post-executor's rejection reason

* Improve preExecutionOutOfTimeBoundsRejectionEntry

* Remove unused import

Co-authored-by: Samir Talwar <[email protected]>

Moved DamlSubmission* into separate proto file [KVL-980] (digital-ass…

…et#10362)

CHANGELOG_BEGIN
[Integration Kit] Moved definitions of `DamlSubmission` and `DamlSubmissionBatch` to a separate proto file under the package `com.daml.ledger.participant.state.kvutils.wire`. In case you are directly referencing these messages you will have to update your imports.
CHANGELOG_END