Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingestion) cleaning up ingestion page and aligning with SaaS #12710

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

jayacryl
Copy link
Collaborator

@jayacryl jayacryl commented Feb 22, 2025

image

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added the product PR or Issue related to the DataHub UI/UX label Feb 22, 2025
@@ -61,6 +54,29 @@ function IngestionSourceTable({
}: Props) {
const isShowNavBarRedesign = useShowNavBarRedesign();

const tableData = sources.map((source) => ({
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved this up above columns so we can reference its type

@@ -83,20 +99,7 @@ function IngestionSourceTable({
render: ScheduleColumn,
},
{
title: 'Execution Count',
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consolidated these columns under status

@jayacryl jayacryl requested a review from gabe-lyons February 22, 2025 21:14
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Feb 22, 2025
Copy link

codecov bot commented Feb 22, 2025

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
1816 2 1814 41
View the top 1 failed test(s) by shortest run time
tests.cli.ingest_cmd.test_timeseries_rollback::test_timeseries_rollback
Stack Traces | 191s run time
auth_session = <tests.utils.TestSessionWrapper object at 0x7f3621b564a0>

    def test_timeseries_rollback(auth_session) -> None:
        pipeline = ingest_file_via_rest(
            auth_session, ".../cli/ingest_cmd/test_timeseries_rollback.json"
        )
        test_aspect_name: str = "datasetProfile"
        test_dataset_urn: str = builder.make_dataset_urn(
            "test_rollback",
            "rollback_test_dataset",
            "TEST",
        )
>       datahub_rollback(auth_session, pipeline.config.run_id)

.../cli/ingest_cmd/test_timeseries_rollback.py:72: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

auth_session = <tests.utils.TestSessionWrapper object at 0x7f3621b564a0>
run_id = 'file-2025_02_24-15_06_01-9icwp4'

    def datahub_rollback(auth_session, run_id: str) -> None:
        sync_elastic()
        rollback_args: List[str] = ["ingest", "rollback", "--run-id", run_id, "-f"]
        rollback_result: Result = runner.invoke(
            datahub,
            rollback_args,
            env={
                "DATAHUB_GMS_URL": auth_session.gms_url(),
                "DATAHUB_GMS_TOKEN": auth_session.gms_token(),
            },
        )
>       assert rollback_result.exit_code == 0
E       assert 1 == 0
E        +  where 1 = <Result ConnectionError(MaxRetryError('HTTPConnectionPool(host=\'localhost\', port=8080): Max retries exceeded with url: /runs?action=rollback (Caused by ReadTimeoutError("HTTPConnectionPool(host=\'localhost\', port=8080): Read timed out. (read timeout=30.0)"))'))>.exit_code

.../cli/ingest_cmd/test_timeseries_rollback.py:30: AssertionError
View the full list of 1 ❄️ flaky tests
tests.cypress.integration_test::test_run_cypress

Flake rate in main: 11.32% (Passed 94 times, Failed 12 times)

Stack Traces | 1180s run time
auth_session = <tests.utils.TestSessionWrapper object at 0x7fcb866a56f0>

    def test_run_cypress(auth_session):
        # Run with --record option only if CYPRESS_RECORD_KEY is non-empty
        record_key = os.getenv("CYPRESS_RECORD_KEY")
        tag_arg = ""
        test_strategy = os.getenv("TEST_STRATEGY", None)
        if record_key:
            record_arg = " --record "
            batch_number = os.getenv("BATCH_NUMBER")
            batch_count = os.getenv("BATCH_COUNT")
            if batch_number and batch_count:
                batch_suffix = f"-{batch_number}{batch_count}"
            else:
                batch_suffix = ""
            tag_arg = f" --tag {test_strategy}{batch_suffix}"
        else:
            record_arg = " "
    
        print(f"test strategy is {test_strategy}")
        test_spec_arg = ""
        specs_str = ",".join([f"**/{f}" for f in _get_cypress_tests_batch()])
        test_spec_arg = f" --spec '{specs_str}' "
    
        print("Running Cypress tests with command")
        command = f"NO_COLOR=1 npx cypress run {record_arg} {test_spec_arg} {tag_arg}"
        print(command)
        # Add --headed --spec '**/mutations/mutations.js' (change spec name)
        # in case you want to see the browser for debugging
        print_now()
        proc = subprocess.Popen(
            command,
            shell=True,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            cwd=f"{CYPRESS_TEST_DATA_DIR}",
        )
        assert proc.stdout is not None
        assert proc.stderr is not None
        stdout = proc.stdout.read()
        stderr = proc.stderr.read()
        return_code = proc.wait()
        print(stdout.decode("utf-8"))
        print("stderr output:")
        print(stderr.decode("utf-8"))
        print("return code", return_code)
        print_now()
>       assert return_code == 0
E       assert 1 == 0

tests/cypress/integration_test.py:250: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-review Label for PRs that need review from a maintainer. product PR or Issue related to the DataHub UI/UX
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant