[Logs+] Extract custom integration resources to package (elastic#165510)

## Summary 

This closes elastic#163788.

## High level overview

- Adds a new `kbn-custom-integrations` package.
- This package adds a new top level custom integrations state machine,
which manages a child create custom integration state machine. In the
future we will have additional modes (such as adding a dataset to an
existing integration, and various "uplift" flows).
- Adds connected (to the machine) components that consumers can use to
facilitate custom integration workflows.
- Adds a `kbn-xstate-utils` package (as these utils were in 2 plugins
and now 1 package).
- Replaces the integration creation inside of the onboarding wizard flow
with this package.
- At the moment this is locked down to `logs`, and one dataset, but it
can be easily extended in the future to support all types and multiple
datasets. The state machine is ready, it just needs exposed in the UI.
- Some thought has gone in to how this will work with multiple "modes",
and the foundations are there (imagining that certain types will be
unions etc), however it's worth not getting too bogged down in those
specific implementation details as I'd rather base that evolution on the
real world usage when we have it.

The Configure integration section should more or less work the same as
before.

![Screenshot 2023-09-05 at 16 24
44](https://github.com/elastic/kibana/assets/471693/8891dc0f-0ba2-48e0-83ac-99336369bc50)

## Testing

- When utilising the onboarding flow for custom logs at
`/app/observabilityOnboarding/customLogs` can you:
- Create a custom integration? (It's worth verifying the network
requests, and the assets are installed).
- If you navigate forward, then back, make a change to the integration
fields, and navigate forward again is the previously created integration
deleted?
- Is the success callout with the integration name shown on the next
wizard panel?
  - Do field validations work?
- Are errors displayed when you try to create an integration with a name
that already exists?
- Can you retry when there is a server error? (you can block network
requests to the custom integrations API to test this)

## Screenshots

![Screenshot 2023-09-06 at 10 51
35](https://github.com/elastic/kibana/assets/471693/95cd895c-02a3-482a-af35-b23f30dcba56)

![Screenshot 2023-09-06 at 10 51
57](https://github.com/elastic/kibana/assets/471693/9848dfe6-dae8-43b4-892e-bcfe199248f2)

![Screenshot 2023-09-06 at 10 49
40](https://github.com/elastic/kibana/assets/471693/2cb52e17-bba9-4901-bf77-9e12519f36a9)

![Screenshot 2023-09-06 at 10 52
21](https://github.com/elastic/kibana/assets/471693/4d871ccb-0948-46ee-a095-d1b60fb63d50)

## State machine diagram

(The top level management machine is super basic, so this is just the
create machine)

![Screenshot 2023-09-08 at 16 30
26](https://github.com/elastic/kibana/assets/471693/ccbaa270-e450-4eeb-b8cb-8ae9a41afa39)

## Followups

- Tests (the current onboarding UI implementation doesn't have tests so
whilst it's not ideal technically this coverage stays the same)
- Storybook 
- Replace other plugins' usage with xstate-utils (not urgent)

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Yngrid Coello <[email protected]>

.github/CODEOWNERS

@@ -297,6 +297,7 @@ x-pack/plugins/cross_cluster_replication @elastic/platform-deployment-management
 packages/kbn-crypto @elastic/kibana-security
 packages/kbn-crypto-browser @elastic/kibana-core
 x-pack/plugins/custom_branding @elastic/appex-sharedux
+packages/kbn-custom-integrations @elastic/infra-monitoring-ui
 src/plugins/custom_integrations @elastic/fleet
 packages/kbn-cypress-config @elastic/kibana-operations
 x-pack/plugins/dashboard_enhanced @elastic/kibana-presentation
@@ -806,6 +807,7 @@ src/plugins/visualizations @elastic/kibana-visualizations
 x-pack/plugins/watcher @elastic/platform-deployment-management
 packages/kbn-web-worker-stub @elastic/kibana-operations
 packages/kbn-whereis-pkg-cli @elastic/kibana-operations
+packages/kbn-xstate-utils @elastic/infra-monitoring-ui
 packages/kbn-yarn-lock-validator @elastic/kibana-operations
 ####
 ## Everything below this line overrides the default assignments for each package.

.i18nrc.json

@@ -18,6 +18,7 @@
       "packages/core"
     ],
     "customIntegrations": "src/plugins/custom_integrations",
+    "customIntegrationsPackage": "packages/kbn-custom-integrations",
     "dashboard": "src/plugins/dashboard",
     "domDragDrop": "packages/kbn-dom-drag-drop",
     "controls": "src/plugins/controls",

package.json

@@ -347,6 +347,7 @@
     "@kbn/crypto": "link:packages/kbn-crypto",
     "@kbn/crypto-browser": "link:packages/kbn-crypto-browser",
     "@kbn/custom-branding-plugin": "link:x-pack/plugins/custom_branding",
+    "@kbn/custom-integrations": "link:packages/kbn-custom-integrations",
     "@kbn/custom-integrations-plugin": "link:src/plugins/custom_integrations",
     "@kbn/dashboard-enhanced-plugin": "link:x-pack/plugins/dashboard_enhanced",
     "@kbn/dashboard-plugin": "link:src/plugins/dashboard",
@@ -795,6 +796,7 @@
     "@kbn/visualization-ui-components": "link:packages/kbn-visualization-ui-components",
     "@kbn/visualizations-plugin": "link:src/plugins/visualizations",
     "@kbn/watcher-plugin": "link:x-pack/plugins/watcher",
+    "@kbn/xstate-utils": "link:packages/kbn-xstate-utils",
     "@loaders.gl/core": "^3.4.7",
     "@loaders.gl/json": "^3.4.7",
     "@loaders.gl/shapefile": "^3.4.7",

packages/kbn-custom-integrations/README.md

@@ -0,0 +1,90 @@
+# Custom integrations package
+
+This package provides UI components and state machines to assist with the creation (and in the future other operations) of custom integrations. For consumers the process *should* be as simple as dropping in the provider and connected components.
+
+## Basic / quickstart usage
+
+1. Add provider
+
+```ts
+<CustomIntegrationsProvider
+    services={{ http }}
+    onIntegrationCreation={onIntegrationCreation}
+    initialState={{
+        mode: 'create',
+        fields: {
+            integrationName,
+            datasets: [{ name: datasetName, type: 'logs' as const }],
+        },
+        previouslyCreatedIntegration: lastCreatedIntegrationOptions,
+    }}
+>
+    <ConfigureLogsContent />
+</CustomIntegrationsProvider>
+```
+
+2. Include Connected form and button components
+
+```ts
+ <ConnectedCustomIntegrationsForm />
+ ```
+
+ The form will internally interact with the backing state machines.
+
+```ts
+<ConnectedCustomIntegrationsButton
+    isDisabled={logFilePathNotConfigured || !namespace}
+    onClick={onContinue}
+/>
+```
+
+Most props are optional, here for example you may conditionally add an extra set of `isDisabled` conditions. They will be applied on top of the internal state machine conditions that ensure the button is disabled when necessary. TypeScript types can be checked for available options.
+
+## Initial state
+
+Initial state is just that, initial state, and isn't "reactive".
+
+## Provider callbacks
+
+The provider accepts some callbacks, for example `onIntegrationCreation`. Changes to these references are tracked internally, so feel free to have a callback handler that changes it's identity if needed.
+
+An example handler:
+
+```ts
+const onIntegrationCreation: OnIntegrationCreationCallback = (
+    integrationOptions
+  ) => {
+    const {
+      integrationName: createdIntegrationName,
+      datasets: createdDatasets,
+    } = integrationOptions;
+
+    setState((state) => ({
+      ...state,
+      integrationName: createdIntegrationName,
+      datasetName: createdDatasets[0].name,
+      lastCreatedIntegrationOptions: integrationOptions,
+    }));
+    goToStep('installElasticAgent');
+  };
+```
+
+## Manual dispatching of events
+
+Sometimes you may have a flow where it is necessary to manually update the internal state machines and bypass the connected components. This is discouraged, but it is possible for some operations. These events are exposed as `DispatchableEvents`, and these are exposed by the `useConsumerCustomIntegrations()` hook.
+
+For example `updateCreateFields` will update the fields of the creation form in the same manner as the UI components would. 
+
+These functions will either exist, or be `undefined`, the presence of these functions means that the corresponding state checks against the machine have already passed. For instance, `saveCreateFields()` will only exist (and not be `undefined`) when the creation form is valid. These functions therefore also fulfill the role of condition checking if needed.
+
+Example usage:
+
+```ts
+const {
+  dispatchableEvents: { updateCreateFields },
+} = useConsumerCustomIntegrations();
+```
+
+## Cleanup
+
+- For the create flow the machine will try to cleanup a previously created integration if needed (if `options.deletePrevious` is `true`). For example, imagine a wizard flow where someone has navigated forward, then navigates back, makes a change, and saves again, the machine will attempt to delete the previously created integration so that lots of rogue custom integrations aren't left behind. The provider accepts an optional `previouslyCreatedIntegration` prop that can serve as initial state.

packages/kbn-custom-integrations/index.ts

@@ -0,0 +1,19 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+export {
+  ConnectedCustomIntegrationsForm,
+  ConnectedCustomIntegrationsButton,
+} from './src/components';
+export { useConsumerCustomIntegrations, useCustomIntegrations } from './src/hooks';
+export { CustomIntegrationsProvider } from './src/state_machines';
+
+// Types
+export type { DispatchableEvents } from './src/hooks';
+export type { Callbacks, InitialState } from './src/state_machines';
+export type { CustomIntegrationOptions } from './src/types';

packages/kbn-custom-integrations/jest.config.js

@@ -0,0 +1,13 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+module.exports = {
+  preset: '@kbn/test/jest_node',
+  rootDir: '../..',
+  roots: ['<rootDir>/packages/kbn-custom-integrations'],
+};

packages/kbn-custom-integrations/kibana.jsonc

@@ -0,0 +1,5 @@
+{
+  "type": "shared-common",
+  "id": "@kbn/custom-integrations",
+  "owner": "@elastic/infra-monitoring-ui"
+}

packages/kbn-custom-integrations/package.json

@@ -0,0 +1,6 @@
+{
+  "name": "@kbn/custom-integrations",
+  "private": true,
+  "version": "1.0.0",
+  "license": "SSPL-1.0 OR Elastic License 2.0"
+}

packages/kbn-custom-integrations/src/components/create/button.tsx

@@ -0,0 +1,91 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { EuiButton } from '@elastic/eui';
+import { i18n } from '@kbn/i18n';
+import { useActor, useSelector } from '@xstate/react';
+import React, { useCallback } from 'react';
+import { isSubmittingSelector, isValidSelector } from '../../state_machines/create/selectors';
+import { CreateCustomIntegrationActorRef } from '../../state_machines/create/state_machine';
+
+const SUBMITTING_TEXT = i18n.translate('customIntegrationsPackage.create.button.submitting', {
+  defaultMessage: 'Creating integration...',
+});
+
+const CONTINUE_TEXT = i18n.translate('customIntegrationsPackage.create.button.continue', {
+  defaultMessage: 'Continue',
+});
+
+interface ConnectedCreateCustomIntegrationButtonProps {
+  machine: CreateCustomIntegrationActorRef;
+  isDisabled?: boolean;
+  onClick?: () => void;
+  submittingText?: string;
+  continueText?: string;
+  testSubj: string;
+}
+export const ConnectedCreateCustomIntegrationButton = ({
+  machine,
+  isDisabled = false,
+  onClick: consumerOnClick,
+  submittingText = SUBMITTING_TEXT,
+  continueText = CONTINUE_TEXT,
+  testSubj,
+}: ConnectedCreateCustomIntegrationButtonProps) => {
+  const [, send] = useActor(machine);
+
+  const onClick = useCallback(() => {
+    if (consumerOnClick) {
+      consumerOnClick();
+    }
+    send({ type: 'SAVE' });
+  }, [consumerOnClick, send]);
+
+  const isValid = useSelector(machine, isValidSelector);
+  const isSubmitting = useSelector(machine, isSubmittingSelector);
+
+  return (
+    <CreateCustomIntegrationButton
+      onClick={onClick}
+      isValid={isValid}
+      isSubmitting={isSubmitting}
+      isDisabled={isDisabled}
+      submittingText={submittingText}
+      continueText={continueText}
+      testSubj={testSubj}
+    />
+  );
+};
+
+type CreateCustomIntegrationButtonProps = {
+  isValid: boolean;
+  isSubmitting: boolean;
+} & Omit<ConnectedCreateCustomIntegrationButtonProps, 'machine'>;
+
+const CreateCustomIntegrationButton = ({
+  onClick,
+  isValid,
+  isSubmitting,
+  isDisabled,
+  submittingText,
+  continueText,
+  testSubj,
+}: CreateCustomIntegrationButtonProps) => {
+  return (
+    <EuiButton
+      data-test-subj={testSubj}
+      color="primary"
+      fill
+      onClick={onClick}
+      isLoading={isSubmitting}
+      isDisabled={isDisabled || !isValid}
+    >
+      {isSubmitting ? submittingText : continueText}
+    </EuiButton>
+  );
+};

packages/kbn-custom-integrations/src/components/create/error_callout.tsx

@@ -0,0 +1,94 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import React from 'react';
+import { i18n } from '@kbn/i18n';
+import { EuiButton, EuiCallOut } from '@elastic/eui';
+import {
+  AuthorizationError,
+  IntegrationError,
+  IntegrationNotInstalledError,
+  UnknownError,
+} from '../../types';
+import { CreateTestSubjects } from './form';
+
+const TITLE = i18n.translate('customIntegrationsPackage.create.errorCallout.title', {
+  defaultMessage: 'Sorry, there was an error',
+});
+
+const RETRY_TEXT = i18n.translate('customIntegrationsPackage.create.errorCallout.retryText', {
+  defaultMessage: 'Retry',
+});
+
+export const ErrorCallout = ({
+  error,
+  onRetry,
+  testSubjects,
+}: {
+  error: IntegrationError;
+  onRetry?: () => void;
+  testSubjects?: CreateTestSubjects['errorCallout'];
+}) => {
+  if (error instanceof AuthorizationError) {
+    const authorizationDescription = i18n.translate(
+      'customIntegrationsPackage.create.errorCallout.authorization.description',
+      {
+        defaultMessage: 'This user does not have permissions to create an integration.',
+      }
+    );
+    return (
+      <BaseErrorCallout
+        message={authorizationDescription}
+        onRetry={onRetry}
+        testSubjects={testSubjects}
+      />
+    );
+  } else if (error instanceof UnknownError || error instanceof IntegrationNotInstalledError) {
+    return (
+      <BaseErrorCallout message={error.message} onRetry={onRetry} testSubjects={testSubjects} />
+    );
+  } else {
+    return null;
+  }
+};
+
+const BaseErrorCallout = ({
+  message,
+  onRetry,
+  testSubjects,
+}: {
+  message: string;
+  onRetry?: () => void;
+  testSubjects?: CreateTestSubjects['errorCallout'];
+}) => {
+  return (
+    <EuiCallOut
+      title={TITLE}
+      color="danger"
+      iconType="error"
+      data-test-subj={testSubjects?.callout ?? 'customIntegrationsPackageCreateFormErrorCallout'}
+    >
+      <>
+        <p>{message}</p>
+        {onRetry ? (
+          <EuiButton
+            data-test-subj={
+              testSubjects?.retryButton ??
+              'customIntegrationsPackageCreateFormErrorCalloutRetryButton'
+            }
+            color="danger"
+            size="s"
+            onClick={onRetry}
+          >
+            {RETRY_TEXT}
+          </EuiButton>
+        ) : null}
+      </>
+    </EuiCallOut>
+  );
+};