feat(cli): improved setup & allow workspace in base url & refactor wo…

…rkspaces/remotes to unify (windmill-labs#966)

* Setup V2 & Allow Workspace in base url

* Handle login conflict information

* Rework workspace & remote logic

* Add login logic

* Add token storage logic

* 🚀 finish refactor

* :Fix Pull

* Remove setup

* Add create-token

* Remove legacy typesc

* Fix change

* Fix warns

* fix warning

* Update README

* Switch to new workspace by default

* Update demo video

* Update Images

* remove duplicate

* Change wording

* Add to main README

* Fix main readme

* Fix videos

README.md

@@ -22,8 +22,7 @@ Open-source developer infrastructure for internal tools. Self-hostable alternati
 
 ---
 
-**Try it (personal workspaces are free forever)**:
-<https://app.windmill.dev>
+**Try it (personal workspaces are free forever)**: <https://app.windmill.dev>
 
 **Documentation**: <https://docs.windmill.dev>
 
@@ -37,7 +36,8 @@ Open-source developer infrastructure for internal tools. Self-hostable alternati
 
 You can show your support for the project by starring this repo.
 
-Windmill Labs offers commercial licenses, an enterprise edition, local hub mirrors, and support: contact [email protected].
+Windmill Labs offers commercial licenses, an enterprise edition, local hub
+mirrors, and support: contact [email protected].
 
 ---
 
@@ -49,6 +49,7 @@ Windmill Labs offers commercial licenses, an enterprise edition, local hub mirro
 
 ![Windmill Screenshot](./imgs/windmill-flow.png)
 ![Windmill Screenshot](./imgs/windmill.png)
+![CLI Screencast](./cli/vhs/output/setup.gif)
 
 Windmill is <b>fully open-sourced (AGPLv3)</b>:
 
@@ -78,23 +79,23 @@ Windmill is <b>fully open-sourced (AGPLv3)</b>:
 
 ## Main Concepts
 
-1. Define a minimal and generic script in Python, Typescript, Go or Bash that solves a
-   specific task. Here sending an email with SMTP. The code can be defined in
-   the provided Web IDE or synchronized with your own github repo:
+1. Define a minimal and generic script in Python, Typescript, Go or Bash that
+   solves a specific task. Here sending an email with SMTP. The code can be
+   defined in the provided Web IDE or synchronized with your own github repo:
    ![Step 1](./imgs/windmill-editor.png)
 
-2. Your scripts parameters are automatically parsed and generate a frontend. 
-   ![Step 2](./imgs/windmill-run.png)
-   ![Step 3](./imgs/windmill-result.png)
+2. Your scripts parameters are automatically parsed and generate a frontend.
+   ![Step 2](./imgs/windmill-run.png) ![Step 3](./imgs/windmill-result.png)
 
 3. Make it flow! You can chain your scripts or scripts made by the community
-   shared on [WindmillHub](https://hub.windmill.dev). 
-  ![Step 4](./imgs/windmill-flow.png)
+   shared on [WindmillHub](https://hub.windmill.dev).
+   ![Step 4](./imgs/windmill-flow.png)
 
-4. (Coming soon) Build complex UI on top of your scripts and flows. 
-  ![Step 5](./imgs/windmill-builder.png)
+4. (Coming soon) Build complex UI on top of your scripts and flows.
+   ![Step 5](./imgs/windmill-builder.png)
 
-Scripts and flows can also be triggered by a cron schedule '*/5 * * * *' or through webhooks.
+Scripts and flows can also be triggered by a cron schedule '*/5 * * * *' or
+through webhooks.
 
 You can build your entire infra on top of Windmill!
 
@@ -148,7 +149,11 @@ That is what we do at <https://app.windmill.dev>.
 
 ## Performance
 
-Once a job started, there is no overhead compared to running the same script on the node with its corresponding runner (Deno/Go/Python/Bash). The added latency from a job being pulled from the queue, started, and then having its result sent back to the database is ~50ms. A typical lightweight deno job will take around 100ms total.
+Once a job started, there is no overhead compared to running the same script on
+the node with its corresponding runner (Deno/Go/Python/Bash). The added latency
+from a job being pulled from the queue, started, and then having its result sent
+back to the database is ~50ms. A typical lightweight deno job will take around
+100ms total.
 
 ## Architecture
 
@@ -200,8 +205,8 @@ be AGPLv3 or you must get a commercial license. Contact us at
 
 In addition, a commercial license grants you a dedicated engineer to transition
 your current infrastructure to Windmill, support with tight SLA, audit logs
-export features, SSO, unlimited users creation, advanced permission managing features
-such as groups and the ability to create more than one workspace.
+export features, SSO, unlimited users creation, advanced permission managing
+features such as groups and the ability to create more than one workspace.
 
 ### OAuth for self-hosting (very optional)
 
@@ -252,7 +257,6 @@ You may also add your own custom OAuth2 IdP and OAuth2 Resource provider:
 }
 ```
 
-
 ### Resource types
 
 You will also want to import all the approved resource types from
@@ -265,7 +269,7 @@ upcoming CLI tool.
 | Environment Variable name | Default                | Description                                                                                                                                                                                        | Api Server/Worker/All |
 | ------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
 | DATABASE_URL              |                        | The Postgres database url.                                                                                                                                                                         | All                   |
-| DISABLE_NSJAIL            | true                   | Disable Nsjail Sandboxing                                                                                                                                                                          |                       | Worker |
+| DISABLE_NSJAIL            | true                   | Disable Nsjail Sandboxing                                                                                                                                                                          |                       |
 | NUM_WORKERS               | 3                      | The number of worker per Worker instance (set to 1 on Eks to have 1 pod = 1 worker)                                                                                                                | Worker                |
 | METRICS_ADDR              | None                   | The socket addr at which to expose Prometheus metrics at the /metrics path. Set to "true" to expose it on port 8001                                                                                | All                   |
 | JSON_FMT                  | false                  | Output the logs in json format instead of logfmt                                                                                                                                                   | All                   |
@@ -285,11 +289,10 @@ upcoming CLI tool.
 | PYTHON_PATH               | /usr/local/bin/python3 | The path to the python binary.                                                                                                                                                                     | Worker                |
 | GO_PATH                   | /usr/bin/go            | The path to the go binary.                                                                                                                                                                         | Worker                |
 | PIP_INDEX_URL             | None                   | The index url to pass for pip.                                                                                                                                                                     | Worker                |
-| PIP_EXTRA_INDEX_URL       | None                   | The extra index url to  pass to pip.                                                                                                                                                               | Worker                |
+| PIP_EXTRA_INDEX_URL       | None                   | The extra index url to pass to pip.                                                                                                                                                                | Worker                |
 | PIP_TRUSTED_HOST          | None                   | The trusted host to pass to pip.                                                                                                                                                                   | Worker                |
 | PATH                      | None                   | The path environment variable, usually inherited                                                                                                                                                   | Worker                |
-| HOME                      | None                   | The home directory to use for Go and Bash                                                                                        , usually inherited                                               | Worker                |
-
+| HOME                      | None                   | The home directory to use for Go and Bash , usually inherited                                                                                                                                      | Worker                |
 
 ## Run a local dev setup
 
@@ -310,9 +313,9 @@ See the [./frontend/README_DEV.md](./frontend/README_DEV.md) file for all
 running options.
 
 1. Create a Postgres Database for Windmill and create an admin role inside your
-   Postgres setup.
-   The easiest way to get a working postgres is running `cargo install sqlx-cli && sqlx migrate run`.
-   This will also avoid compile time issue with sqlx's `query!` macro
+   Postgres setup. The easiest way to get a working postgres is running
+   `cargo install sqlx-cli && sqlx migrate run`. This will also avoid compile
+   time issue with sqlx's `query!` macro
 2. Install [nsjail](https://github.com/google/nsjail) and have it accessible in
    your PATH
 3. Install deno and python3, have the bins at `/usr/bin/deno` and
@@ -321,7 +324,8 @@ running options.
 5. Install the [lld linker](https://lld.llvm.org/)
 6. Go to `frontend/`:
    1. `npm install`, `npm run generate-backend-client` then `npm run dev`
-   2. In another shell `npm run build` otherwise the backend will not find the `frontend/build` folder and will crash
+   2. In another shell `npm run build` otherwise the backend will not find the
+      `frontend/build` folder and will crash
    3. In another shell `sudo caddy run --config Caddyfile`
 7. Go to `backend/`:
    `DATABASE_URL=<DATABASE_URL_TO_YOUR_WINDMILL_DB> RUST_LOG=info cargo run`

backend/windmill-api/src/workspaces.rs

@@ -26,7 +26,7 @@ use windmill_common::{
     flows::Flow,
     scripts::{Schema, Script, ScriptLang},
     utils::{paginate, rd_string, require_admin, Pagination},
-    variables::ListableVariable,
+    variables::ExportableListableVariable,
 };
 
 use hyper::{header, StatusCode};
@@ -649,7 +649,7 @@ async fn tarball_workspace(
     }
 
     {
-        let variables = sqlx::query_as::<_, ListableVariable>(
+        let variables = sqlx::query_as::<_, ExportableListableVariable>(
             "SELECT *, false as is_expired FROM variable WHERE workspace_id = $1 AND is_secret = false",
         )
         .bind(&w_id)

backend/windmill-common/src/variables.rs

@@ -33,6 +33,22 @@ pub struct ListableVariable {
     pub is_linked: Option<bool>,
 }
 
+#[derive(Serialize, Deserialize)]
+#[cfg_attr(feature = "sqlx", derive(sqlx::FromRow))]
+
+pub struct ExportableListableVariable {
+    pub workspace_id: String,
+    pub path: String,
+    pub value: Option<String>,
+    pub is_secret: bool,
+    pub description: String,
+    pub extra_perms: serde_json::Value,
+    pub account: Option<i32>,
+    pub is_oauth: Option<bool>,
+    pub is_expired: Option<bool>,
+    pub is_linked: Option<bool>,
+}
+
 #[derive(Deserialize)]
 pub struct CreateVariable {
     pub path: String,

cli/README.md

@@ -5,64 +5,61 @@ A simple CLI allowing interactions with windmill from the command line.
 
 ## Installation
 
-Install the `wmill` CLI tool using `deno install --unstable -A https://deno.land/x/wmill/main.ts`.
+Install the `wmill` CLI tool using
+`deno install --unstable -A https://deno.land/x/wmill/main.ts`.
 
 Update to the latest version using `wmill upgrade`.
 
-## Setup
+## Workspaces
 
-Setup the CLI by running `wmill setup`. This will guide you through the setup process easily.
+To get started run `wmill workspace add` or use the instructions from the
+workspace settings.
 
 ## Running Flows & Scripts
 
-Run a script or flow using `wmill flow/script run u/username/path/to/script` and pass any inputs using `--input/-i <name>=<value>` curl-style syntax using `-i @-` for stdin or `-i @<filename>` is also supported
+Run a script or flow using `wmill flow/script run u/username/path/to/script` and
+pass any inputs using `--input/-i <name>=<value>` curl-style syntax using
+`-i @-` for stdin or `-i @<filename>` is also supported
 
 Flow Steps and Logs will be streamed during execution automatically.
-![](./vhs/output/run-flow.gif)
 
 ## Pushing Resources, Scripts & More
 
-The CLI can push specifications to a windmill instance. See the [examples/](./examples/) folder for formats.
+The CLI can push specifications to a windmill instance. See the
+[examples/](./examples/) folder for formats.
 
 ### Pushing a folder
 
-You can push all files in a folder at once using `wmill push`
-Files MUST be named resource_name.\<type\>.json. They will be pushed to the remote path they are in, for example the file `u/admin/fib/fib.script.json` will be pushed as a script to u/admin/fib/fib.
+You can push all files in a folder at once using `wmill push` Files MUST be
+named resource_name.\<type\>.json. They will be pushed to the remote path they
+are in, for example the file `u/admin/fib/fib.script.json` will be pushed as a
+script to u/admin/fib/fib.
 
 ### Pushing individual files
 
-You can push individual resources using `wmill <type> push <file_name> \<remote_name\>`. This does not require a special folder layout or file name, as this is given at runtime.
+You can push individual resources using
+`wmill <type> push <file_name> \<remote_name\>`. This does not require a special
+folder layout or file name, as this is given at runtime.
 
 ## Listing
 
-All commands support listing by just not providing a subcommand, ie `wmill script` will result in a list of scripts. Some allow additional options, learn about this by specifying `--help`.
+All commands support listing by just not providing a subcommand, ie
+`wmill script` will result in a list of scripts. Some allow additional options,
+learn about this by specifying `--help`.
 
 ## User Management
 
-You can add & remove users via `wmill user add/remove`, and list them using `wmill user`
+You can add & remove users via `wmill user add/remove`, and list them using
+`wmill user`
 
-## Login
+## Pulling
 
-Logging in using `wmill login` or the setup will save a token to your local computer, into `~/.config/windmill/<hash>/token` (or `C:\Users\<username>\AppData\Roaming\windmill\<hash>\token` on windows).
-This is inherently unsafe, so do not log into the CLI on untrusted devices.
-
-## Managing Remotes
-
-Advanced users may use multiple remotes at once, which the CLI supports using `wmill remote`.
-
-Add remotes using `wmill remote add <name> <base_url>` & Remove them using `wmill remote remove <name>`.
-
-You can use a remote by either setting it as default using `wmill remote set-default <name>` or by overriding the remote using `--remote <name>` on any command.
-
-If you don't want to save the URL locally, you can always override the command using `--base-url <base_url>` on any command.
-
-### Login on multiple remotes
-
-You will have to login on each remote, either do this using `wmill login` or override the token/credentials on each command using `--token <token>`/`--username <username> --password <password>`.
+You can pull the entire workspace using `wmill pull`
 
 ## Completion
 
-The CLI comes with completions out of the box via `wmill completions <shell>`. (Via [cliffy](https://cliffy.io/))
+The CLI comes with completions out of the box via `wmill completions <shell>`.
+(Via [cliffy](https://cliffy.io/))
 
 ### Bash
 
@@ -74,7 +71,8 @@ source <(wmill completions bash)
 
 ### Fish
 
-To enable fish completions add the following line to your `~/.config/fish/config.fish`:
+To enable fish completions add the following line to your
+`~/.config/fish/config.fish`:
 
 ```
 source (wmill completions fish | psub)

cli/context.ts

@@ -1,13 +1,13 @@
+// deno-lint-ignore-file no-explicit-any
 import { colors } from "https://deno.land/x/[email protected]/ansi/colors.ts";
-import {
-  setClient,
-  UserService,
-} from "https://deno.land/x/[email protected]/mod.ts";
-import { getToken } from "./login.ts";
-import { getDefaultRemote, getRemote } from "./remote.ts";
-import { getStore } from "./store.ts";
+import { setClient } from "https://deno.land/x/[email protected]/mod.ts";
+import { tryGetLoginInfo } from "./login.ts";
 import { GlobalOptions } from "./types.ts";
-import { getDefaultWorkspaceId } from "./workspace.ts";
+import {
+  getActiveWorkspace,
+  getWorkspaceByName,
+  Workspace,
+} from "./workspace.ts";
 
 export type Context = {
   workspace: string;
@@ -16,36 +16,38 @@ export type Context = {
   token: string;
 };
 
-export async function getContext({
-  baseUrl,
-  remote,
-  workspace,
-  token,
-  email,
-  password,
-}: GlobalOptions): Promise<Context> {
-  if (remote) {
-    baseUrl = baseUrl ?? (await getRemote(remote))?.baseUrl;
+export async function resolveWorkspace(
+  opts: GlobalOptions,
+): Promise<Workspace> {
+  const cache = (opts as any).__secret_workspace;
+  if (cache) return cache;
+
+  if (opts.workspace) {
+    const e = await getWorkspaceByName(opts.workspace);
+    if (!e) {
+      console.log(colors.red.underline("Given workspace does not exist."));
+      return Deno.exit(-1);
+    }
+    (opts as any).__secret_workspace = e;
+    return e;
   }
-  baseUrl = baseUrl ?? (await getDefaultRemote())?.baseUrl;
-  baseUrl = baseUrl ?? "https://app.windmill.dev";
-  if (email && password) {
-    setClient("no-token", baseUrl);
-    token =
-      token ?? (await UserService.login({ requestBody: { email, password } }));
+
+  const defaultWorkspace = await getActiveWorkspace(opts);
+  if (!defaultWorkspace) {
+    console.log(colors.red.underline("No workspace given and no default set."));
+    return Deno.exit(-3);
   }
-  token = token ?? (await getToken(baseUrl));
-  setClient(token, baseUrl);
-  const urlStore = await getStore(baseUrl);
-  const workspaceId = workspace ?? (await getDefaultWorkspaceId(urlStore));
-  if (!workspaceId) {
-    console.log(colors.red("No default workspace set and no override given."));
-    Deno.exit(-2);
+
+  return defaultWorkspace;
+}
+
+export async function requireLogin(opts: GlobalOptions) {
+  const workspace = await resolveWorkspace(opts);
+
+  let token = await tryGetLoginInfo(opts);
+  if (!token) {
+    token = workspace.token;
   }
-  return {
-    workspace: workspaceId,
-    baseUrl: baseUrl,
-    urlStore: urlStore,
-    token: token,
-  };
+
+  setClient(token, workspace.remote.substring(0, workspace.remote.length - 1));
 }