Skip to content

Commit

Permalink
feat: add npm_config_registry support for bun, deno and being settabl…
Browse files Browse the repository at this point in the history 
…e from UI (windmill-labs#2373)

* foo

* foo

* npm config
  • Loading branch information
@rubenfiszel
rubenfiszel authored Oct 2, 2023
1 parent fc9adbe commit c42b875
Show file tree
Hide file tree
Showing 15 changed files with 228 additions and 127 deletions.
81 changes: 5 additions & 76 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -226,11 +226,9 @@ From there, you can follow the setup app and create other users.
We publish helm charts at:
<https://github.com/windmill-labs/windmill-helm-charts>.

### Postgres without superuser
### OAuth, SSO & SMTP

If you do not want, or cannot (for instance, in AWS Aurora or Cloud sql) use a
postgres superuser, you can run `./init-db-as-superuser.sql` to init the
required users for Windmill.
Windmill Community Edition allows to configure the OAuth, SSO (including Google Workspace SSO, Microsoft/Azure and Okta) directly from the UI in the superadmin settings. Do note that there is a limit of 50 SSO users on the community edition.

### Commercial license

Expand All @@ -249,76 +247,6 @@ your current infrastructure to Windmill, support with tight SLA, and our global
cache sync for high-performance/no dependency cache miss of cluster from 10+
nodes to 200+ nodes.

### OAuth for self-hosting

To get the same oauth integrations as Windmill Cloud, mount `oauth.json` with
the following format:

```json
{
"<client>": {
"id": "<CLIENT_ID>",
"secret": "<CLIENT_SECRET>",
"allowed_domains": ["windmill.dev"] //restrict a client OAuth login to some domains
}
}
```

and mount it at `/usr/src/app/oauth.json`.

The redirect url for the oauth clients is:
`<instance_url>/user/login_callback/<client>`

Even if you setup oauth, you will still want to **login as [email protected] /
changeme** to setup your instance as a super-admin and give yourself admin
rights.

[The list of all possible "connect an app" oauth clients](https://github.com/windmill-labs/windmill/blob/main/backend/oauth_connect.json)

To add more "connect an app" OAuth clients to the Windmill project, read the
[Contributor's guide](https://www.windmill.dev/docs/misc/contributing). We
welcome contributions!

You may also add your own custom OAuth2 IdP and OAuth2 Resource provider:

```json
{
"<client>": {
"id": "<CLIENT_ID>",
"secret": "<CLIENT_SECRET>",
// To add a new OAuth2 IdP
"login_config": {
"auth_url": "<auth_endpoint>",
"token_url": "<token_endpoint>",
"userinfo_url": "<userinfo endpoint>",
"scopes": ["scope1", "scope2"],
"extra_params": "<if_needed>"
},
// To add a new OAuth2 Resource
"connect_config": {
"auth_url": "<auth_endpoint>",
"token_url": "<token_endpoint>",
"scopes": ["scope1", "scope2"],
"extra_params": "<if_needed>"
}
}
}
```

### smtp for self-hosting

For users to receive emails when you invite them to workspaces or add them to
the instances using their emails, configure the SMTP env variables in the
servers:

```
[email protected]
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
[email protected]
SMTP_PASSWORD=yourpasswordapp
```

### Resource types

You will also want to import all the approved resource types from
Expand All @@ -330,7 +258,7 @@ it being synced automatically everyday.
| Environment Variable name | Default | Description | Api Server/Worker/All |
| --------------------------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| DATABASE_URL | | The Postgres database url. | All |
| DISABLE_NSJAIL | true | Disable Nsjail Sandboxing | Worker |
| WORKER_GROUP | default | The worker group the worker belongs to and get its configuration pulled from | Worker |
| SERVER_BIND_ADDR | 0.0.0.0 | IP Address on which to bind listening socket | Server |
| PORT | 8000 | Exposed port | Server |
| NUM_WORKERS | 1 | The number of worker per Worker instance (Set to 0 for API/Server instances, Set to 1 for normal workers, and > 1 for workers dedicated to native jobs) | Worker |
Expand Down Expand Up @@ -395,12 +323,13 @@ it being synced automatically everyday.
| SMTP_USERNAME | None | username for the smtp server to send invite emails | Server |
| SMTP_PASSWORD | None | password for the smtp server to send invite emails | Server |
| SMTP_TLS_IMPLICIT | false | https://docs.rs/mail-send/latest/mail_send/struct.SmtpClientBuilder.html#method.implicit_tlsemails | Server |
| CREATE_WORKSPACE_REQUIRE_SUPERADMIN | true | If true, only superadmin can create workspaces | Server |
| CREATE_WORKSPACE_REQUIRE_SUPERADMIN | true | If true, only superadmin can create workspaces | Server |
| GLOBAL_ERROR_HANDLER_PATH_IN_ADMINS_WORKSPACE | None | Path to a script to run when a root job fails. The script will be run in and from the admins workspace | Server |
| WHITELIST_ENVS | None | List of envs variables, separated by a ',' that are whitelisted as being safe to passthrough the workers | Worker |
| SAML_METADATA | None | SAML Metadata URL to enable SAML SSO (EE only) | Server |
| SECRET_SALT | None | Secret Salt used for encryption and decryption of secrets. If defined, the secrets will not be decryptable unless the right salt is passed in, which is the case for the workers and the server | Server + Worker |
| OPENAI_AZURE_BASE_PATH | None | Azure OpenAI API base path (no trailing slash) | Server |
| DISABLE_NSJAIL | true | Disable Nsjail Sandboxing | Worker |

## Run a local dev setup

Expand Down
14 changes: 14 additions & 0 deletions backend/.sqlx/query-025e5bf6aef56e9832239489daf21b4970d84529cf7ed7928ce2dcfec8b007f0.json
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions backend/migrations/20231002123723_notify_delete_config.down.sql
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
-- Add down migration script here
15 changes: 15 additions & 0 deletions backend/migrations/20231002123723_notify_delete_config.up.sql
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
-- Add up migration script here

CREATE FUNCTION "notify_global_setting_delete" ()
RETURNS TRIGGER AS $$
BEGIN
PERFORM pg_notify('notify_global_setting_change', OLD.name::text);
RETURN OLD;
END;
$$ LANGUAGE PLPGSQL;

CREATE OR REPLACE TRIGGER "notify_global_setting_delete"
AFTER DELETE ON "global_settings"
FOR EACH ROW
EXECUTE FUNCTION "notify_global_setting_delete" ();

27 changes: 14 additions & 13 deletions backend/src/main.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ use tokio::{
};
use windmill_common::{
global_settings::{
BASE_URL_SETTING, CUSTOM_TAGS_SETTING, ENV_SETTINGS, LICENSE_KEY_SETTING, OAUTH_SETTING,
BASE_URL_SETTING, CUSTOM_TAGS_SETTING, ENV_SETTINGS, EXTRA_PIP_INDEX_URL_SETTING,
LICENSE_KEY_SETTING, NPM_CONFIG_REGISTRY_SETTING, OAUTH_SETTING,
REQUEST_SIZE_LIMIT_SETTING, RETENTION_PERIOD_SECS_SETTING,
},
utils::rd_string,
Expand All @@ -36,8 +37,9 @@ use windmill_worker::{
};

use crate::monitor::{
initial_load, monitor_db, reload_base_url_setting, reload_license_key,
reload_retention_period_setting, reload_server_config, reload_worker_config,
initial_load, monitor_db, reload_base_url_setting, reload_extra_pip_index_url_setting,
reload_license_key, reload_npm_config_registry_setting, reload_retention_period_setting,
reload_server_config, reload_worker_config,
};

const GIT_VERSION: &str = git_version!(args = ["--tag", "--always"], fallback = "unknown-version");
Expand Down Expand Up @@ -220,52 +222,51 @@ Windmill Community Edition {GIT_VERSION}
tracing::info!("Received new pg notification: {n:?}");
match n.channel() {
"notify_config_change" => {
tracing::info!("Config change detected");
tracing::info!("Config change detected: {}", n.payload());
match n.payload() {
"server" if server_mode => {
tracing::info!("Server config change detected");
reload_server_config(&db).await;
},
a@ _ if worker_mode && a == format!("worker__{}", *WORKER_GROUP) => {
tracing::info!("Worker config change detected");
reload_worker_config(&db, tx.clone(), true).await;
},
_ => {
()
tracing::error!("config target neither a server or a worker");
}
}
},
"notify_global_setting_change" => {
tracing::info!("Global setting change detected");
tracing::info!("Global setting change detected: {}", n.payload());
match n.payload() {
BASE_URL_SETTING => {
tracing::info!("Base URL setting change detected");
if let Err(e) = reload_base_url_setting(&db).await {
tracing::error!(error = %e, "Could not reload base url setting");
}
},
OAUTH_SETTING => {
tracing::info!("OAuth setting change detected");
if let Err(e) = reload_base_url_setting(&db).await {
tracing::error!(error = %e, "Could not reload oauth setting");
}
},
CUSTOM_TAGS_SETTING => {
tracing::info!("Custom tags setting change detected");
if let Err(e) = reload_custom_tags_setting(&db).await {
tracing::error!(error = %e, "Could not reload custom tags setting");
}
},
LICENSE_KEY_SETTING => {
tracing::info!("License Key setting change detected");
if let Err(e) = reload_license_key(&db).await {
tracing::error!(error = %e, "Could not reload license key setting");
}
},
RETENTION_PERIOD_SECS_SETTING => {
tracing::info!("Retention period setting change detected");
reload_retention_period_setting(&db).await
},
EXTRA_PIP_INDEX_URL_SETTING => {
reload_extra_pip_index_url_setting(&db).await
},
NPM_CONFIG_REGISTRY_SETTING => {
reload_npm_config_registry_setting(&db).await
},
REQUEST_SIZE_LIMIT_SETTING => {
tracing::info!("Request limit size change detected, killing server expecting to be restarted");
// we wait a bit randomly to avoid having all servers shutdown at same time
Expand Down
86 changes: 83 additions & 3 deletions backend/src/monitor.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ use windmill_api::{
use windmill_common::{
error,
global_settings::{
BASE_URL_SETTING, LICENSE_KEY_SETTING, OAUTH_SETTING, REQUEST_SIZE_LIMIT_SETTING,
BASE_URL_SETTING, EXTRA_PIP_INDEX_URL_SETTING, LICENSE_KEY_SETTING,
NPM_CONFIG_REGISTRY_SETTING, OAUTH_SETTING, REQUEST_SIZE_LIMIT_SETTING,
RETENTION_PERIOD_SECS_SETTING,
},
jobs::{JobKind, QueuedJob},
Expand All @@ -25,7 +26,8 @@ use windmill_common::{
BASE_URL, DB, METRICS_ENABLED,
};
use windmill_worker::{
create_token_for_owner, handle_job_error, AuthedClient, SCRIPT_TOKEN_EXPIRY,
create_token_for_owner, handle_job_error, AuthedClient, NPM_CONFIG_REGISTRY,
PIP_EXTRA_INDEX_URL, SCRIPT_TOKEN_EXPIRY,
};

#[cfg(feature = "enterprise")]
Expand Down Expand Up @@ -118,14 +120,28 @@ pub async fn initial_load(
}
};

let reload_extra_pip_index_url_f = async {
if worker_mode {
reload_extra_pip_index_url_setting(&db).await;
}
};

let reload_npm_config_registry_f = async {
if worker_mode {
reload_npm_config_registry_setting(&db).await;
}
};

join!(
reload_worker_config_f,
reload_server_config_f,
reload_custom_tags_f,
reload_request_size_f,
reload_base_url_f,
reload_retention_period_f,
reload_license_key_f
reload_license_key_f,
reload_extra_pip_index_url_f,
reload_npm_config_registry_f
);
}

Expand Down Expand Up @@ -201,6 +217,32 @@ pub async fn delete_expired_items(db: &DB) -> () {
}
}

pub async fn reload_extra_pip_index_url_setting(db: &DB) {
if let Err(e) = reload_option_string_setting(
db,
EXTRA_PIP_INDEX_URL_SETTING,
"PIP_EXTRA_INDEX_URL",
PIP_EXTRA_INDEX_URL.clone(),
)
.await
{
tracing::error!("Error reloading extra_pip_index_url period: {:?}", e)
}
}

pub async fn reload_npm_config_registry_setting(db: &DB) {
if let Err(e) = reload_option_string_setting(
db,
NPM_CONFIG_REGISTRY_SETTING,
"NPM_CONFIG_REGISTRY",
NPM_CONFIG_REGISTRY.clone(),
)
.await
{
tracing::error!("Error reloading npm_config_registry period: {:?}", e)
}
}

pub async fn reload_retention_period_setting(db: &DB) {
if let Err(e) = reload_setting(
db,
Expand Down Expand Up @@ -261,6 +303,44 @@ pub async fn reload_license_key(db: &DB) -> error::Result<()> {
Ok(())
}

pub async fn reload_option_string_setting(
db: &DB,
setting_name: &str,
std_env_var: &str,
lock: Arc<RwLock<Option<String>>>,
) -> error::Result<()> {
let q = sqlx::query!(
"SELECT value FROM global_settings WHERE name = $1",
setting_name
)
.fetch_optional(db)
.await?;

let mut value = std::env::var(std_env_var).ok();

if let Some(q) = q {
if let Ok(v) = serde_json::from_value::<String>(q.value.clone()) {
tracing::info!(
"Loaded setting {setting_name} from db config: {:#?}",
&q.value
);
value = Some(v)
} else {
tracing::error!("Could not parse {setting_name} found: {:#?}", &q.value);
}
};

{
if value.is_none() {
tracing::info!("Loaded {setting_name} setting to None");
}
let mut l = lock.write().await;
*l = value;
}

Ok(())
}

pub async fn reload_setting<T: FromStr + DeserializeOwned + Display>(
db: &DB,
setting_name: &str,
Expand Down
Loading

0 comments on commit c42b875

Please sign in to comment.