-
+[](https://listmonk.app)
-listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a PostgreSQL database as its data store.
+listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a PostgreSQL (⩾ 12) database as its data store.
-[](https://listmonk.app)
-Visit [listmonk.app](https://listmonk.app)
+[](https://listmonk.app)
+
+Visit [listmonk.app](https://listmonk.app) for more info. Check out the [**live demo**](https://demo.listmonk.app).
## Installation
### Docker
-The latest image is available on DockerHub at `listmonk/listmonk:latest`. Use the sample [docker-compose.yml](https://github.com/knadh/listmonk/blob/master/docker-compose.yml) to run listmonk and Postgres DB with docker-compose as follows:
-
-#### Demo
-
-```bash
-mkdir listmonk-demo
-sh -c "$(curl -fsSL https://raw.githubusercontent.com/knadh/listmonk/master/install-demo.sh)"
-```
+The latest image is available on DockerHub at [`listmonk/listmonk:latest`](https://hub.docker.com/r/listmonk/listmonk/tags?page=1&ordering=last_updated&name=latest).
+Download and use the sample [docker-compose.yml](https://github.com/knadh/listmonk/blob/master/docker-compose.yml).
-The demo does not persist Postgres after the containers are removed. DO NOT use this demo setup in production.
-#### Production
+```shell
+# Download the compose file to the current directory.
+curl -LO https://github.com/knadh/listmonk/raw/master/docker-compose.yml
-##### Easy Docker install
-
-This setup is recommended if you want to _quickly_ setup `listmonk` in production.
-
-```bash
-mkdir listmonk
-sh -c "$(curl -fsSL https://raw.githubusercontent.com/knadh/listmonk/master/install-prod.sh)"
+# Run the services in the background.
+docker compose up -d
```
+Visit `http://localhost:9000`
-The above shell script performs the following actions:
-
-- Downloads `docker-compose.yml` and generates a `config.toml`.
-- Runs a Postgres container and installs the database schema.
-- Runs the `listmonk` container.
-
-**NOTE**: It's recommended to examine the contents of the shell script, before running in your environment.
-
-##### Manual Docker install
-
-The following workflow is recommended to setup `listmonk` manually using `docker-compose`. You are encouraged to customise the contents of `docker-compose.yml` to your needs. The overall setup looks like:
-
-- `docker-compose up db` to run the Postgres DB.
-- `docker-compose run --rm app ./listmonk --install` to setup the DB (or `--upgrade` to upgrade an existing DB).
-- Copy `config.toml.sample` to your directory and make the following changes:
- - `app.address` => `0.0.0.0:9000` (Port forwarding on Docker will work only if the app is advertising on all interfaces.)
- - `db.host` => `listmonk_db` (Container Name of the DB container)
-- Run `docker-compose up app` and visit `http://localhost:9000`.
-
-More information on [docs](https://listmonk.app/docs).
+See [installation docs](https://listmonk.app/docs/installation)
__________________
### Binary
- Download the [latest release](https://github.com/knadh/listmonk/releases) and extract the listmonk binary.
-- `./listmonk --new-config` to generate config.toml. Then, edit the file.
+- `./listmonk --new-config` to generate config.toml. Edit it.
- `./listmonk --install` to setup the Postgres DB (or `--upgrade` to upgrade an existing DB. Upgrades are idempotent and running them multiple times have no side effects).
-- Run `./listmonk` and visit `http://localhost:9000`.
+- Run `./listmonk` and visit `http://localhost:9000`
+See [installation docs](https://listmonk.app/docs/installation)
__________________
-### Heroku
-[](https://heroku.com/deploy?template=https://github.com/knadh/listmonk-heroku)
-
## Developers
-listmonk is a free and open source software licensed under AGPLv3. If you are interested in contributing, refer to the [developer setup](https://listmonk.app/docs/developer-setup). The backend is written in Go and the frontend is Vue with Buefy for UI.
+listmonk is free and open source software licensed under AGPLv3. If you are interested in contributing, refer to the [developer setup](https://listmonk.app/docs/developer-setup). The backend is written in Go and the frontend is Vue with Buefy for UI.
## License
diff --git a/TODO.md b/TODO.md
deleted file mode 100644
index 0e5e53f91..000000000
--- a/TODO.md
+++ /dev/null
@@ -1,8 +0,0 @@
-- [ ] Add a "running campaigns" widget on the dashboard
-- [ ] Add more analytics and stats
-- [ ] Add bounce tracking
-- [ ] Pause campaigns on % errors in addition to an absolute numbers
-- [ ] Support DB migrations for easy upgrades
-- [ ] Add materialized views for analytics and stats (and more?)
-- [ ] Add user management and permissions
-- [ ] Add tests
diff --git a/cmd/admin.go b/cmd/admin.go
index 618ae19fe..2541dbfb5 100644
--- a/cmd/admin.go
+++ b/cmd/admin.go
@@ -1,31 +1,40 @@
package main
import (
+ "encoding/json"
"fmt"
"net/http"
- "sort"
"syscall"
"time"
- "github.com/jmoiron/sqlx/types"
- "github.com/labstack/echo"
+ "github.com/labstack/echo/v4"
)
type serverConfig struct {
- Messengers []string `json:"messengers"`
- Langs []i18nLang `json:"langs"`
- Lang string `json:"lang"`
- Update *AppUpdate `json:"update"`
- NeedsRestart bool `json:"needs_restart"`
- Version string `json:"version"`
+ RootURL string `json:"root_url"`
+ FromEmail string `json:"from_email"`
+ Messengers []string `json:"messengers"`
+ Langs []i18nLang `json:"langs"`
+ Lang string `json:"lang"`
+ Permissions json.RawMessage `json:"permissions"`
+ Update *AppUpdate `json:"update"`
+ NeedsRestart bool `json:"needs_restart"`
+ HasLegacyUser bool `json:"has_legacy_user"`
+ Version string `json:"version"`
}
// handleGetServerConfig returns general server config.
func handleGetServerConfig(c echo.Context) error {
var (
app = c.Get("app").(*App)
- out = serverConfig{}
)
+ out := serverConfig{
+ RootURL: app.constants.RootURL,
+ FromEmail: app.constants.FromEmail,
+ Lang: app.constants.Lang,
+ Permissions: app.constants.PermissionsRaw,
+ HasLegacyUser: app.constants.HasLegacyUser,
+ }
// Language list.
langList, err := getI18nLangList(app.constants.Lang, app)
@@ -34,19 +43,11 @@ func handleGetServerConfig(c echo.Context) error {
fmt.Sprintf("Error loading language list: %v", err))
}
out.Langs = langList
- out.Lang = app.constants.Lang
- // Sort messenger names with `email` always as the first item.
- var names []string
- for name := range app.messengers {
- if name == emailMsgr {
- continue
- }
- names = append(names, name)
+ out.Messengers = make([]string, 0, len(app.messengers))
+ for _, m := range app.messengers {
+ out.Messengers = append(out.Messengers, m.Name())
}
- sort.Strings(names)
- out.Messengers = append(out.Messengers, emailMsgr)
- out.Messengers = append(out.Messengers, names...)
app.Lock()
out.NeedsRestart = app.needsRestart
@@ -61,12 +62,11 @@ func handleGetServerConfig(c echo.Context) error {
func handleGetDashboardCharts(c echo.Context) error {
var (
app = c.Get("app").(*App)
- out types.JSONText
)
- if err := app.queries.GetDashboardCharts.Get(&out); err != nil {
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching", "name", "dashboard charts", "error", pqErrMsg(err)))
+ out, err := app.core.GetDashboardCharts()
+ if err != nil {
+ return err
}
return c.JSON(http.StatusOK, okResp{out})
@@ -76,12 +76,11 @@ func handleGetDashboardCharts(c echo.Context) error {
func handleGetDashboardCounts(c echo.Context) error {
var (
app = c.Get("app").(*App)
- out types.JSONText
)
- if err := app.queries.GetDashboardCounts.Get(&out); err != nil {
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching", "name", "dashboard stats", "error", pqErrMsg(err)))
+ out, err := app.core.GetDashboardCounts()
+ if err != nil {
+ return err
}
return c.JSON(http.StatusOK, okResp{out})
@@ -92,7 +91,7 @@ func handleReloadApp(c echo.Context) error {
app := c.Get("app").(*App)
go func() {
<-time.After(time.Millisecond * 500)
- app.sigChan <- syscall.SIGHUP
+ app.chReload <- syscall.SIGHUP
}()
return c.JSON(http.StatusOK, okResp{true})
}
diff --git a/cmd/archive.go b/cmd/archive.go
new file mode 100644
index 000000000..173dbf402
--- /dev/null
+++ b/cmd/archive.go
@@ -0,0 +1,276 @@
+package main
+
+import (
+ "bytes"
+ "encoding/json"
+ "html/template"
+ "net/http"
+ "net/url"
+
+ "github.com/gorilla/feeds"
+ "github.com/knadh/listmonk/internal/manager"
+ "github.com/knadh/listmonk/models"
+ "github.com/labstack/echo/v4"
+ null "gopkg.in/volatiletech/null.v6"
+)
+
+type campArchive struct {
+ UUID string `json:"uuid"`
+ Subject string `json:"subject"`
+ Content string `json:"content"`
+ CreatedAt null.Time `json:"created_at"`
+ SendAt null.Time `json:"send_at"`
+ URL string `json:"url"`
+}
+
+// handleGetCampaignArchives renders the public campaign archives page.
+func handleGetCampaignArchives(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ pg = app.paginator.NewFromURL(c.Request().URL.Query())
+ )
+
+ camps, total, err := getCampaignArchives(pg.Offset, pg.Limit, false, app)
+ if err != nil {
+ return err
+ }
+
+ var out models.PageResults
+ if len(camps) == 0 {
+ out.Results = []campArchive{}
+ return c.JSON(http.StatusOK, okResp{out})
+ }
+
+ // Meta.
+ out.Results = camps
+ out.Total = total
+ out.Page = pg.Page
+ out.PerPage = pg.PerPage
+
+ return c.JSON(200, okResp{out})
+}
+
+// handleGetCampaignArchivesFeed renders the public campaign archives RSS feed.
+func handleGetCampaignArchivesFeed(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ pg = app.paginator.NewFromURL(c.Request().URL.Query())
+ showFullContent = app.constants.EnablePublicArchiveRSSContent
+ )
+
+ camps, _, err := getCampaignArchives(pg.Offset, pg.Limit, showFullContent, app)
+ if err != nil {
+ return err
+ }
+
+ out := make([]*feeds.Item, 0, len(camps))
+ for _, c := range camps {
+ pubDate := c.CreatedAt.Time
+
+ if c.SendAt.Valid {
+ pubDate = c.SendAt.Time
+ }
+
+ out = append(out, &feeds.Item{
+ Title: c.Subject,
+ Link: &feeds.Link{Href: c.URL},
+ Content: c.Content,
+ Created: pubDate,
+ })
+ }
+
+ feed := &feeds.Feed{
+ Title: app.constants.SiteName,
+ Link: &feeds.Link{Href: app.constants.RootURL},
+ Description: app.i18n.T("public.archiveTitle"),
+ Items: out,
+ }
+
+ if err := feed.WriteRss(c.Response().Writer); err != nil {
+ app.log.Printf("error generating archive RSS feed: %v", err)
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("public.errorProcessingRequest"))
+ }
+
+ return nil
+}
+
+// handleCampaignArchivesPage renders the public campaign archives page.
+func handleCampaignArchivesPage(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ pg = app.paginator.NewFromURL(c.Request().URL.Query())
+ )
+
+ out, total, err := getCampaignArchives(pg.Offset, pg.Limit, false, app)
+ if err != nil {
+ return err
+ }
+ pg.SetTotal(total)
+
+ title := app.i18n.T("public.archiveTitle")
+ return c.Render(http.StatusOK, "archive", struct {
+ Title string
+ Description string
+ Campaigns []campArchive
+ TotalPages int
+ Pagination template.HTML
+ }{title, title, out, pg.TotalPages, template.HTML(pg.HTML("?page=%d"))})
+}
+
+// handleCampaignArchivePage renders the public campaign archives page.
+func handleCampaignArchivePage(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ id = c.Param("id")
+ uuid = ""
+ slug = ""
+ )
+
+ // ID can be the UUID or slug.
+ if reUUID.MatchString(id) {
+ uuid = id
+ } else {
+ slug = id
+ }
+
+ pubCamp, err := app.core.GetArchivedCampaign(0, uuid, slug)
+ if err != nil || pubCamp.Type != models.CampaignTypeRegular {
+ notFound := false
+ if er, ok := err.(*echo.HTTPError); ok {
+ if er.Code == http.StatusBadRequest {
+ notFound = true
+ }
+ } else if pubCamp.Type != models.CampaignTypeRegular {
+ notFound = true
+ }
+
+ if notFound {
+ return c.Render(http.StatusNotFound, tplMessage,
+ makeMsgTpl(app.i18n.T("public.notFoundTitle"), "", app.i18n.T("public.campaignNotFound")))
+ }
+
+ return c.Render(http.StatusInternalServerError, tplMessage,
+ makeMsgTpl(app.i18n.T("public.errorTitle"), "", app.i18n.Ts("public.errorFetchingCampaign")))
+ }
+
+ out, err := compileArchiveCampaigns([]models.Campaign{pubCamp}, app)
+ if err != nil {
+ return c.Render(http.StatusInternalServerError, tplMessage,
+ makeMsgTpl(app.i18n.T("public.errorTitle"), "", app.i18n.Ts("public.errorFetchingCampaign")))
+ }
+
+ // Render the message body.
+ camp := out[0].Campaign
+ msg, err := app.manager.NewCampaignMessage(camp, out[0].Subscriber)
+ if err != nil {
+ app.log.Printf("error rendering message: %v", err)
+ return c.Render(http.StatusInternalServerError, tplMessage,
+ makeMsgTpl(app.i18n.T("public.errorTitle"), "", app.i18n.Ts("public.errorFetchingCampaign")))
+ }
+
+ return c.HTML(http.StatusOK, string(msg.Body()))
+}
+
+// handleCampaignArchivePageLatest renders the latest public campaign.
+func handleCampaignArchivePageLatest(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ )
+
+ camps, _, err := getCampaignArchives(0, 1, true, app)
+ if err != nil {
+ return err
+ }
+
+ if len(camps) == 0 {
+ return c.Render(http.StatusNotFound, tplMessage,
+ makeMsgTpl(app.i18n.T("public.notFoundTitle"), "", app.i18n.T("public.campaignNotFound")))
+ }
+
+ camp := camps[0]
+
+ return c.HTML(http.StatusOK, camp.Content)
+}
+
+func getCampaignArchives(offset, limit int, renderBody bool, app *App) ([]campArchive, int, error) {
+ pubCamps, total, err := app.core.GetArchivedCampaigns(offset, limit)
+ if err != nil {
+ return []campArchive{}, total, echo.NewHTTPError(http.StatusInternalServerError, app.i18n.T("public.errorFetchingCampaign"))
+ }
+
+ msgs, err := compileArchiveCampaigns(pubCamps, app)
+ if err != nil {
+ return []campArchive{}, total, err
+ }
+
+ out := make([]campArchive, 0, len(msgs))
+ for _, m := range msgs {
+ camp := m.Campaign
+
+ archive := campArchive{
+ UUID: camp.UUID,
+ Subject: camp.Subject,
+ CreatedAt: camp.CreatedAt,
+ SendAt: camp.SendAt,
+ }
+
+ if camp.ArchiveSlug.Valid {
+ archive.URL, _ = url.JoinPath(app.constants.ArchiveURL, camp.ArchiveSlug.String)
+ } else {
+ archive.URL, _ = url.JoinPath(app.constants.ArchiveURL, camp.UUID)
+ }
+
+ if renderBody {
+ msg, err := app.manager.NewCampaignMessage(camp, m.Subscriber)
+ if err != nil {
+ return []campArchive{}, total, err
+ }
+ archive.Content = string(msg.Body())
+ }
+
+ out = append(out, archive)
+ }
+
+ return out, total, nil
+}
+
+func compileArchiveCampaigns(camps []models.Campaign, app *App) ([]manager.CampaignMessage, error) {
+ var (
+ b = bytes.Buffer{}
+ )
+
+ out := make([]manager.CampaignMessage, 0, len(camps))
+ for _, c := range camps {
+ camp := c
+ if err := camp.CompileTemplate(app.manager.TemplateFuncs(&camp)); err != nil {
+ app.log.Printf("error compiling template: %v", err)
+ return nil, echo.NewHTTPError(http.StatusInternalServerError, app.i18n.T("public.errorFetchingCampaign"))
+ }
+
+ // Load the dummy subscriber meta.
+ var sub models.Subscriber
+ if err := json.Unmarshal([]byte(camp.ArchiveMeta), &sub); err != nil {
+ app.log.Printf("error unmarshalling campaign archive meta: %v", err)
+ return nil, echo.NewHTTPError(http.StatusInternalServerError, app.i18n.T("public.errorFetchingCampaign"))
+ }
+
+ m := manager.CampaignMessage{
+ Campaign: &camp,
+ Subscriber: sub,
+ }
+
+ // Render the subject if it's a template.
+ if camp.SubjectTpl != nil {
+ if err := camp.SubjectTpl.ExecuteTemplate(&b, models.ContentTpl, m); err != nil {
+ return nil, err
+ }
+ camp.Subject = b.String()
+ b.Reset()
+
+ }
+
+ out = append(out, m)
+ }
+
+ return out, nil
+}
diff --git a/cmd/auth.go b/cmd/auth.go
new file mode 100644
index 000000000..2d368ba7d
--- /dev/null
+++ b/cmd/auth.go
@@ -0,0 +1,404 @@
+package main
+
+import (
+ "encoding/base64"
+ "encoding/json"
+ "errors"
+ "net/http"
+ "net/mail"
+ "net/url"
+ "strings"
+ "time"
+
+ "github.com/knadh/listmonk/internal/auth"
+ "github.com/knadh/listmonk/internal/utils"
+ "github.com/knadh/listmonk/models"
+ "github.com/labstack/echo/v4"
+ "github.com/zerodha/simplesessions/v3"
+ "gopkg.in/volatiletech/null.v6"
+)
+
+type loginTpl struct {
+ Title string
+ Description string
+
+ NextURI string
+ Nonce string
+ PasswordEnabled bool
+ OIDCProvider string
+ OIDCProviderLogo string
+ Error string
+}
+
+type oidcState struct {
+ Nonce string `json:"nonce"`
+ Next string `json:"next"`
+}
+
+var oidcProviders = map[string]bool{
+ "google.com": true,
+ "microsoftonline.com": true,
+ "auth0.com": true,
+ "github.com": true,
+}
+
+// handleLoginPage renders the login page and handles the login form.
+func handleLoginPage(c echo.Context) error {
+ app := c.Get("app").(*App)
+
+ // Has the user been setup?
+ app.Lock()
+ needsUserSetup := app.needsUserSetup
+ app.Unlock()
+
+ if needsUserSetup {
+ return handleLoginSetupPage(c)
+ }
+
+ // Process POST login request.
+ var loginErr error
+ if c.Request().Method == http.MethodPost {
+ loginErr = doLogin(c)
+ if loginErr == nil {
+ return c.Redirect(http.StatusFound, utils.SanitizeURI(c.FormValue("next")))
+ }
+ }
+
+ return renderLoginPage(c, loginErr)
+}
+
+// handleLoginSetupPage renders the first time user login page and handles the login form.
+func handleLoginSetupPage(c echo.Context) error {
+ app := c.Get("app").(*App)
+
+ // Process POST login request.
+ var loginErr error
+
+ if c.Request().Method == http.MethodPost {
+ loginErr = doLoginSetup(c)
+ if loginErr == nil {
+ app.Lock()
+ app.needsUserSetup = false
+ app.Unlock()
+ return c.Redirect(http.StatusFound, utils.SanitizeURI(c.FormValue("next")))
+ }
+ }
+
+ return renderLoginSetupPage(c, loginErr)
+}
+
+// handleLogout logs a user out.
+func handleLogout(c echo.Context) error {
+ var (
+ sess = c.Get(auth.SessionKey).(*simplesessions.Session)
+ )
+
+ // Clear the session.
+ _ = sess.Destroy()
+
+ return c.JSON(http.StatusOK, okResp{true})
+}
+
+// handleOIDCLogin initializes an OIDC request and redirects to the OIDC provider for login.
+func handleOIDCLogin(c echo.Context) error {
+ app := c.Get("app").(*App)
+
+ // Verify that the request came from the login page (CSRF).
+ nonce, err := c.Cookie("nonce")
+ if err != nil || nonce.Value == "" || nonce.Value != c.FormValue("nonce") {
+ return echo.NewHTTPError(http.StatusUnauthorized, app.i18n.T("users.invalidRequest"))
+ }
+
+ next := utils.SanitizeURI(c.FormValue("next"))
+ if next == "/" {
+ next = uriAdmin
+ }
+
+ state := oidcState{
+ Nonce: nonce.Value,
+ Next: next,
+ }
+
+ stateJSON, err := json.Marshal(state)
+ if err != nil {
+ app.log.Printf("error marshalling OIDC state: %v", err)
+ return echo.NewHTTPError(http.StatusInternalServerError, app.i18n.T("globals.messages.internalError"))
+ }
+
+ return c.Redirect(http.StatusFound, app.auth.GetOIDCAuthURL(base64.URLEncoding.EncodeToString(stateJSON), nonce.Value))
+}
+
+// handleOIDCFinish receives the redirect callback from the OIDC provider and completes the handshake.
+func handleOIDCFinish(c echo.Context) error {
+ app := c.Get("app").(*App)
+
+ nonce, err := c.Cookie("nonce")
+ if err != nil || nonce.Value == "" {
+ return renderLoginPage(c, echo.NewHTTPError(http.StatusUnauthorized, app.i18n.T("users.invalidRequest")))
+ }
+
+ // Validate the OIDC token.
+ oidcToken, claims, err := app.auth.ExchangeOIDCToken(c.Request().URL.Query().Get("code"), nonce.Value)
+ if err != nil {
+ return renderLoginPage(c, err)
+ }
+
+ // Validate the state.
+ var state oidcState
+ stateB, err := base64.URLEncoding.DecodeString(c.QueryParam("state"))
+ if err != nil {
+ app.log.Printf("error decoding OIDC state: %v", err)
+ return echo.NewHTTPError(http.StatusInternalServerError, app.i18n.T("globals.messages.internalError"))
+ }
+ if err := json.Unmarshal(stateB, &state); err != nil {
+ app.log.Printf("error unmarshalling OIDC state: %v", err)
+ return echo.NewHTTPError(http.StatusInternalServerError, app.i18n.T("globals.messages.internalError"))
+ }
+ if state.Nonce != nonce.Value {
+ return renderLoginPage(c, echo.NewHTTPError(http.StatusUnauthorized, app.i18n.T("users.invalidRequest")))
+ }
+
+ // Validate e-mail from the claim.
+ email := strings.TrimSpace(claims.Email)
+ if email == "" {
+ return renderLoginPage(c, errors.New(app.i18n.Ts("globals.messages.invalidFields", "name", "email")))
+ }
+
+ em, err := mail.ParseAddress(email)
+ if err != nil {
+ return renderLoginPage(c, err)
+ }
+ email = strings.ToLower(em.Address)
+
+ // Get the user by e-mail received from OIDC.
+ user, err := app.core.GetUser(0, "", email)
+ if err != nil {
+ return renderLoginPage(c, err)
+ }
+
+ // Update user login.
+ if err := app.core.UpdateUserLogin(user.ID, claims.Picture); err != nil {
+ return renderLoginPage(c, err)
+ }
+
+ // Set the session.
+ if err := app.auth.SaveSession(user, oidcToken, c); err != nil {
+ return renderLoginPage(c, err)
+ }
+
+ return c.Redirect(http.StatusFound, utils.SanitizeURI(state.Next))
+}
+
+// renderLoginPage renders the login page and handles the login form.
+func renderLoginPage(c echo.Context, loginErr error) error {
+ var (
+ app = c.Get("app").(*App)
+ next = utils.SanitizeURI(c.FormValue("next"))
+ )
+
+ if next == "/" {
+ next = uriAdmin
+ }
+
+ oidcProvider := ""
+ oidcProviderLogo := ""
+ if app.constants.Security.OIDC.Enabled {
+ oidcProviderLogo = "oidc.png"
+ u, err := url.Parse(app.constants.Security.OIDC.Provider)
+ if err == nil {
+ h := strings.Split(u.Hostname(), ".")
+
+ // Get the last two h for the root domain
+ if len(h) >= 2 {
+ oidcProvider = h[len(h)-2] + "." + h[len(h)-1]
+ } else {
+ oidcProvider = u.Hostname()
+ }
+
+ if _, ok := oidcProviders[oidcProvider]; ok {
+ oidcProviderLogo = oidcProvider + ".png"
+ }
+ }
+ }
+
+ out := loginTpl{
+ Title: app.i18n.T("users.login"),
+ PasswordEnabled: true,
+ OIDCProvider: oidcProvider,
+ OIDCProviderLogo: oidcProviderLogo,
+ NextURI: next,
+ }
+
+ if loginErr != nil {
+ if e, ok := loginErr.(*echo.HTTPError); ok {
+ out.Error = e.Message.(string)
+ } else {
+ out.Error = loginErr.Error()
+ }
+ }
+
+ // Generate and set a nonce for preventing CSRF requests.
+ nonce, err := utils.GenerateRandomString(16)
+ if err != nil {
+ app.log.Printf("error generating OIDC nonce: %v", err)
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.internalError"))
+ }
+ c.SetCookie(&http.Cookie{
+ Name: "nonce",
+ Value: nonce,
+ HttpOnly: true,
+ Path: "/",
+ SameSite: http.SameSiteLaxMode,
+ })
+ out.Nonce = nonce
+
+ return c.Render(http.StatusOK, "admin-login", out)
+}
+
+// renderLoginSetupPage renders the first time user setup page.
+func renderLoginSetupPage(c echo.Context, loginErr error) error {
+ var (
+ app = c.Get("app").(*App)
+ next = utils.SanitizeURI(c.FormValue("next"))
+ )
+
+ if next == "/" {
+ next = uriAdmin
+ }
+
+ out := loginTpl{
+ Title: app.i18n.T("users.login"),
+ PasswordEnabled: true,
+ NextURI: next,
+ }
+
+ if loginErr != nil {
+ if e, ok := loginErr.(*echo.HTTPError); ok {
+ out.Error = e.Message.(string)
+ } else {
+ out.Error = loginErr.Error()
+ }
+ }
+
+ return c.Render(http.StatusOK, "admin-login-setup", out)
+}
+
+// doLogin logs a user in with a username and password.
+func doLogin(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ )
+
+ // Verify that the request came from the login page (CSRF).
+ // nonce, err := c.Cookie("nonce")
+ // if err != nil || nonce.Value == "" || nonce.Value != c.FormValue("nonce") {
+ // return echo.NewHTTPError(http.StatusUnauthorized, app.i18n.T("users.invalidRequest"))
+ // }
+
+ var (
+ username = strings.TrimSpace(c.FormValue("username"))
+ password = strings.TrimSpace(c.FormValue("password"))
+ )
+
+ if !strHasLen(username, 3, stdInputMaxLen) {
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidFields", "name", "username"))
+ }
+
+ if !strHasLen(password, 8, stdInputMaxLen) {
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidFields", "name", "password"))
+ }
+
+ start := time.Now()
+
+ user, err := app.core.LoginUser(username, password)
+ if err != nil {
+ return err
+ }
+
+ // Resist potential constant-time-comparison attacks with a min response time.
+ if ms := time.Now().Sub(start).Milliseconds(); ms < 100 {
+ time.Sleep(time.Duration(ms))
+ }
+
+ // Set the session.
+ if err := app.auth.SaveSession(user, "", c); err != nil {
+ return err
+ }
+
+ return nil
+}
+
+// doLoginSetup sets a user up for the first time.
+func doLoginSetup(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ )
+
+ // Verify that the request came from the login page (CSRF).
+ // nonce, err := c.Cookie("nonce")
+ // if err != nil || nonce.Value == "" || nonce.Value != c.FormValue("nonce") {
+ // return echo.NewHTTPError(http.StatusUnauthorized, app.i18n.T("users.invalidRequest"))
+ // }
+
+ var (
+ email = strings.TrimSpace(c.FormValue("email"))
+ username = strings.TrimSpace(c.FormValue("username"))
+ password = strings.TrimSpace(c.FormValue("password"))
+ password2 = strings.TrimSpace(c.FormValue("password2"))
+ )
+
+ if !utils.ValidateEmail(email) {
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidFields", "name", "email"))
+ }
+ if !strHasLen(username, 3, stdInputMaxLen) {
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidFields", "name", "username"))
+ }
+ if !strHasLen(password, 8, stdInputMaxLen) {
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidFields", "name", "password"))
+ }
+ if password != password2 {
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("users.passwordMismatch"))
+ }
+
+ // Create the default "Super Admin".
+ r := models.Role{
+ Type: models.RoleTypeUser,
+ Name: null.NewString("Super Admin", true),
+ }
+ for p := range app.constants.Permissions {
+ r.Permissions = append(r.Permissions, p)
+ }
+ role, err := app.core.CreateRole(r)
+ if err != nil {
+ return err
+ }
+
+ // Create the super admin user.
+ u := models.User{
+ Type: models.UserTypeUser,
+ HasPassword: true,
+ PasswordLogin: true,
+ Username: username,
+ Name: username,
+ Password: null.NewString(password, true),
+ Email: null.NewString(email, true),
+ UserRoleID: role.ID,
+ Status: models.UserStatusEnabled,
+ }
+ if _, err := app.core.CreateUser(u); err != nil {
+ return err
+ }
+
+ // Log the user in.
+ user, err := app.core.LoginUser(username, password)
+ if err != nil {
+ return err
+ }
+
+ // Set the session.
+ if err := app.auth.SaveSession(user, "", c); err != nil {
+ return err
+ }
+
+ return nil
+}
diff --git a/cmd/bounce.go b/cmd/bounce.go
index f32ec5091..33b7b5491 100644
--- a/cmd/bounce.go
+++ b/cmd/bounce.go
@@ -2,33 +2,20 @@ package main
import (
"encoding/json"
- "fmt"
- "io/ioutil"
+ "io"
"net/http"
"strconv"
- "strings"
"time"
- "github.com/knadh/listmonk/internal/subimporter"
"github.com/knadh/listmonk/models"
- "github.com/labstack/echo"
- "github.com/lib/pq"
+ "github.com/labstack/echo/v4"
)
-type bouncesWrap struct {
- Results []models.Bounce `json:"results"`
-
- Total int `json:"total"`
- PerPage int `json:"per_page"`
- Page int `json:"page"`
-}
-
// handleGetBounces handles retrieval of bounce records.
func handleGetBounces(c echo.Context) error {
var (
app = c.Get("app").(*App)
- pg = getPagination(c.QueryParams(), 50)
- out bouncesWrap
+ pg = app.paginator.NewFromURL(c.Request().URL.Query())
id, _ = strconv.Atoi(c.Param("id"))
campID, _ = strconv.Atoi(c.QueryParam("campaign_id"))
@@ -37,38 +24,30 @@ func handleGetBounces(c echo.Context) error {
order = c.FormValue("order")
)
- // Fetch one list.
- single := false
+ // Fetch one bounce.
if id > 0 {
- single = true
+ out, err := app.core.GetBounce(id)
+ if err != nil {
+ return err
+ }
+ return c.JSON(http.StatusOK, okResp{out})
}
- // Sort params.
- if !strSliceContains(orderBy, bounceQuerySortFields) {
- orderBy = "created_at"
- }
- if order != sortAsc && order != sortDesc {
- order = sortDesc
+ res, total, err := app.core.QueryBounces(campID, 0, source, orderBy, order, pg.Offset, pg.Limit)
+ if err != nil {
+ return err
}
- stmt := fmt.Sprintf(app.queries.QueryBounces, orderBy, order)
- if err := db.Select(&out.Results, stmt, id, campID, 0, source, pg.Offset, pg.Limit); err != nil {
- app.log.Printf("error fetching bounces: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.bounce}", "error", pqErrMsg(err)))
- }
- if len(out.Results) == 0 {
+ // No results.
+ var out models.PageResults
+ if len(res) == 0 {
out.Results = []models.Bounce{}
return c.JSON(http.StatusOK, okResp{out})
}
- if single {
- return c.JSON(http.StatusOK, okResp{out.Results[0]})
- }
-
// Meta.
- out.Total = out.Results[0].Total
+ out.Results = res
+ out.Total = total
out.Page = pg.Page
out.PerPage = pg.PerPage
@@ -78,22 +57,17 @@ func handleGetBounces(c echo.Context) error {
// handleGetSubscriberBounces retrieves a subscriber's bounce records.
func handleGetSubscriberBounces(c echo.Context) error {
var (
- app = c.Get("app").(*App)
- subID = c.Param("id")
+ app = c.Get("app").(*App)
+ subID, _ = strconv.Atoi(c.Param("id"))
)
- id, _ := strconv.ParseInt(subID, 10, 64)
- if id < 1 {
+ if subID < 1 {
return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidID"))
}
- out := []models.Bounce{}
- stmt := fmt.Sprintf(app.queries.QueryBounces, "created_at", "ASC")
- if err := db.Select(&out, stmt, 0, 0, subID, "", 0, 1000); err != nil {
- app.log.Printf("error fetching bounces: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.bounce}", "error", pqErrMsg(err)))
+ out, _, err := app.core.QueryBounces(0, subID, "", "", "", 0, 1000)
+ if err != nil {
+ return err
}
return c.JSON(http.StatusOK, okResp{out})
@@ -105,12 +79,12 @@ func handleDeleteBounces(c echo.Context) error {
app = c.Get("app").(*App)
pID = c.Param("id")
all, _ = strconv.ParseBool(c.QueryParam("all"))
- IDs = pq.Int64Array{}
+ IDs = []int{}
)
// Is it an /:id call?
if pID != "" {
- id, _ := strconv.ParseInt(pID, 10, 64)
+ id, _ := strconv.Atoi(pID)
if id < 1 {
return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidID"))
}
@@ -130,11 +104,8 @@ func handleDeleteBounces(c echo.Context) error {
IDs = i
}
- if _, err := app.queries.DeleteBounces.Exec(IDs); err != nil {
- app.log.Printf("error deleting bounces: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorDeleting",
- "name", "{globals.terms.bounce}", "error", pqErrMsg(err)))
+ if err := app.core.DeleteBounces(IDs); err != nil {
+ return err
}
return c.JSON(http.StatusOK, okResp{true})
@@ -149,8 +120,8 @@ func handleBounceWebhook(c echo.Context) error {
bounces []models.Bounce
)
- // Read the request body instead of using using c.Bind() to read to save the entire raw request as meta.
- rawReq, err := ioutil.ReadAll(c.Request().Body)
+ // Read the request body instead of using c.Bind() to read to save the entire raw request as meta.
+ rawReq, err := io.ReadAll(c.Request().Body)
if err != nil {
app.log.Printf("error reading ses notification body: %v", err)
return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.internalError"))
@@ -161,15 +132,15 @@ func handleBounceWebhook(c echo.Context) error {
case service == "":
var b models.Bounce
if err := json.Unmarshal(rawReq, &b); err != nil {
- return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidData"))
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidData")+":"+err.Error())
}
- if err := validateBounceFields(b, app); err != nil {
+ if bv, err := validateBounceFields(b, app); err != nil {
return err
+ } else {
+ b = bv
}
- b.Email = strings.ToLower(b.Email)
-
if len(b.Meta) == 0 {
b.Meta = json.RawMessage("{}")
}
@@ -220,6 +191,36 @@ func handleBounceWebhook(c echo.Context) error {
}
bounces = append(bounces, bs...)
+ // Postmark.
+ case service == "postmark" && app.constants.BouncePostmarkEnabled:
+ bs, err := app.bounce.Postmark.ProcessBounce(rawReq, c)
+ if err != nil {
+ app.log.Printf("error processing postmark notification: %v", err)
+ if _, ok := err.(*echo.HTTPError); ok {
+ return err
+ }
+
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidData"))
+ }
+ bounces = append(bounces, bs...)
+
+ // ForwardEmail.
+ case service == "forwardemail" && app.constants.BounceForwardemailEnabled:
+ var (
+ sig = c.Request().Header.Get("X-Webhook-Signature")
+ )
+
+ bs, err := app.bounce.Forwardemail.ProcessBounce(sig, rawReq)
+ if err != nil {
+ app.log.Printf("error processing forwardemail notification: %v", err)
+ if _, ok := err.(*echo.HTTPError); ok {
+ return err
+ }
+
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidData"))
+ }
+ bounces = append(bounces, bs...)
+
default:
return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("bounces.unknownService"))
}
@@ -234,22 +235,26 @@ func handleBounceWebhook(c echo.Context) error {
return c.JSON(http.StatusOK, okResp{true})
}
-func validateBounceFields(b models.Bounce, app *App) error {
+func validateBounceFields(b models.Bounce, app *App) (models.Bounce, error) {
if b.Email == "" && b.SubscriberUUID == "" {
- return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidData"))
+ return b, echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidFields", "name", "email / subscriber_uuid"))
}
- if b.Type != models.BounceTypeHard && b.Type != models.BounceTypeSoft {
- return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidData"))
+ if b.SubscriberUUID != "" && !reUUID.MatchString(b.SubscriberUUID) {
+ return b, echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidFields", "name", "subscriber_uuid"))
}
- if b.Email != "" && !subimporter.IsEmail(b.Email) {
- return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidEmail"))
+ if b.Email != "" {
+ em, err := app.importer.SanitizeEmail(b.Email)
+ if err != nil {
+ return b, echo.NewHTTPError(http.StatusBadRequest, err.Error())
+ }
+ b.Email = em
}
- if b.SubscriberUUID != "" && !reUUID.MatchString(b.SubscriberUUID) {
- return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidUUID"))
+ if b.Type != models.BounceTypeHard && b.Type != models.BounceTypeSoft && b.Type != models.BounceTypeComplaint {
+ return b, echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.invalidFields", "name", "type"))
}
- return nil
+ return b, nil
}
diff --git a/cmd/campaigns.go b/cmd/campaigns.go
index 312f79980..0514dd79b 100644
--- a/cmd/campaigns.go
+++ b/cmd/campaigns.go
@@ -2,7 +2,7 @@ package main
import (
"bytes"
- "database/sql"
+ "encoding/json"
"errors"
"fmt"
"html/template"
@@ -13,32 +13,27 @@ import (
"strings"
"time"
- "github.com/gofrs/uuid"
- "github.com/knadh/listmonk/internal/subimporter"
"github.com/knadh/listmonk/models"
- "github.com/labstack/echo"
+ "github.com/labstack/echo/v4"
"github.com/lib/pq"
- null "gopkg.in/volatiletech/null.v6"
+ "gopkg.in/volatiletech/null.v6"
)
// campaignReq is a wrapper over the Campaign model for receiving
-// campaign creation and updation data from APIs.
+// campaign creation and update data from APIs.
type campaignReq struct {
models.Campaign
- // Indicates if the "send_at" date should be written or set to null.
- SendLater bool `db:"-" json:"send_later"`
-
// This overrides Campaign.Lists to receive and
// write a list of int IDs during creation and updation.
// Campaign.Lists is JSONText for sending lists children
// to the outside world.
- ListIDs pq.Int64Array `db:"-" json:"lists"`
+ ListIDs []int `json:"lists"`
+
+ MediaIDs []int `json:"media"`
// This is only relevant to campaign test requests.
SubscriberEmails pq.StringArray `json:"subscribers"`
-
- Type string `json:"type"`
}
// campaignContentReq wraps params coming from API requests for converting
@@ -49,111 +44,68 @@ type campaignContentReq struct {
To string `json:"to"`
}
-type campaignStats struct {
- ID int `db:"id" json:"id"`
- Status string `db:"status" json:"status"`
- ToSend int `db:"to_send" json:"to_send"`
- Sent int `db:"sent" json:"sent"`
- Started null.Time `db:"started_at" json:"started_at"`
- UpdatedAt null.Time `db:"updated_at" json:"updated_at"`
- Rate float64 `json:"rate"`
-}
-
-type campsWrap struct {
- Results models.Campaigns `json:"results"`
-
- Query string `json:"query"`
- Total int `json:"total"`
- PerPage int `json:"per_page"`
- Page int `json:"page"`
-}
-
var (
- regexFromAddress = regexp.MustCompile(`(.+?)\s<(.+?)@(.+?)>`)
- regexFullTextQuery = regexp.MustCompile(`\s+`)
-
- campaignQuerySortFields = []string{"name", "status", "created_at", "updated_at"}
- bounceQuerySortFields = []string{"email", "campaign_name", "source", "created_at"}
+ regexFromAddress = regexp.MustCompile(`((.+?)\s)?<(.+?)@(.+?)>`)
+ regexSlug = regexp.MustCompile(`[^\p{L}\p{M}\p{N}]`)
)
// handleGetCampaigns handles retrieval of campaigns.
func handleGetCampaigns(c echo.Context) error {
var (
app = c.Get("app").(*App)
- pg = getPagination(c.QueryParams(), 20)
- out campsWrap
+ pg = app.paginator.NewFromURL(c.Request().URL.Query())
- id, _ = strconv.Atoi(c.Param("id"))
status = c.QueryParams()["status"]
+ tags = c.QueryParams()["tag"]
query = strings.TrimSpace(c.FormValue("query"))
orderBy = c.FormValue("order_by")
order = c.FormValue("order")
noBody, _ = strconv.ParseBool(c.QueryParam("no_body"))
)
- // Fetch one list.
- single := false
- if id > 0 {
- single = true
- }
- if query != "" {
- query = `%` +
- string(regexFullTextQuery.ReplaceAll([]byte(query), []byte("&"))) + `%`
+ res, total, err := app.core.QueryCampaigns(query, status, tags, orderBy, order, pg.Offset, pg.Limit)
+ if err != nil {
+ return err
}
- // Sort params.
- if !strSliceContains(orderBy, campaignQuerySortFields) {
- orderBy = "created_at"
- }
- if order != sortAsc && order != sortDesc {
- order = sortDesc
+ if noBody {
+ for i := 0; i < len(res); i++ {
+ res[i].Body = ""
+ }
}
- stmt := fmt.Sprintf(app.queries.QueryCampaigns, orderBy, order)
-
- // Unsafe to ignore scanning fields not present in models.Campaigns.
- if err := db.Select(&out.Results, stmt, id, pq.StringArray(status), query, pg.Offset, pg.Limit); err != nil {
- app.log.Printf("error fetching campaigns: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
- }
- if single && len(out.Results) == 0 {
- return echo.NewHTTPError(http.StatusBadRequest,
- app.i18n.Ts("campaigns.notFound", "name", "{globals.terms.campaign}"))
- }
- if len(out.Results) == 0 {
+ var out models.PageResults
+ if len(res) == 0 {
out.Results = []models.Campaign{}
return c.JSON(http.StatusOK, okResp{out})
}
- for i := 0; i < len(out.Results); i++ {
- // Replace null tags.
- if out.Results[i].Tags == nil {
- out.Results[i].Tags = make(pq.StringArray, 0)
- }
+ // Meta.
+ out.Query = query
+ out.Results = res
+ out.Total = total
+ out.Page = pg.Page
+ out.PerPage = pg.PerPage
- if noBody {
- out.Results[i].Body = ""
- }
- }
+ return c.JSON(http.StatusOK, okResp{out})
+}
- // Lazy load stats.
- if err := out.Results.LoadStats(app.queries.GetCampaignStats); err != nil {
- app.log.Printf("error fetching campaign stats: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
- }
+// handleGetCampaign handles retrieval of campaigns.
+func handleGetCampaign(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ id, _ = strconv.Atoi(c.Param("id"))
+ noBody, _ = strconv.ParseBool(c.QueryParam("no_body"))
+ )
- if single {
- return c.JSON(http.StatusOK, okResp{out.Results[0]})
+ out, err := app.core.GetCampaign(id, "", "")
+ if err != nil {
+ return err
}
- // Meta.
- out.Total = out.Results[0].Total
- out.Page = pg.Page
- out.PerPage = pg.PerPage
+ if noBody {
+ out.Body = ""
+ }
return c.JSON(http.StatusOK, okResp{out})
}
@@ -161,26 +113,18 @@ func handleGetCampaigns(c echo.Context) error {
// handlePreviewCampaign renders the HTML preview of a campaign body.
func handlePreviewCampaign(c echo.Context) error {
var (
- app = c.Get("app").(*App)
- id, _ = strconv.Atoi(c.Param("id"))
+ app = c.Get("app").(*App)
+ id, _ = strconv.Atoi(c.Param("id"))
+ tplID, _ = strconv.Atoi(c.FormValue("template_id"))
)
if id < 1 {
return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidID"))
}
- var camp models.Campaign
- err := app.queries.GetCampaignForPreview.Get(&camp, id)
+ camp, err := app.core.GetCampaignForPreview(id, tplID)
if err != nil {
- if err == sql.ErrNoRows {
- return echo.NewHTTPError(http.StatusBadRequest,
- app.i18n.Ts("globals.messages.notFound", "name", "{globals.terms.campaign}"))
- }
-
- app.log.Printf("error fetching campaign: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
+ return err
}
// There's a body in the request to preview instead of the body in the DB.
@@ -206,6 +150,10 @@ func handlePreviewCampaign(c echo.Context) error {
app.i18n.Ts("templates.errorRendering", "error", err.Error()))
}
+ if camp.ContentType == models.CampaignContentTypePlain {
+ return c.String(http.StatusOK, string(msg.Body()))
+ }
+
return c.HTML(http.StatusOK, string(msg.Body()))
}
@@ -252,6 +200,15 @@ func handleCreateCampaign(c echo.Context) error {
return err
}
o = op
+ } else if o.Type == "" {
+ o.Type = models.CampaignTypeRegular
+ }
+
+ if o.ContentType == "" {
+ o.ContentType = models.CampaignContentTypeRichtext
+ }
+ if o.Messenger == "" {
+ o.Messenger = "email"
}
// Validate.
@@ -261,44 +218,16 @@ func handleCreateCampaign(c echo.Context) error {
o = c
}
- uu, err := uuid.NewV4()
- if err != nil {
- app.log.Printf("error generating UUID: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorUUID", "error", err.Error()))
- }
-
- // Insert and read ID.
- var newID int
- if err := app.queries.CreateCampaign.Get(&newID,
- uu,
- o.Type,
- o.Name,
- o.Subject,
- o.FromEmail,
- o.Body,
- o.AltBody,
- o.ContentType,
- o.SendAt,
- pq.StringArray(normalizeTags(o.Tags)),
- o.Messenger,
- o.TemplateID,
- o.ListIDs,
- ); err != nil {
- if err == sql.ErrNoRows {
- return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("campaigns.noSubs"))
- }
+ if o.ArchiveTemplateID == 0 {
+ o.ArchiveTemplateID = o.TemplateID
+ }
- app.log.Printf("error creating campaign: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorCreating",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
+ out, err := app.core.CreateCampaign(o.Campaign, o.ListIDs, o.MediaIDs)
+ if err != nil {
+ return err
}
- // Hand over to the GET handler to return the last insertion.
- return handleGetCampaigns(copyEchoCtx(c, map[string]string{
- "id": fmt.Sprintf("%d", newID),
- }))
+ return c.JSON(http.StatusOK, okResp{out})
}
// handleUpdateCampaign handles campaign modification.
@@ -314,25 +243,17 @@ func handleUpdateCampaign(c echo.Context) error {
}
- var cm models.Campaign
- if err := app.queries.GetCampaign.Get(&cm, id, nil); err != nil {
- if err == sql.ErrNoRows {
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.notFound", "name", "{globals.terms.campaign}"))
- }
-
- app.log.Printf("error fetching campaign: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
+ cm, err := app.core.GetCampaign(id, "", "")
+ if err != nil {
+ return err
}
- if isCampaignalMutable(cm.Status) {
+ if !canEditCampaign(cm.Status) {
return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("campaigns.cantUpdate"))
}
// Read the incoming params into the existing campaign fields from the DB.
- // This allows updating of values that have been sent where as fields
+ // This allows updating of values that have been sent whereas fields
// that are not in the request retain the old values.
o := campaignReq{Campaign: cm}
if err := c.Bind(&o); err != nil {
@@ -345,27 +266,12 @@ func handleUpdateCampaign(c echo.Context) error {
o = c
}
- _, err := app.queries.UpdateCampaign.Exec(cm.ID,
- o.Name,
- o.Subject,
- o.FromEmail,
- o.Body,
- o.AltBody,
- o.ContentType,
- o.SendAt,
- o.SendLater,
- pq.StringArray(normalizeTags(o.Tags)),
- o.Messenger,
- o.TemplateID,
- o.ListIDs)
+ out, err := app.core.UpdateCampaign(id, o.Campaign, o.ListIDs, o.MediaIDs)
if err != nil {
- app.log.Printf("error updating campaign: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorUpdating",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
+ return err
}
- return handleGetCampaigns(c)
+ return c.JSON(http.StatusOK, okResp{out})
}
// handleUpdateCampaignStatus handles campaign status modification.
@@ -379,73 +285,58 @@ func handleUpdateCampaignStatus(c echo.Context) error {
return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidID"))
}
- var cm models.Campaign
- if err := app.queries.GetCampaign.Get(&cm, id, nil); err != nil {
- if err == sql.ErrNoRows {
- return echo.NewHTTPError(http.StatusBadRequest,
- app.i18n.Ts("globals.message.notFound", "name", "{globals.terms.campaign}"))
- }
-
- app.log.Printf("error fetching campaign: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
+ var o struct {
+ Status string `json:"status"`
}
- // Incoming params.
- var o campaignReq
if err := c.Bind(&o); err != nil {
return err
}
- errMsg := ""
- switch o.Status {
- case models.CampaignStatusDraft:
- if cm.Status != models.CampaignStatusScheduled {
- errMsg = app.i18n.T("campaigns.onlyScheduledAsDraft")
- }
- case models.CampaignStatusScheduled:
- if cm.Status != models.CampaignStatusDraft {
- errMsg = app.i18n.T("campaigns.onlyDraftAsScheduled")
- }
- if !cm.SendAt.Valid {
- errMsg = app.i18n.T("campaigns.needsSendAt")
- }
-
- case models.CampaignStatusRunning:
- if cm.Status != models.CampaignStatusPaused && cm.Status != models.CampaignStatusDraft {
- errMsg = app.i18n.T("campaigns.onlyPausedDraft")
- }
- case models.CampaignStatusPaused:
- if cm.Status != models.CampaignStatusRunning {
- errMsg = app.i18n.T("campaigns.onlyActivePause")
- }
- case models.CampaignStatusCancelled:
- if cm.Status != models.CampaignStatusRunning && cm.Status != models.CampaignStatusPaused {
- errMsg = app.i18n.T("campaigns.onlyActiveCancel")
- }
+ out, err := app.core.UpdateCampaignStatus(id, o.Status)
+ if err != nil {
+ return err
}
- if len(errMsg) > 0 {
- return echo.NewHTTPError(http.StatusBadRequest, errMsg)
+ if o.Status == models.CampaignStatusPaused || o.Status == models.CampaignStatusCancelled {
+ app.manager.StopCampaign(id)
}
- res, err := app.queries.UpdateCampaignStatus.Exec(cm.ID, o.Status)
- if err != nil {
- app.log.Printf("error updating campaign status: %v", err)
+ return c.JSON(http.StatusOK, okResp{out})
+}
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorUpdating",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
+// handleUpdateCampaignArchive handles campaign status modification.
+func handleUpdateCampaignArchive(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ id, _ = strconv.Atoi(c.Param("id"))
+ )
+
+ req := struct {
+ Archive bool `json:"archive"`
+ TemplateID int `json:"archive_template_id"`
+ Meta models.JSON `json:"archive_meta"`
+ ArchiveSlug string `json:"archive_slug"`
+ }{}
+
+ // Get and validate fields.
+ if err := c.Bind(&req); err != nil {
+ return err
}
- if n, _ := res.RowsAffected(); n == 0 {
- return echo.NewHTTPError(http.StatusBadRequest,
- app.i18n.Ts("globals.messages.notFound",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
+ if req.ArchiveSlug != "" {
+ // Format the slug to be alpha-numeric-dash.
+ s := strings.ToLower(req.ArchiveSlug)
+ s = strings.TrimSpace(regexSlug.ReplaceAllString(s, " "))
+ s = regexpSpaces.ReplaceAllString(s, "-")
+ req.ArchiveSlug = s
+ }
+
+ if err := app.core.UpdateCampaignArchive(id, req.Archive, req.TemplateID, req.Meta, req.ArchiveSlug); err != nil {
+ return err
}
- return handleGetCampaigns(c)
+ return c.JSON(http.StatusOK, okResp{req})
}
// handleDeleteCampaign handles campaign deletion.
@@ -460,26 +351,8 @@ func handleDeleteCampaign(c echo.Context) error {
return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidID"))
}
- var cm models.Campaign
- if err := app.queries.GetCampaign.Get(&cm, id, nil); err != nil {
- if err == sql.ErrNoRows {
- return echo.NewHTTPError(http.StatusBadRequest,
- app.i18n.Ts("globals.messages.notFound",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
- }
-
- app.log.Printf("error fetching campaign: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
- }
-
- if _, err := app.queries.DeleteCampaign.Exec(cm.ID); err != nil {
- app.log.Printf("error deleting campaign: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorDeleting",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
-
+ if err := app.core.DeleteCampaign(id); err != nil {
+ return err
}
return c.JSON(http.StatusOK, okResp{true})
@@ -489,36 +362,35 @@ func handleDeleteCampaign(c echo.Context) error {
func handleGetRunningCampaignStats(c echo.Context) error {
var (
app = c.Get("app").(*App)
- out []campaignStats
)
- if err := app.queries.GetCampaignStatus.Select(&out, models.CampaignStatusRunning); err != nil {
- if err == sql.ErrNoRows {
- return c.JSON(http.StatusOK, okResp{[]struct{}{}})
- }
+ out, err := app.core.GetRunningCampaignStats()
+ if err != nil {
+ return err
+ }
- app.log.Printf("error fetching campaign stats: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
- } else if len(out) == 0 {
+ if len(out) == 0 {
return c.JSON(http.StatusOK, okResp{[]struct{}{}})
}
// Compute rate.
for i, c := range out {
if c.Started.Valid && c.UpdatedAt.Valid {
- diff := c.UpdatedAt.Time.Sub(c.Started.Time).Minutes()
- if diff > 0 {
- var (
- sent = float64(c.Sent)
- rate = sent / diff
- )
- if rate > sent || rate > float64(c.ToSend) {
- rate = sent
- }
- out[i].Rate = rate
+ diff := int(c.UpdatedAt.Time.Sub(c.Started.Time).Minutes())
+ if diff < 1 {
+ diff = 1
+ }
+
+ rate := c.Sent / diff
+ if rate > c.Sent || rate > c.ToSend {
+ rate = c.Sent
}
+
+ // Rate since the starting of the campaign.
+ out[i].NetRate = rate
+
+ // Realtime running rate over the last minute.
+ out[i].Rate = app.manager.GetCampaignStats(c.ID).SendRate
}
}
@@ -531,6 +403,7 @@ func handleTestCampaign(c echo.Context) error {
var (
app = c.Get("app").(*App)
campID, _ = strconv.Atoi(c.Param("id"))
+ tplID, _ = strconv.Atoi(c.FormValue("template_id"))
req campaignReq
)
@@ -557,29 +430,16 @@ func handleTestCampaign(c echo.Context) error {
for i := 0; i < len(req.SubscriberEmails); i++ {
req.SubscriberEmails[i] = strings.ToLower(strings.TrimSpace(req.SubscriberEmails[i]))
}
- var subs models.Subscribers
- if err := app.queries.GetSubscribersByEmails.Select(&subs, req.SubscriberEmails); err != nil {
- app.log.Printf("error fetching subscribers: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.subscribers}", "error", pqErrMsg(err)))
- } else if len(subs) == 0 {
- return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("campaigns.noKnownSubsToTest"))
+
+ subs, err := app.core.GetSubscribersByEmail(req.SubscriberEmails)
+ if err != nil {
+ return err
}
// The campaign.
- var camp models.Campaign
- if err := app.queries.GetCampaignForPreview.Get(&camp, campID); err != nil {
- if err == sql.ErrNoRows {
- return echo.NewHTTPError(http.StatusBadRequest,
- app.i18n.Ts("globals.messages.notFound",
- "name", "{globals.terms.campaign}"))
- }
-
- app.log.Printf("error fetching campaign: %v", err)
- return echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.campaign}", "error", pqErrMsg(err)))
+ camp, err := app.core.GetCampaignForPreview(campID, tplID)
+ if err != nil {
+ return err
}
// Override certain values from the DB with incoming values.
@@ -590,12 +450,19 @@ func handleTestCampaign(c echo.Context) error {
camp.AltBody = req.AltBody
camp.Messenger = req.Messenger
camp.ContentType = req.ContentType
+ camp.Headers = req.Headers
camp.TemplateID = req.TemplateID
+ for _, id := range req.MediaIDs {
+ if id > 0 {
+ camp.MediaIDs = append(camp.MediaIDs, int64(id))
+ }
+ }
// Send the test messages.
for _, s := range subs {
sub := s
- if err := sendTestMessage(sub, &camp, app); err != nil {
+ c := camp
+ if err := sendTestMessage(sub, &c, app); err != nil {
app.log.Printf("error sending test message: %v", err)
return echo.NewHTTPError(http.StatusInternalServerError,
app.i18n.Ts("campaigns.errorSendTest", "error", err.Error()))
@@ -605,7 +472,51 @@ func handleTestCampaign(c echo.Context) error {
return c.JSON(http.StatusOK, okResp{true})
}
-// sendTestMessage takes a campaign and a subsriber and sends out a sample campaign message.
+// handleGetCampaignViewAnalytics retrieves view counts for a campaign.
+func handleGetCampaignViewAnalytics(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+
+ typ = c.Param("type")
+ from = c.QueryParams().Get("from")
+ to = c.QueryParams().Get("to")
+ )
+
+ ids, err := parseStringIDs(c.Request().URL.Query()["id"])
+ if err != nil {
+ return echo.NewHTTPError(http.StatusBadRequest,
+ app.i18n.Ts("globals.messages.errorInvalidIDs", "error", err.Error()))
+ }
+
+ if len(ids) == 0 {
+ return echo.NewHTTPError(http.StatusBadRequest,
+ app.i18n.Ts("globals.messages.missingFields", "name", "`id`"))
+ }
+
+ if !strHasLen(from, 10, 30) || !strHasLen(to, 10, 30) {
+ return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("analytics.invalidDates"))
+ }
+
+ // Campaign link stats.
+ if typ == "links" {
+ out, err := app.core.GetCampaignAnalyticsLinks(ids, typ, from, to)
+ if err != nil {
+ return err
+ }
+
+ return c.JSON(http.StatusOK, okResp{out})
+ }
+
+ // View, click, bounce stats.
+ out, err := app.core.GetCampaignAnalyticsCounts(ids, typ, from, to)
+ if err != nil {
+ return err
+ }
+
+ return c.JSON(http.StatusOK, okResp{out})
+}
+
+// sendTestMessage takes a campaign and a subscriber and sends out a sample campaign message.
func sendTestMessage(sub models.Subscriber, camp *models.Campaign, app *App) error {
if err := camp.CompileTemplate(app.manager.TemplateFuncs(camp)); err != nil {
app.log.Printf("error compiling template: %v", err)
@@ -629,7 +540,7 @@ func validateCampaignFields(c campaignReq, app *App) (campaignReq, error) {
if c.FromEmail == "" {
c.FromEmail = app.constants.FromEmail
} else if !regexFromAddress.Match([]byte(c.FromEmail)) {
- if !subimporter.IsEmail(c.FromEmail) {
+ if _, err := app.importer.SanitizeEmail(c.FromEmail); err != nil {
return c, errors.New(app.i18n.T("campaigns.fieldInvalidFromEmail"))
}
}
@@ -637,14 +548,12 @@ func validateCampaignFields(c campaignReq, app *App) (campaignReq, error) {
if !strHasLen(c.Name, 1, stdInputMaxLen) {
return c, errors.New(app.i18n.T("campaigns.fieldInvalidName"))
}
- if !strHasLen(c.Subject, 1, stdInputMaxLen) {
+
+ // Larger char limit for subject as it can contain {{ go templating }} logic.
+ if !strHasLen(c.Subject, 1, 5000) {
return c, errors.New(app.i18n.T("campaigns.fieldInvalidSubject"))
}
- // if !hasLen(c.Body, 1, bodyMaxLen) {
- // return c,errors.New("invalid length for `body`")
- // }
-
// If there's a "send_at" date, it should be in the future.
if c.SendAt.Valid {
if c.SendAt.Time.Before(time.Now()) {
@@ -665,15 +574,35 @@ func validateCampaignFields(c campaignReq, app *App) (campaignReq, error) {
return c, errors.New(app.i18n.Ts("campaigns.fieldInvalidBody", "error", err.Error()))
}
+ if len(c.Headers) == 0 {
+ c.Headers = make([]map[string]string, 0)
+ }
+
+ if len(c.ArchiveMeta) == 0 {
+ c.ArchiveMeta = json.RawMessage("{}")
+ }
+
+ if c.ArchiveSlug.String != "" {
+ // Format the slug to be alpha-numeric-dash.
+ s := strings.ToLower(c.ArchiveSlug.String)
+ s = strings.TrimSpace(regexSlug.ReplaceAllString(s, " "))
+ s = regexpSpaces.ReplaceAllString(s, "-")
+
+ c.ArchiveSlug = null.NewString(s, true)
+ } else {
+ // If there's no slug set, set it to NULL in the DB.
+ c.ArchiveSlug.Valid = false
+ }
+
return c, nil
}
-// isCampaignalMutable tells if a campaign's in a state where it's
-// properties can be mutated.
-func isCampaignalMutable(status string) bool {
- return status == models.CampaignStatusRunning ||
- status == models.CampaignStatusCancelled ||
- status == models.CampaignStatusFinished
+// canEditCampaign returns true if a campaign is in a status where updating
+// its properties is allowed.
+func canEditCampaign(status string) bool {
+ return status == models.CampaignStatusDraft ||
+ status == models.CampaignStatusPaused ||
+ status == models.CampaignStatusScheduled
}
// makeOptinCampaignMessage makes a default opt-in campaign message body.
@@ -683,13 +612,9 @@ func makeOptinCampaignMessage(o campaignReq, app *App) (campaignReq, error) {
}
// Fetch double opt-in lists from the given list IDs.
- var lists []models.List
- err := app.queries.GetListsByOptin.Select(&lists, models.ListOptinDouble, pq.Int64Array(o.ListIDs), nil)
+ lists, err := app.core.GetListsByOptin(o.ListIDs, models.ListOptinDouble)
if err != nil {
- app.log.Printf("error fetching lists for opt-in: %s", pqErrMsg(err))
- return o, echo.NewHTTPError(http.StatusInternalServerError,
- app.i18n.Ts("globals.messages.errorFetching",
- "name", "{globals.terms.list}", "error", pqErrMsg(err)))
+ return o, err
}
// No opt-in lists.
@@ -707,7 +632,7 @@ func makeOptinCampaignMessage(o campaignReq, app *App) (campaignReq, error) {
// Prepare sample opt-in message for the campaign.
var b bytes.Buffer
- if err := app.notifTpls.ExecuteTemplate(&b, "optin-campaign", struct {
+ if err := app.notifTpls.tpls.ExecuteTemplate(&b, "optin-campaign", struct {
Lists []models.List
OptinURLAttr template.HTMLAttr
}{lists, optinURLAttr}); err != nil {
diff --git a/cmd/events.go b/cmd/events.go
new file mode 100644
index 000000000..33dda5374
--- /dev/null
+++ b/cmd/events.go
@@ -0,0 +1,51 @@
+package main
+
+import (
+ "encoding/json"
+ "fmt"
+ "log"
+ "time"
+
+ "github.com/labstack/echo/v4"
+)
+
+// handleEventStream serves an endpoint that never closes and pushes a
+// live event stream (text/event-stream) such as a error messages.
+func handleEventStream(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
+ )
+
+ h := c.Response().Header()
+ h.Set(echo.HeaderContentType, "text/event-stream")
+ h.Set(echo.HeaderCacheControl, "no-store")
+ h.Set(echo.HeaderConnection, "keep-alive")
+
+ // Subscribe to the event stream with a random ID.
+ id := fmt.Sprintf("api:%v", time.Now().UnixNano())
+ sub, err := app.events.Subscribe(id)
+ if err != nil {
+ log.Fatalf("error subscribing to events: %v", err)
+ }
+
+ ctx := c.Request().Context()
+ for {
+ select {
+ case e := <-sub:
+ b, err := json.Marshal(e)
+ if err != nil {
+ app.log.Printf("error marshalling event: %v", err)
+ continue
+ }
+
+ c.Response().Write([]byte(fmt.Sprintf("retry: 3000\ndata: %s\n\n", b)))
+ c.Response().Flush()
+
+ case <-ctx.Done():
+ // On HTTP connection close, unsubscribe.
+ app.events.Unsubscribe(id)
+ return nil
+ }
+ }
+
+}
diff --git a/cmd/handlers.go b/cmd/handlers.go
index 7e777ba0f..0c38adb2a 100644
--- a/cmd/handlers.go
+++ b/cmd/handlers.go
@@ -1,22 +1,29 @@
package main
import (
- "crypto/subtle"
+ "bytes"
"net/http"
"net/url"
+ "path"
"regexp"
- "strconv"
- "github.com/labstack/echo"
- "github.com/labstack/echo/middleware"
+ "github.com/knadh/listmonk/internal/auth"
+ "github.com/knadh/paginator"
+ "github.com/labstack/echo/v4"
+ "github.com/labstack/echo/v4/middleware"
)
const (
// stdInputMaxLen is the maximum allowed length for a standard input field.
- stdInputMaxLen = 200
+ stdInputMaxLen = 2000
sortAsc = "asc"
sortDesc = "desc"
+
+ basicAuthd = "basicauthd"
+
+ // URIs.
+ uriAdmin = "/admin"
)
type okResp struct {
@@ -34,153 +41,270 @@ type pagination struct {
var (
reUUID = regexp.MustCompile("^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$")
reLangCode = regexp.MustCompile("[^a-zA-Z_0-9\\-]")
+
+ paginate = paginator.New(paginator.Opt{
+ DefaultPerPage: 20,
+ MaxPerPage: 50,
+ NumPageNums: 10,
+ PageParam: "page",
+ PerPageParam: "per_page",
+ })
)
// registerHandlers registers HTTP handlers.
-func registerHTTPHandlers(e *echo.Echo, app *App) {
- // Group of private handlers with BasicAuth.
- var g *echo.Group
-
- if len(app.constants.AdminUsername) == 0 ||
- len(app.constants.AdminPassword) == 0 {
- g = e.Group("")
- } else {
- g = e.Group("", middleware.BasicAuth(basicAuth))
+func initHTTPHandlers(e *echo.Echo, app *App) {
+ // Default error handler.
+ e.HTTPErrorHandler = func(err error, c echo.Context) {
+ // Generic, non-echo error. Log it.
+ if _, ok := err.(*echo.HTTPError); !ok {
+ app.log.Println(err.Error())
+ }
+ e.DefaultHTTPErrorHandler(err, c)
}
- g.GET("/", handleIndexPage)
- g.GET("/api/health", handleHealthCheck)
- g.GET("/api/config", handleGetServerConfig)
- g.GET("/api/lang/:lang", handleGetI18nLang)
- g.GET("/api/dashboard/charts", handleGetDashboardCharts)
- g.GET("/api/dashboard/counts", handleGetDashboardCounts)
-
- g.GET("/api/settings", handleGetSettings)
- g.PUT("/api/settings", handleUpdateSettings)
- g.POST("/api/admin/reload", handleReloadApp)
- g.GET("/api/logs", handleGetLogs)
-
- g.GET("/api/subscribers/:id", handleGetSubscriber)
- g.GET("/api/subscribers/:id/export", handleExportSubscriberData)
- g.GET("/api/subscribers/:id/bounces", handleGetSubscriberBounces)
- g.DELETE("/api/subscribers/:id/bounces", handleDeleteSubscriberBounces)
- g.POST("/api/subscribers", handleCreateSubscriber)
- g.PUT("/api/subscribers/:id", handleUpdateSubscriber)
- g.POST("/api/subscribers/:id/optin", handleSubscriberSendOptin)
- g.PUT("/api/subscribers/blocklist", handleBlocklistSubscribers)
- g.PUT("/api/subscribers/:id/blocklist", handleBlocklistSubscribers)
- g.PUT("/api/subscribers/lists/:id", handleManageSubscriberLists)
- g.PUT("/api/subscribers/lists", handleManageSubscriberLists)
- g.DELETE("/api/subscribers/:id", handleDeleteSubscribers)
- g.DELETE("/api/subscribers", handleDeleteSubscribers)
-
- g.GET("/api/bounces", handleGetBounces)
- g.DELETE("/api/bounces", handleDeleteBounces)
- g.DELETE("/api/bounces/:id", handleDeleteBounces)
+ var (
+ // Authenticated /api/* handlers.
+ api = e.Group("", app.auth.Middleware, func(next echo.HandlerFunc) echo.HandlerFunc {
+ return func(c echo.Context) error {
+ u := c.Get(auth.UserKey)
+
+ // On no-auth, respond with a JSON error.
+ if err, ok := u.(*echo.HTTPError); ok {
+ return err
+ }
+
+ return next(c)
+ }
+ })
+
+ // Authenticated non /api handlers.
+ a = e.Group("", app.auth.Middleware, func(next echo.HandlerFunc) echo.HandlerFunc {
+ return func(c echo.Context) error {
+ u := c.Get(auth.UserKey)
+ // On no-auth, redirect to login page
+ if _, ok := u.(*echo.HTTPError); ok {
+ u, _ := url.Parse(app.constants.LoginURL)
+ q := url.Values{}
+ q.Set("next", c.Request().RequestURI)
+ u.RawQuery = q.Encode()
+ return c.Redirect(http.StatusTemporaryRedirect, u.String())
+ }
+
+ return next(c)
+ }
+ })
+
+ // Public unauthenticated endpoints.
+ p = e.Group("")
+ )
+
+ // Authenticated endpoints.
+ a.GET(path.Join(uriAdmin, ""), handleAdminPage)
+ a.GET(path.Join(uriAdmin, "/custom.css"), serveCustomAppearance("admin.custom_css"))
+ a.GET(path.Join(uriAdmin, "/custom.js"), serveCustomAppearance("admin.custom_js"))
+ a.GET(path.Join(uriAdmin, "/*"), handleAdminPage)
+
+ pm := app.auth.Perm
+
+ // API endpoints.
+ api.GET("/api/health", handleHealthCheck)
+ api.GET("/api/config", handleGetServerConfig)
+ api.GET("/api/lang/:lang", handleGetI18nLang)
+ api.GET("/api/dashboard/charts", handleGetDashboardCharts)
+ api.GET("/api/dashboard/counts", handleGetDashboardCounts)
+
+ api.GET("/api/settings", pm(handleGetSettings, "settings:get"))
+ api.PUT("/api/settings", pm(handleUpdateSettings, "settings:manage"))
+ api.POST("/api/settings/smtp/test", pm(handleTestSMTPSettings, "settings:manage"))
+ api.POST("/api/admin/reload", pm(handleReloadApp, "settings:manage"))
+ api.GET("/api/logs", pm(handleGetLogs, "settings:get"))
+ api.GET("/api/events", pm(handleEventStream, "settings:get"))
+ api.GET("/api/about", handleGetAboutInfo)
+
+ api.GET("/api/subscribers", pm(handleQuerySubscribers, "subscribers:get_all", "subscribers:get"))
+ api.GET("/api/subscribers/:id", pm(handleGetSubscriber, "subscribers:get_all", "subscribers:get"))
+ api.GET("/api/subscribers/:id/export", pm(handleExportSubscriberData, "subscribers:get_all", "subscribers:get"))
+ api.GET("/api/subscribers/:id/bounces", pm(handleGetSubscriberBounces, "bounces:get"))
+ api.DELETE("/api/subscribers/:id/bounces", pm(handleDeleteSubscriberBounces, "bounces:manage"))
+ api.POST("/api/subscribers", pm(handleCreateSubscriber, "subscribers:manage"))
+ api.PUT("/api/subscribers/:id", pm(handleUpdateSubscriber, "subscribers:manage"))
+ api.POST("/api/subscribers/:id/optin", pm(handleSubscriberSendOptin, "subscribers:manage"))
+ api.PUT("/api/subscribers/blocklist", pm(handleBlocklistSubscribers, "subscribers:manage"))
+ api.PUT("/api/subscribers/:id/blocklist", pm(handleBlocklistSubscribers, "subscribers:manage"))
+ api.PUT("/api/subscribers/lists/:id", pm(handleManageSubscriberLists, "subscribers:manage"))
+ api.PUT("/api/subscribers/lists", pm(handleManageSubscriberLists, "subscribers:manage"))
+ api.DELETE("/api/subscribers/:id", pm(handleDeleteSubscribers, "subscribers:manage"))
+ api.DELETE("/api/subscribers", pm(handleDeleteSubscribers, "subscribers:manage"))
+
+ api.GET("/api/bounces", pm(handleGetBounces, "bounces:get"))
+ api.GET("/api/bounces/:id", pm(handleGetBounces, "bounces:get"))
+ api.DELETE("/api/bounces", pm(handleDeleteBounces, "bounces:manage"))
+ api.DELETE("/api/bounces/:id", pm(handleDeleteBounces, "bounces:manage"))
// Subscriber operations based on arbitrary SQL queries.
// These aren't very REST-like.
- g.POST("/api/subscribers/query/delete", handleDeleteSubscribersByQuery)
- g.PUT("/api/subscribers/query/blocklist", handleBlocklistSubscribersByQuery)
- g.PUT("/api/subscribers/query/lists", handleManageSubscriberListsByQuery)
- g.GET("/api/subscribers", handleQuerySubscribers)
- g.GET("/api/subscribers/export",
- middleware.GzipWithConfig(middleware.GzipConfig{Level: 9})(handleExportSubscribers))
-
- g.GET("/api/import/subscribers", handleGetImportSubscribers)
- g.GET("/api/import/subscribers/logs", handleGetImportSubscriberStats)
- g.POST("/api/import/subscribers", handleImportSubscribers)
- g.DELETE("/api/import/subscribers", handleStopImportSubscribers)
-
- g.GET("/api/lists", handleGetLists)
- g.GET("/api/lists/:id", handleGetLists)
- g.POST("/api/lists", handleCreateList)
- g.PUT("/api/lists/:id", handleUpdateList)
- g.DELETE("/api/lists/:id", handleDeleteLists)
-
- g.GET("/api/campaigns", handleGetCampaigns)
- g.GET("/api/campaigns/running/stats", handleGetRunningCampaignStats)
- g.GET("/api/campaigns/:id", handleGetCampaigns)
- g.GET("/api/campaigns/:id/preview", handlePreviewCampaign)
- g.POST("/api/campaigns/:id/preview", handlePreviewCampaign)
- g.POST("/api/campaigns/:id/content", handleCampaignContent)
- g.POST("/api/campaigns/:id/text", handlePreviewCampaign)
- g.POST("/api/campaigns/:id/test", handleTestCampaign)
- g.POST("/api/campaigns", handleCreateCampaign)
- g.PUT("/api/campaigns/:id", handleUpdateCampaign)
- g.PUT("/api/campaigns/:id/status", handleUpdateCampaignStatus)
- g.DELETE("/api/campaigns/:id", handleDeleteCampaign)
-
- g.GET("/api/media", handleGetMedia)
- g.POST("/api/media", handleUploadMedia)
- g.DELETE("/api/media/:id", handleDeleteMedia)
-
- g.GET("/api/templates", handleGetTemplates)
- g.GET("/api/templates/:id", handleGetTemplates)
- g.GET("/api/templates/:id/preview", handlePreviewTemplate)
- g.POST("/api/templates/preview", handlePreviewTemplate)
- g.POST("/api/templates", handleCreateTemplate)
- g.PUT("/api/templates/:id", handleUpdateTemplate)
- g.PUT("/api/templates/:id/default", handleTemplateSetDefault)
- g.DELETE("/api/templates/:id", handleDeleteTemplate)
-
- // Static admin views.
- g.GET("/lists", handleIndexPage)
- g.GET("/lists/forms", handleIndexPage)
- g.GET("/subscribers", handleIndexPage)
- g.GET("/subscribers/lists/:listID", handleIndexPage)
- g.GET("/subscribers/import", handleIndexPage)
- g.GET("/subscribers/bounces", handleIndexPage)
- g.GET("/campaigns", handleIndexPage)
- g.GET("/campaigns/new", handleIndexPage)
- g.GET("/campaigns/media", handleIndexPage)
- g.GET("/campaigns/templates", handleIndexPage)
- g.GET("/campaigns/:campignID", handleIndexPage)
- g.GET("/settings", handleIndexPage)
- g.GET("/settings/logs", handleIndexPage)
+ api.POST("/api/subscribers/query/delete", pm(handleDeleteSubscribersByQuery, "subscribers:manage"))
+ api.PUT("/api/subscribers/query/blocklist", pm(handleBlocklistSubscribersByQuery, "subscribers:manage"))
+ api.PUT("/api/subscribers/query/lists", pm(handleManageSubscriberListsByQuery, "subscribers:manage"))
+ api.GET("/api/subscribers/export",
+ pm(middleware.GzipWithConfig(middleware.GzipConfig{Level: 9})(handleExportSubscribers), "subscribers:get_all", "subscribers:get"))
+
+ api.GET("/api/import/subscribers", pm(handleGetImportSubscribers, "subscribers:import"))
+ api.GET("/api/import/subscribers/logs", pm(handleGetImportSubscriberStats, "subscribers:import"))
+ api.POST("/api/import/subscribers", pm(handleImportSubscribers, "subscribers:import"))
+ api.DELETE("/api/import/subscribers", pm(handleStopImportSubscribers, "subscribers:import"))
+
+ // Individual list permissions are applied directly within handleGetLists.
+ api.GET("/api/lists", handleGetLists)
+ api.GET("/api/lists/:id", listPerm(handleGetList))
+ api.POST("/api/lists", pm(handleCreateList, "lists:manage_all"))
+ api.PUT("/api/lists/:id", listPerm(handleUpdateList))
+ api.DELETE("/api/lists/:id", listPerm(handleDeleteLists))
+
+ api.GET("/api/campaigns", pm(handleGetCampaigns, "campaigns:get"))
+ api.GET("/api/campaigns/running/stats", pm(handleGetRunningCampaignStats, "campaigns:get"))
+ api.GET("/api/campaigns/:id", pm(handleGetCampaign, "campaigns:get"))
+ api.GET("/api/campaigns/analytics/:type", pm(handleGetCampaignViewAnalytics, "campaigns:get_analytics"))
+ api.GET("/api/campaigns/:id/preview", pm(handlePreviewCampaign, "campaigns:get"))
+ api.POST("/api/campaigns/:id/preview", pm(handlePreviewCampaign, "campaigns:get"))
+ api.POST("/api/campaigns/:id/content", pm(handleCampaignContent, "campaigns:manage"))
+ api.POST("/api/campaigns/:id/text", pm(handlePreviewCampaign, "campaigns:manage"))
+ api.POST("/api/campaigns/:id/test", pm(handleTestCampaign, "campaigns:manage"))
+ api.POST("/api/campaigns", pm(handleCreateCampaign, "campaigns:manage"))
+ api.PUT("/api/campaigns/:id", pm(handleUpdateCampaign, "campaigns:manage"))
+ api.PUT("/api/campaigns/:id/status", pm(handleUpdateCampaignStatus, "campaigns:manage"))
+ api.PUT("/api/campaigns/:id/archive", pm(handleUpdateCampaignArchive, "campaigns:manage"))
+ api.DELETE("/api/campaigns/:id", pm(handleDeleteCampaign, "campaigns:manage"))
+
+ api.GET("/api/media", pm(handleGetMedia, "media:get"))
+ api.GET("/api/media/:id", pm(handleGetMedia, "media:get"))
+ api.POST("/api/media", pm(handleUploadMedia, "media:manage"))
+ api.DELETE("/api/media/:id", pm(handleDeleteMedia, "media:manage"))
+
+ api.GET("/api/templates", pm(handleGetTemplates, "templates:get"))
+ api.GET("/api/templates/:id", pm(handleGetTemplates, "templates:get"))
+ api.GET("/api/templates/:id/preview", pm(handlePreviewTemplate, "templates:get"))
+ api.POST("/api/templates/preview", pm(handlePreviewTemplate, "templates:get"))
+ api.POST("/api/templates", pm(handleCreateTemplate, "templates:manage"))
+ api.PUT("/api/templates/:id", pm(handleUpdateTemplate, "templates:manage"))
+ api.PUT("/api/templates/:id/default", pm(handleTemplateSetDefault, "templates:manage"))
+ api.DELETE("/api/templates/:id", pm(handleDeleteTemplate, "templates:manage"))
+
+ api.DELETE("/api/maintenance/subscribers/:type", pm(handleGCSubscribers, "settings:maintain"))
+ api.DELETE("/api/maintenance/analytics/:type", pm(handleGCCampaignAnalytics, "settings:maintain"))
+ api.DELETE("/api/maintenance/subscriptions/unconfirmed", pm(handleGCSubscriptions, "settings:maintain"))
+
+ api.POST("/api/tx", pm(handleSendTxMessage, "tx:send"))
+
+ api.GET("/api/profile", handleGetUserProfile)
+ api.PUT("/api/profile", handleUpdateUserProfile)
+ api.GET("/api/users", pm(handleGetUsers, "users:get"))
+ api.GET("/api/users/:id", pm(handleGetUsers, "users:get"))
+ api.POST("/api/users", pm(handleCreateUser, "users:manage"))
+ api.PUT("/api/users/:id", pm(handleUpdateUser, "users:manage"))
+ api.DELETE("/api/users", pm(handleDeleteUsers, "users:manage"))
+ api.DELETE("/api/users/:id", pm(handleDeleteUsers, "users:manage"))
+ api.POST("/api/logout", handleLogout)
+
+ api.GET("/api/roles/users", pm(handleGetUserRoles, "roles:get"))
+ api.GET("/api/roles/lists", pm(handleGeListRoles, "roles:get"))
+ api.POST("/api/roles/users", pm(handleCreateUserRole, "roles:manage"))
+ api.POST("/api/roles/lists", pm(handleCreateListRole, "roles:manage"))
+ api.PUT("/api/roles/users/:id", pm(handleUpdateUserRole, "roles:manage"))
+ api.PUT("/api/roles/lists/:id", pm(handleUpdateListRole, "roles:manage"))
+ api.DELETE("/api/roles/:id", pm(handleDeleteRole, "roles:manage"))
if app.constants.BounceWebhooksEnabled {
// Private authenticated bounce endpoint.
- g.POST("/webhooks/bounce", handleBounceWebhook)
+ api.POST("/webhooks/bounce", pm(handleBounceWebhook, "webhooks:post_bounce"))
// Public bounce endpoints for webservices like SES.
- e.POST("/webhooks/service/:service", handleBounceWebhook)
+ p.POST("/webhooks/service/:service", handleBounceWebhook)
}
+ // =================================================================
+ // Public API endpoints.
+
+ // Landing page.
+ p.GET("/", func(c echo.Context) error {
+ return c.Render(http.StatusOK, "home", publicTpl{Title: "listmonk"})
+ })
+
+ // Public admin endpoints (login page, OIDC endpoints).
+ p.GET(path.Join(uriAdmin, "/login"), handleLoginPage)
+ p.POST(path.Join(uriAdmin, "/login"), handleLoginPage)
+
+ if app.constants.Security.OIDC.Enabled {
+ p.POST("/auth/oidc", handleOIDCLogin)
+ p.GET("/auth/oidc", handleOIDCFinish)
+ }
+
+ // Public APIs.
+ p.GET("/api/public/lists", handleGetPublicLists)
+ p.POST("/api/public/subscription", handlePublicSubscription)
+ if app.constants.EnablePublicArchive {
+ p.GET("/api/public/archive", handleGetCampaignArchives)
+ }
+
+ // /public/static/* file server is registered in initHTTPServer().
// Public subscriber facing views.
- e.GET("/subscription/form", handleSubscriptionFormPage)
- e.POST("/subscription/form", handleSubscriptionForm)
- e.GET("/subscription/:campUUID/:subUUID", noIndex(validateUUID(subscriberExists(handleSubscriptionPage),
+ p.GET("/subscription/form", handleSubscriptionFormPage)
+ p.POST("/subscription/form", handleSubscriptionForm)
+ p.GET("/subscription/:campUUID/:subUUID", noIndex(validateUUID(subscriberExists(handleSubscriptionPage),
"campUUID", "subUUID")))
- e.POST("/subscription/:campUUID/:subUUID", validateUUID(subscriberExists(handleSubscriptionPage),
+ p.POST("/subscription/:campUUID/:subUUID", validateUUID(subscriberExists(handleSubscriptionPrefs),
"campUUID", "subUUID"))
- e.GET("/subscription/optin/:subUUID", noIndex(validateUUID(subscriberExists(handleOptinPage), "subUUID")))
- e.POST("/subscription/optin/:subUUID", validateUUID(subscriberExists(handleOptinPage), "subUUID"))
- e.POST("/subscription/export/:subUUID", validateUUID(subscriberExists(handleSelfExportSubscriberData),
+ p.GET("/subscription/optin/:subUUID", noIndex(validateUUID(subscriberExists(handleOptinPage), "subUUID")))
+ p.POST("/subscription/optin/:subUUID", validateUUID(subscriberExists(handleOptinPage), "subUUID"))
+ p.POST("/subscription/export/:subUUID", validateUUID(subscriberExists(handleSelfExportSubscriberData),
"subUUID"))
- e.POST("/subscription/wipe/:subUUID", validateUUID(subscriberExists(handleWipeSubscriberData),
+ p.POST("/subscription/wipe/:subUUID", validateUUID(subscriberExists(handleWipeSubscriberData),
"subUUID"))
- e.GET("/link/:linkUUID/:campUUID/:subUUID", noIndex(validateUUID(handleLinkRedirect,
+ p.GET("/link/:linkUUID/:campUUID/:subUUID", noIndex(validateUUID(handleLinkRedirect,
"linkUUID", "campUUID", "subUUID")))
- e.GET("/campaign/:campUUID/:subUUID", noIndex(validateUUID(handleViewCampaignMessage,
+ p.GET("/campaign/:campUUID/:subUUID", noIndex(validateUUID(handleViewCampaignMessage,
"campUUID", "subUUID")))
- e.GET("/campaign/:campUUID/:subUUID/px.png", noIndex(validateUUID(handleRegisterCampaignView,
+ p.GET("/campaign/:campUUID/:subUUID/px.png", noIndex(validateUUID(handleRegisterCampaignView,
"campUUID", "subUUID")))
+
+ if app.constants.EnablePublicArchive {
+ p.GET("/archive", handleCampaignArchivesPage)
+ p.GET("/archive.xml", handleGetCampaignArchivesFeed)
+ p.GET("/archive/:id", handleCampaignArchivePage)
+ p.GET("/archive/latest", handleCampaignArchivePageLatest)
+ }
+
+ p.GET("/public/custom.css", serveCustomAppearance("public.custom_css"))
+ p.GET("/public/custom.js", serveCustomAppearance("public.custom_js"))
+
// Public health API endpoint.
- e.GET("/health", handleHealthCheck)
+ p.GET("/health", handleHealthCheck)
+
+ // 404 pages.
+ p.RouteNotFound("/*", func(c echo.Context) error {
+ return c.Render(http.StatusNotFound, tplMessage,
+ makeMsgTpl("404 - "+app.i18n.T("public.notFoundTitle"), "", ""))
+ })
+ p.RouteNotFound("/api/*", func(c echo.Context) error {
+ return echo.NewHTTPError(http.StatusNotFound, "404 unknown endpoint")
+ })
+ p.RouteNotFound("/admin/*", func(c echo.Context) error {
+ return echo.NewHTTPError(http.StatusNotFound, "404 page not found")
+ })
}
-// handleIndex is the root handler that renders the Javascript frontend.
-func handleIndexPage(c echo.Context) error {
+// handleAdminPage is the root handler that renders the Javascript admin frontend.
+func handleAdminPage(c echo.Context) error {
app := c.Get("app").(*App)
- b, err := app.fs.Read("/frontend/index.html")
+ b, err := app.fs.Read(path.Join(uriAdmin, "/index.html"))
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, err.Error())
}
- c.Response().Header().Set("Content-Type", "text/html")
- return c.String(http.StatusOK, string(b))
+ b = bytes.ReplaceAll(b, []byte("asset_version"), []byte(app.constants.AssetVersion))
+
+ return c.HTMLBlob(http.StatusOK, b)
}
// handleHealthCheck is a healthcheck endpoint that returns a 200 response.
@@ -188,21 +312,37 @@ func handleHealthCheck(c echo.Context) error {
return c.JSON(http.StatusOK, okResp{true})
}
-// basicAuth middleware does an HTTP BasicAuth authentication for admin handlers.
-func basicAuth(username, password string, c echo.Context) (bool, error) {
- app := c.Get("app").(*App)
+// serveCustomAppearance serves the given custom CSS/JS appearance blob
+// meant for customizing public and admin pages from the admin settings UI.
+func serveCustomAppearance(name string) echo.HandlerFunc {
+ return func(c echo.Context) error {
+ var (
+ app = c.Get("app").(*App)
- // Auth is disabled.
- if len(app.constants.AdminUsername) == 0 &&
- len(app.constants.AdminPassword) == 0 {
- return true, nil
- }
+ out []byte
+ hdr string
+ )
+
+ switch name {
+ case "admin.custom_css":
+ out = app.constants.Appearance.AdminCSS
+ hdr = "text/css; charset=utf-8"
+
+ case "admin.custom_js":
+ out = app.constants.Appearance.AdminJS
+ hdr = "application/javascript; charset=utf-8"
+
+ case "public.custom_css":
+ out = app.constants.Appearance.PublicCSS
+ hdr = "text/css; charset=utf-8"
+
+ case "public.custom_js":
+ out = app.constants.Appearance.PublicJS
+ hdr = "application/javascript; charset=utf-8"
+ }
- if subtle.ConstantTimeCompare([]byte(username), app.constants.AdminUsername) == 1 &&
- subtle.ConstantTimeCompare([]byte(password), app.constants.AdminPassword) == 1 {
- return true, nil
+ return c.Blob(http.StatusOK, hdr, out)
}
- return false, nil
}
// validateUUID middleware validates the UUID string format for a given set of params.
@@ -230,19 +370,17 @@ func subscriberExists(next echo.HandlerFunc, params ...string) echo.HandlerFunc
subUUID = c.Param("subUUID")
)
- var exists bool
- if err := app.queries.SubscriberExists.Get(&exists, 0, subUUID); err != nil {
+ if _, err := app.core.GetSubscriber(0, subUUID, ""); err != nil {
+ if er, ok := err.(*echo.HTTPError); ok && er.Code == http.StatusBadRequest {
+ return c.Render(http.StatusNotFound, tplMessage,
+ makeMsgTpl(app.i18n.T("public.notFoundTitle"), "", er.Message.(string)))
+ }
+
app.log.Printf("error checking subscriber existence: %v", err)
return c.Render(http.StatusInternalServerError, tplMessage,
- makeMsgTpl(app.i18n.T("public.errorTitle"), "",
- app.i18n.T("public.errorProcessingRequest")))
+ makeMsgTpl(app.i18n.T("public.errorTitle"), "", app.i18n.T("public.errorProcessingRequest")))
}
- if !exists {
- return c.Render(http.StatusNotFound, tplMessage,
- makeMsgTpl(app.i18n.T("public.notFoundTitle"), "",
- app.i18n.T("public.subNotFound")))
- }
return next(c)
}
}
@@ -254,54 +392,3 @@ func noIndex(next echo.HandlerFunc, params ...string) echo.HandlerFunc {
return next(c)
}
}
-
-// getPagination takes form values and extracts pagination values from it.
-func getPagination(q url.Values, perPage int) pagination {
- var (
- page, _ = strconv.Atoi(q.Get("page"))
- pp = q.Get("per_page")
- )
-
- if pp == "all" {
- // No limit.
- perPage = 0
- } else {
- ppi, _ := strconv.Atoi(pp)
- if ppi > 0 {
- perPage = ppi
- }
- }
-
- if page < 1 {
- page = 0
- } else {
- page--
- }
-
- return pagination{
- Page: page + 1,
- PerPage: perPage,
- Offset: page * perPage,
- Limit: perPage,
- }
-}
-
-// copyEchoCtx returns a copy of the the current echo.Context in a request
-// with the given params set for the active handler to proxy the request
-// to another handler without mutating its context.
-func copyEchoCtx(c echo.Context, params map[string]string) echo.Context {
- var (
- keys = make([]string, 0, len(params))
- vals = make([]string, 0, len(params))
- )
- for k, v := range params {
- keys = append(keys, k)
- vals = append(vals, v)
- }
-
- b := c.Echo().NewContext(c.Request(), c.Response())
- b.Set("app", c.Get("app").(*App))
- b.SetParamNames(keys...)
- b.SetParamValues(vals...)
- return b
-}
diff --git a/cmd/i18n.go b/cmd/i18n.go
index fc963d1ba..8b9dc74e9 100644
--- a/cmd/i18n.go
+++ b/cmd/i18n.go
@@ -8,7 +8,7 @@ import (
"github.com/knadh/listmonk/internal/i18n"
"github.com/knadh/stuffbin"
- "github.com/labstack/echo"
+ "github.com/labstack/echo/v4"
)
type i18nLang struct {
diff --git a/cmd/import.go b/cmd/import.go
index a3fcf83d2..f4ae7eca0 100644
--- a/cmd/import.go
+++ b/cmd/import.go
@@ -3,13 +3,13 @@ package main
import (
"encoding/json"
"io"
- "io/ioutil"
"net/http"
+ "os"
"strings"
"github.com/knadh/listmonk/internal/subimporter"
"github.com/knadh/listmonk/models"
- "github.com/labstack/echo"
+ "github.com/labstack/echo/v4"
)
// handleImportSubscribers handles the uploading and bulk importing of
@@ -22,7 +22,7 @@ func handleImportSubscribers(c echo.Context) error {
return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("import.alreadyRunning"))
}
- // Unmarsal the JSON params.
+ // Unmarshal the JSON params.
var opt subimporter.SessionOpt
if err := json.Unmarshal([]byte(c.FormValue("params")), &opt); err != nil {
return echo.NewHTTPError(http.StatusBadRequest,
@@ -66,7 +66,7 @@ func handleImportSubscribers(c echo.Context) error {
}
defer src.Close()
- out, err := ioutil.TempFile("", "listmonk")
+ out, err := os.CreateTemp("", "listmonk")
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError,
app.i18n.Ts("import.errorCopyingFile", "error", err.Error()))
@@ -95,7 +95,7 @@ func handleImportSubscribers(c echo.Context) error {
// keeping the global import state (failed / successful) etc. across
// multiple files becomes complex. Instead, it's just easier for the
// end user to concat multiple CSVs (if there are multiple in the first)
- // place and uploada as one in the first place.
+ // place and upload as one in the first place.
dir, files, err := impSess.ExtractZIP(out.Name(), 1)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError,
diff --git a/cmd/init.go b/cmd/init.go
index 06a1c19ca..cdd94009d 100644
--- a/cmd/init.go
+++ b/cmd/init.go
@@ -1,40 +1,53 @@
package main
import (
+ "bytes"
+ "crypto/md5"
+ "database/sql"
"encoding/json"
+ "errors"
"fmt"
"html/template"
+ "net/http"
"os"
"path"
"path/filepath"
+ "runtime"
"strings"
"syscall"
"time"
+ "github.com/Masterminds/sprig/v3"
+ "github.com/gdgvda/cron"
"github.com/jmoiron/sqlx"
"github.com/jmoiron/sqlx/types"
"github.com/knadh/goyesql/v2"
goyesqlx "github.com/knadh/goyesql/v2/sqlx"
- "github.com/knadh/koanf"
"github.com/knadh/koanf/maps"
"github.com/knadh/koanf/parsers/toml"
"github.com/knadh/koanf/providers/confmap"
"github.com/knadh/koanf/providers/file"
"github.com/knadh/koanf/providers/posflag"
+ "github.com/knadh/koanf/v2"
+ "github.com/knadh/listmonk/internal/auth"
"github.com/knadh/listmonk/internal/bounce"
"github.com/knadh/listmonk/internal/bounce/mailbox"
+ "github.com/knadh/listmonk/internal/captcha"
+ "github.com/knadh/listmonk/internal/core"
"github.com/knadh/listmonk/internal/i18n"
"github.com/knadh/listmonk/internal/manager"
"github.com/knadh/listmonk/internal/media"
"github.com/knadh/listmonk/internal/media/providers/filesystem"
"github.com/knadh/listmonk/internal/media/providers/s3"
- "github.com/knadh/listmonk/internal/messenger"
"github.com/knadh/listmonk/internal/messenger/email"
"github.com/knadh/listmonk/internal/messenger/postback"
"github.com/knadh/listmonk/internal/subimporter"
+ "github.com/knadh/listmonk/models"
"github.com/knadh/stuffbin"
- "github.com/labstack/echo"
+ "github.com/labstack/echo/v4"
+ "github.com/lib/pq"
flag "github.com/spf13/pflag"
+ "gopkg.in/volatiletech/null.v6"
)
const (
@@ -43,34 +56,77 @@ const (
// constants contains static, constant config values required by the app.
type constants struct {
- RootURL string `koanf:"root_url"`
- LogoURL string `koanf:"logo_url"`
- FaviconURL string `koanf:"favicon_url"`
- FromEmail string `koanf:"from_email"`
- NotifyEmails []string `koanf:"notify_emails"`
- EnablePublicSubPage bool `koanf:"enable_public_subscription_page"`
- Lang string `koanf:"lang"`
- DBBatchSize int `koanf:"batch_size"`
- Privacy struct {
+ SiteName string `koanf:"site_name"`
+ RootURL string `koanf:"root_url"`
+ LogoURL string `koanf:"logo_url"`
+ FaviconURL string `koanf:"favicon_url"`
+ LoginURL string `koanf:"login_url"`
+ FromEmail string `koanf:"from_email"`
+ NotifyEmails []string `koanf:"notify_emails"`
+ EnablePublicSubPage bool `koanf:"enable_public_subscription_page"`
+ EnablePublicArchive bool `koanf:"enable_public_archive"`
+ EnablePublicArchiveRSSContent bool `koanf:"enable_public_archive_rss_content"`
+ SendOptinConfirmation bool `koanf:"send_optin_confirmation"`
+ Lang string `koanf:"lang"`
+ DBBatchSize int `koanf:"batch_size"`
+ Privacy struct {
IndividualTracking bool `koanf:"individual_tracking"`
+ AllowPreferences bool `koanf:"allow_preferences"`
AllowBlocklist bool `koanf:"allow_blocklist"`
AllowExport bool `koanf:"allow_export"`
AllowWipe bool `koanf:"allow_wipe"`
+ RecordOptinIP bool `koanf:"record_optin_ip"`
+ UnsubHeader bool `koanf:"unsubscribe_header"`
Exportable map[string]bool `koanf:"-"`
+ DomainBlocklist []string `koanf:"-"`
} `koanf:"privacy"`
- AdminUsername []byte `koanf:"admin_username"`
- AdminPassword []byte `koanf:"admin_password"`
-
+ Security struct {
+ OIDC struct {
+ Enabled bool `koanf:"enabled"`
+ Provider string `koanf:"provider_url"`
+ ClientID string `koanf:"client_id"`
+ ClientSecret string `koanf:"client_secret"`
+ } `koanf:"oidc"`
+
+ EnableCaptcha bool `koanf:"enable_captcha"`
+ CaptchaKey string `koanf:"captcha_key"`
+ CaptchaSecret string `koanf:"captcha_secret"`
+ } `koanf:"security"`
+
+ Appearance struct {
+ AdminCSS []byte `koanf:"admin.custom_css"`
+ AdminJS []byte `koanf:"admin.custom_js"`
+ PublicCSS []byte `koanf:"public.custom_css"`
+ PublicJS []byte `koanf:"public.custom_js"`
+ }
+
+ HasLegacyUser bool
UnsubURL string
LinkTrackURL string
ViewTrackURL string
OptinURL string
MessageURL string
- MediaProvider string
+ ArchiveURL string
+ AssetVersion string
+
+ MediaUpload struct {
+ Provider string
+ Extensions []string
+ }
+
+ BounceWebhooksEnabled bool
+ BounceSESEnabled bool
+ BounceSendgridEnabled bool
+ BouncePostmarkEnabled bool
+ BounceForwardemailEnabled bool
+
+ PermissionsRaw json.RawMessage
+ Permissions map[string]struct{}
+}
- BounceWebhooksEnabled bool
- BounceSESEnabled bool
- BounceSendgridEnabled bool
+type notifTpls struct {
+ tpls *template.Template
+ contentType string
}
func initFlags() {
@@ -85,13 +141,14 @@ func initFlags() {
f.StringSlice("config", []string{"config.toml"},
"path to one or more config files (will be merged in order)")
f.Bool("install", false, "setup database (first time)")
- f.Bool("idempotent", false, "make --install run only if the databse isn't already setup")
+ f.Bool("idempotent", false, "make --install run only if the database isn't already setup")
f.Bool("upgrade", false, "upgrade database to the current version")
- f.Bool("version", false, "current version of the build")
+ f.Bool("version", false, "show current version of the build")
f.Bool("new-config", false, "generate sample config file")
f.String("static-dir", "", "(optional) path to directory with static files")
f.String("i18n-dir", "", "(optional) path to directory with i18n language files")
f.Bool("yes", false, "assume 'yes' to prompts during --install/upgrade")
+ f.Bool("passive", false, "run in passive mode where campaigns are not processed")
if err := f.Parse(os.Args[1:]); err != nil {
lo.Fatalf("error loading flags: %v", err)
}
@@ -109,13 +166,13 @@ func initConfigFiles(files []string, ko *koanf.Koanf) {
if os.IsNotExist(err) {
lo.Fatal("config file not found. If there isn't one yet, run --new-config to generate one.")
}
- lo.Fatalf("error loadng config from file: %v.", err)
+ lo.Fatalf("error loading config from file: %v.", err)
}
}
}
// initFileSystem initializes the stuffbin FileSystem to provide
-// access to bunded static assets to the app.
+// access to bundled static assets to the app.
func initFS(appDir, frontendDir, staticDir, i18nDir string) stuffbin.FileSystem {
var (
// stuffbin real_path:virtual_alias paths to map local assets on disk
@@ -126,12 +183,13 @@ func initFS(appDir, frontendDir, staticDir, i18nDir string) stuffbin.FileSystem
"./config.toml.sample:config.toml.sample",
"./queries.sql:queries.sql",
"./schema.sql:schema.sql",
+ "./permissions.json:permissions.json",
}
frontendFiles = []string{
- // The app's frontend assets are accessible at /frontend/js/* during runtime.
- // These paths are joined with frontendDir.
- "./:/frontend",
+ // Admin frontend's static assets accessible at /admin/* during runtime.
+ // These paths are sourced from frontendDir.
+ "./:/admin",
}
staticFiles = []string{
@@ -146,15 +204,15 @@ func initFS(appDir, frontendDir, staticDir, i18nDir string) stuffbin.FileSystem
}
)
- // Get the executable's path.
- path, err := os.Executable()
+ // Get the executable's execPath.
+ execPath, err := os.Executable()
if err != nil {
lo.Fatalf("error getting executable path: %v", err)
}
// Load embedded files in the executable.
hasEmbed := true
- fs, err := stuffbin.UnStuff(path)
+ fs, err := stuffbin.UnStuff(execPath)
if err != nil {
hasEmbed = false
@@ -182,7 +240,7 @@ func initFS(appDir, frontendDir, staticDir, i18nDir string) stuffbin.FileSystem
// Default dir in cwd.
i18nDir = "i18n"
}
- lo.Printf("will load i18n files from: %v", i18nDir)
+ lo.Printf("loading i18n files from: %v", i18nDir)
files = append(files, joinFSPaths(i18nDir, i18nFiles)...)
}
@@ -190,8 +248,19 @@ func initFS(appDir, frontendDir, staticDir, i18nDir string) stuffbin.FileSystem
if staticDir == "" {
// Default dir in cwd.
staticDir = "static"
+ } else {
+ // There is a custom static directory. Any paths that aren't in it, exclude.
+ sf := []string{}
+ for _, def := range staticFiles {
+ s := strings.Split(def, ":")[0]
+ if _, err := os.Stat(path.Join(staticDir, s)); err == nil {
+ sf = append(sf, def)
+ }
+ }
+ staticFiles = sf
}
- lo.Printf("will load static files from: %v", staticDir)
+
+ lo.Printf("loading static files from: %v", staticDir)
files = append(files, joinFSPaths(staticDir, staticFiles)...)
}
@@ -216,22 +285,38 @@ func initFS(appDir, frontendDir, staticDir, i18nDir string) stuffbin.FileSystem
// initDB initializes the main DB connection pool and parse and loads the app's
// SQL queries into a prepared query map.
func initDB() *sqlx.DB {
- var dbCfg dbConf
- if err := ko.Unmarshal("db", &dbCfg); err != nil {
+ var c struct {
+ Host string `koanf:"host"`
+ Port int `koanf:"port"`
+ User string `koanf:"user"`
+ Password string `koanf:"password"`
+ DBName string `koanf:"database"`
+ SSLMode string `koanf:"ssl_mode"`
+ Params string `koanf:"params"`
+ MaxOpen int `koanf:"max_open"`
+ MaxIdle int `koanf:"max_idle"`
+ MaxLifetime time.Duration `koanf:"max_lifetime"`
+ }
+ if err := ko.Unmarshal("db", &c); err != nil {
lo.Fatalf("error loading db config: %v", err)
}
- lo.Printf("connecting to db: %s:%d/%s", dbCfg.Host, dbCfg.Port, dbCfg.DBName)
- db, err := connectDB(dbCfg)
+ lo.Printf("connecting to db: %s:%d/%s", c.Host, c.Port, c.DBName)
+ db, err := sqlx.Connect("postgres",
+ fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=%s %s", c.Host, c.Port, c.User, c.Password, c.DBName, c.SSLMode, c.Params))
if err != nil {
lo.Fatalf("error connecting to DB: %v", err)
}
+
+ db.SetMaxOpenConns(c.MaxOpen)
+ db.SetMaxIdleConns(c.MaxIdle)
+ db.SetConnMaxLifetime(c.MaxLifetime)
+
return db
}
-// initQueries loads named SQL queries from the queries file and optionally
-// prepares them.
-func initQueries(sqlFile string, db *sqlx.DB, fs stuffbin.FileSystem, prepareQueries bool) (goyesql.Queries, *Queries) {
+// readQueries reads named SQL queries from the SQL queries file into a query map.
+func readQueries(sqlFile string, db *sqlx.DB, fs stuffbin.FileSystem) goyesql.Queries {
// Load SQL queries.
qB, err := fs.Read(sqlFile)
if err != nil {
@@ -242,24 +327,52 @@ func initQueries(sqlFile string, db *sqlx.DB, fs stuffbin.FileSystem, prepareQue
lo.Fatalf("error parsing SQL queries: %v", err)
}
- if !prepareQueries {
- return qMap, nil
+ return qMap
+}
+
+// prepareQueries queries prepares a query map and returns a *Queries
+func prepareQueries(qMap goyesql.Queries, db *sqlx.DB, ko *koanf.Koanf) *models.Queries {
+ var (
+ countQuery = "get-campaign-analytics-counts"
+ linkSel = "*"
+ )
+ if ko.Bool("privacy.individual_tracking") {
+ countQuery = "get-campaign-analytics-unique-counts"
+ linkSel = "DISTINCT subscriber_id"
}
- // Prepare queries.
- var q Queries
+ // These don't exist in the SQL file but are in the queries struct to be prepared.
+ qMap["get-campaign-view-counts"] = &goyesql.Query{
+ Query: fmt.Sprintf(qMap[countQuery].Query, "campaign_views"),
+ Tags: map[string]string{"name": "get-campaign-view-counts"},
+ }
+ qMap["get-campaign-click-counts"] = &goyesql.Query{
+ Query: fmt.Sprintf(qMap[countQuery].Query, "link_clicks"),
+ Tags: map[string]string{"name": "get-campaign-click-counts"},
+ }
+ qMap["get-campaign-link-counts"].Query = fmt.Sprintf(qMap["get-campaign-link-counts"].Query, linkSel)
+
+ // Scan and prepare all queries.
+ var q models.Queries
if err := goyesqlx.ScanToStruct(&q, qMap, db.Unsafe()); err != nil {
lo.Fatalf("error preparing SQL queries: %v", err)
}
- return qMap, &q
+ return &q
}
-// initSettings loads settings from the DB.
-func initSettings(q *sqlx.Stmt) {
+// initSettings loads settings from the DB into the given Koanf map.
+func initSettings(query string, db *sqlx.DB, ko *koanf.Koanf) {
var s types.JSONText
- if err := q.Get(&s); err != nil {
- lo.Fatalf("error reading settings from DB: %s", pqErrMsg(err))
+ if err := db.Get(&s, query); err != nil {
+ msg := err.Error()
+ if err, ok := err.(*pq.Error); ok {
+ if err.Detail != "" {
+ msg = fmt.Sprintf("%s. %s", err, err.Detail)
+ }
+ }
+
+ lo.Fatalf("error reading settings from DB: %s", msg)
}
// Setting keys are dot separated, eg: app.favicon_url. Unflatten them into
@@ -280,13 +393,23 @@ func initConstants() *constants {
lo.Fatalf("error loading app config: %v", err)
}
if err := ko.Unmarshal("privacy", &c.Privacy); err != nil {
- lo.Fatalf("error loading app config: %v", err)
+ lo.Fatalf("error loading app.privacy config: %v", err)
+ }
+ if err := ko.Unmarshal("security", &c.Security); err != nil {
+ lo.Fatalf("error loading app.security config: %v", err)
+ }
+
+ if err := ko.UnmarshalWithConf("appearance", &c.Appearance, koanf.UnmarshalConf{FlatPaths: true}); err != nil {
+ lo.Fatalf("error loading app.appearance config: %v", err)
}
c.RootURL = strings.TrimRight(c.RootURL, "/")
+ c.LoginURL = path.Join(uriAdmin, "/login")
c.Lang = ko.String("app.lang")
c.Privacy.Exportable = maps.StringSliceToLookupMap(ko.Strings("privacy.exportable"))
- c.MediaProvider = ko.String("upload.provider")
+ c.MediaUpload.Provider = ko.String("upload.provider")
+ c.MediaUpload.Extensions = ko.Strings("upload.extensions")
+ c.Privacy.DomainBlocklist = ko.Strings("privacy.domain_blocklist")
// Static URLS.
// url.com/subscription/{campaign_uuid}/{subscriber_uuid}
@@ -301,12 +424,44 @@ func initConstants() *constants {
// url.com/link/{campaign_uuid}/{subscriber_uuid}
c.MessageURL = fmt.Sprintf("%s/campaign/%%s/%%s", c.RootURL)
+ // url.com/archive
+ c.ArchiveURL = c.RootURL + "/archive"
+
// url.com/campaign/{campaign_uuid}/{subscriber_uuid}/px.png
c.ViewTrackURL = fmt.Sprintf("%s/campaign/%%s/%%s/px.png", c.RootURL)
c.BounceWebhooksEnabled = ko.Bool("bounce.webhooks_enabled")
c.BounceSESEnabled = ko.Bool("bounce.ses_enabled")
c.BounceSendgridEnabled = ko.Bool("bounce.sendgrid_enabled")
+ c.BouncePostmarkEnabled = ko.Bool("bounce.postmark.enabled")
+ c.BounceForwardemailEnabled = ko.Bool("bounce.forwardemail.enabled")
+ c.HasLegacyUser = ko.Exists("app.admin_username") || ko.Exists("app.admin_password")
+
+ b := md5.Sum([]byte(time.Now().String()))
+ c.AssetVersion = fmt.Sprintf("%x", b)[0:10]
+
+ pm, err := fs.Read("/permissions.json")
+ if err != nil {
+ lo.Fatalf("error reading permissions file: %v", err)
+ }
+ c.PermissionsRaw = pm
+
+ // Make a lookup map of permissions.
+ permGroups := []struct {
+ Group string `json:"group"`
+ Permissions []string `json:"permissions"`
+ }{}
+ if err := json.Unmarshal(pm, &permGroups); err != nil {
+ lo.Fatalf("error loading permissions file: %v", err)
+ }
+
+ c.Permissions = map[string]struct{}{}
+ for _, group := range permGroups {
+ for _, g := range group.Permissions {
+ c.Permissions[g] = struct{}{}
+ }
+ }
+
return &c
}
@@ -327,16 +482,13 @@ func initI18n(lang string, fs stuffbin.FileSystem) *i18n.I18n {
}
// initCampaignManager initializes the campaign manager.
-func initCampaignManager(q *Queries, cs *constants, app *App) *manager.Manager {
+func initCampaignManager(q *models.Queries, cs *constants, app *App) *manager.Manager {
campNotifCB := func(subject string, data interface{}) error {
- return app.sendNotification(cs.NotifyEmails, subject, notifTplCampaign, data)
+ return app.sendNotification(cs.NotifyEmails, subject, notifTplCampaign, data, nil)
}
- if ko.Int("app.concurrency") < 1 {
- lo.Fatal("app.concurrency should be at least 1")
- }
- if ko.Int("app.message_rate") < 1 {
- lo.Fatal("app.message_rate should be at least 1")
+ if ko.Bool("passive") {
+ lo.Println("running in passive mode. won't process campaigns.")
}
return manager.New(manager.Config{
@@ -351,41 +503,60 @@ func initCampaignManager(q *Queries, cs *constants, app *App) *manager.Manager {
LinkTrackURL: cs.LinkTrackURL,
ViewTrackURL: cs.ViewTrackURL,
MessageURL: cs.MessageURL,
+ ArchiveURL: cs.ArchiveURL,
+ RootURL: cs.RootURL,
UnsubHeader: ko.Bool("privacy.unsubscribe_header"),
SlidingWindow: ko.Bool("app.message_sliding_window"),
SlidingWindowDuration: ko.Duration("app.message_sliding_window_duration"),
SlidingWindowRate: ko.Int("app.message_sliding_window_rate"),
- }, newManagerStore(q), campNotifCB, app.i18n, lo)
+ ScanInterval: time.Second * 5,
+ ScanCampaigns: !ko.Bool("passive"),
+ }, newManagerStore(q, app.core, app.media), campNotifCB, app.i18n, lo)
+}
+
+func initTxTemplates(m *manager.Manager, app *App) {
+ tpls, err := app.core.GetTemplates(models.TemplateTypeTx, false)
+ if err != nil {
+ lo.Fatalf("error loading transactional templates: %v", err)
+ }
+
+ for _, t := range tpls {
+ tpl := t
+ if err := tpl.Compile(app.manager.GenericTemplateFuncs()); err != nil {
+ lo.Printf("error compiling transactional template %d: %v", tpl.ID, err)
+ continue
+ }
+ m.CacheTpl(tpl.ID, &tpl)
+ }
}
// initImporter initializes the bulk subscriber importer.
-func initImporter(q *Queries, db *sqlx.DB, app *App) *subimporter.Importer {
+func initImporter(q *models.Queries, db *sqlx.DB, core *core.Core, app *App) *subimporter.Importer {
return subimporter.New(
subimporter.Options{
+ DomainBlocklist: app.constants.Privacy.DomainBlocklist,
UpsertStmt: q.UpsertSubscriber.Stmt,
BlocklistStmt: q.UpsertBlocklistSubscriber.Stmt,
UpdateListDateStmt: q.UpdateListsDate.Stmt,
NotifCB: func(subject string, data interface{}) error {
- app.sendNotification(app.constants.NotifyEmails, subject, notifTplImport, data)
+ // Refresh cached subscriber counts and stats.
+ core.RefreshMatViews(true)
+
+ app.sendNotification(app.constants.NotifyEmails, subject, notifTplImport, data, nil)
return nil
},
- }, db.DB)
+ }, db.DB, app.i18n)
}
-// initSMTPMessenger initializes the SMTP messenger.
-func initSMTPMessenger(m *manager.Manager) messenger.Messenger {
+// initSMTPMessenger initializes the combined and individual SMTP messengers.
+func initSMTPMessengers() []manager.Messenger {
var (
- mapKeys = ko.MapKeys("smtp")
- servers = make([]email.Server, 0, len(mapKeys))
+ servers = []email.Server{}
+ out = []manager.Messenger{}
)
- items := ko.Slices("smtp")
- if len(items) == 0 {
- lo.Fatalf("no SMTP servers found in config")
- }
-
- // Load the config for multipme SMTP servers.
- for _, item := range items {
+ // Load the config for multiple SMTP servers.
+ for _, item := range ko.Slices("smtp") {
if !item.Bool("enabled") {
continue
}
@@ -397,31 +568,45 @@ func initSMTPMessenger(m *manager.Manager) messenger.Messenger {
}
servers = append(servers, s)
- lo.Printf("loaded email (SMTP) messenger: %s@%s",
- item.String("username"), item.String("host"))
- }
- if len(servers) == 0 {
- lo.Fatalf("no SMTP servers enabled in settings")
+ lo.Printf("initialized email (SMTP) messenger: %s@%s", item.String("username"), item.String("host"))
+
+ // If the server has a name, initialize it as a standalone e-mail messenger
+ // allowing campaigns to select individual SMTPs. In the UI and config, it'll appear as `email / $name`.
+ if s.Name != "" {
+ msgr, err := email.New(s.Name, s)
+ if err != nil {
+ lo.Fatalf("error initializing e-mail messenger: %v", err)
+ }
+ out = append(out, msgr)
+ }
}
- // Initialize the e-mail messenger with multiple SMTP servers.
- msgr, err := email.New(servers...)
+ // Initialize the 'email' messenger with all SMTP servers.
+ msgr, err := email.New(email.MessengerName, servers...)
if err != nil {
- lo.Fatalf("error loading e-mail messenger: %v", err)
+ lo.Fatalf("error initializing e-mail messenger: %v", err)
}
- return msgr
+ // If it's just one server, return the default "email" messenger.
+ if len(servers) == 1 {
+ return []manager.Messenger{msgr}
+ }
+
+ // If there are multiple servers, prepend the group "email" to be the first one.
+ out = append([]manager.Messenger{msgr}, out...)
+
+ return out
}
// initPostbackMessengers initializes and returns all the enabled
// HTTP postback messenger backends.
-func initPostbackMessengers(m *manager.Manager) []messenger.Messenger {
+func initPostbackMessengers() []manager.Messenger {
items := ko.Slices("messengers")
if len(items) == 0 {
return nil
}
- var out []messenger.Messenger
+ var out []manager.Messenger
for _, item := range items {
if !item.Bool("enabled") {
continue
@@ -455,6 +640,7 @@ func initMediaStore() media.Store {
case "s3":
var o s3.Opt
ko.Unmarshal("upload.s3", &o)
+
up, err := s3.NewS3Store(o)
if err != nil {
lo.Fatalf("error initializing s3 upload provider %s", err)
@@ -469,7 +655,7 @@ func initMediaStore() media.Store {
o.RootURL = ko.String("app.root_url")
o.UploadPath = filepath.Clean(o.UploadPath)
o.UploadURI = filepath.Clean(o.UploadURI)
- up, err := filesystem.NewDiskStore(o)
+ up, err := filesystem.New(o)
if err != nil {
lo.Fatalf("error initializing filesystem upload provider %s", err)
}
@@ -484,37 +670,64 @@ func initMediaStore() media.Store {
// initNotifTemplates compiles and returns e-mail notification templates that are
// used for sending ad-hoc notifications to admins and subscribers.
-func initNotifTemplates(path string, fs stuffbin.FileSystem, i *i18n.I18n, cs *constants) *template.Template {
- // Register utility functions that the e-mail templates can use.
- funcs := template.FuncMap{
- "RootURL": func() string {
- return cs.RootURL
- },
- "LogoURL": func() string {
- return cs.LogoURL
- },
- "L": func() *i18n.I18n {
- return i
- },
+func initNotifTemplates(path string, fs stuffbin.FileSystem, i *i18n.I18n, cs *constants) *notifTpls {
+ tpls, err := stuffbin.ParseTemplatesGlob(initTplFuncs(i, cs), fs, "/static/email-templates/*.html")
+ if err != nil {
+ lo.Fatalf("error parsing e-mail notif templates: %v", err)
}
- tpl, err := stuffbin.ParseTemplatesGlob(funcs, fs, "/static/email-templates/*.html")
+ html, err := fs.Read("/static/email-templates/base.html")
if err != nil {
- lo.Fatalf("error parsing e-mail notif templates: %v", err)
+ lo.Fatalf("error reading static/email-templates/base.html: %v", err)
+ }
+
+ out := ¬ifTpls{
+ tpls: tpls,
+ contentType: models.CampaignContentTypeHTML,
+ }
+
+ // Determine whether the notification templates are HTML or plaintext.
+ // Copy the first few (arbitrary) bytes of the template and check if has the tag.
+ ln := 256
+ if len(html) < ln {
+ ln = len(html)
+ }
+ h := make([]byte, ln)
+ copy(h, html[0:ln])
+
+ if !bytes.Contains(bytes.ToLower(h), []byte(" 2 && len(password) > 6 {
+ u := models.User{
+ Username: username,
+ Password: null.String{Valid: true, String: password},
+ PasswordLogin: true,
+ HasPassword: true,
+ Status: models.UserStatusEnabled,
+ Type: models.UserTypeAPI,
+ }
+ u.UserRole.ID = auth.SuperAdminRoleID
+ a.CacheAPIUser(u)
+
+ lo.Println(`WARNING: Remove the admin_username and admin_password fields from the TOML configuration file. If you are using APIs, create and use new credentials. Users are now managed via the Admin -> Settings -> Users dashboard.`)
+ }
+
+ return hasUsers, a
+}
diff --git a/cmd/install.go b/cmd/install.go
index 0596b0d1c..3d61ca2a6 100644
--- a/cmd/install.go
+++ b/cmd/install.go
@@ -1,24 +1,21 @@
package main
import (
+ "encoding/json"
"fmt"
- "io/ioutil"
"os"
- "regexp"
"strings"
- "github.com/gofrs/uuid"
+ "github.com/gofrs/uuid/v5"
"github.com/jmoiron/sqlx"
- goyesqlx "github.com/knadh/goyesql/v2/sqlx"
"github.com/knadh/listmonk/models"
"github.com/knadh/stuffbin"
"github.com/lib/pq"
)
-// install runs the first time setup of creating and
-// migrating the database and creating the super user.
+// install runs the first time setup of setting up the database.
func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempotent bool) {
- qMap, _ := initQueries(queryFilePath, db, fs, false)
+ qMap := readQueries(queryFilePath, db, fs)
fmt.Println("")
if !idempotent {
@@ -32,7 +29,7 @@ func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempo
if prompt {
var ok string
- fmt.Print("continue (y/n)? ")
+ fmt.Print("continue (y/N)? ")
if _, err := fmt.Scanf("%s", &ok); err != nil {
lo.Fatalf("error reading value from terminal: %v", err)
}
@@ -61,12 +58,56 @@ func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempo
}
// Load the queries.
- var q Queries
- if err := goyesqlx.ScanToStruct(&q, qMap, db.Unsafe()); err != nil {
- lo.Fatalf("error loading SQL queries: %v", err)
- }
+ q := prepareQueries(qMap, db, ko)
// Sample list.
+ defList, optinList := installLists(q)
+
+ // Sample subscribers.
+ installSubs(defList, optinList, q)
+
+ // Templates.
+ campTplID, archiveTplID := installTemplates(q)
+
+ // Sample campaign.
+ installCampaign(campTplID, archiveTplID, q)
+
+ // Setup the user optionally.
+ var (
+ user = os.Getenv("LISTMONK_ADMIN_USER")
+ password = os.Getenv("LISTMONK_ADMIN_PASSWORD")
+ )
+ if user != "" && password != "" {
+ if len(user) < 3 || len(password) < 8 {
+ lo.Fatal("LISTMONK_ADMIN_USER should be min 3 chars and LISTMONK_ADMIN_PASSWORD should be min 8 chars")
+ }
+
+ lo.Printf("creating Super Admin user '%s'", user)
+ installUser(user, password, q)
+ } else {
+ lo.Printf("no Super Admin user created. Visit webpage to create user.")
+ }
+
+ lo.Printf("setup complete")
+ lo.Printf(`run the program and access the dashboard at %s`, ko.MustString("app.address"))
+}
+
+// installSchema executes the SQL schema and creates the necessary tables and types.
+func installSchema(curVer string, db *sqlx.DB, fs stuffbin.FileSystem) error {
+ q, err := fs.Read("/schema.sql")
+ if err != nil {
+ return err
+ }
+
+ if _, err := db.Exec(string(q)); err != nil {
+ return err
+ }
+
+ // Insert the current migration version.
+ return recordMigrationVersion(curVer, db)
+}
+
+func installLists(q *models.Queries) (int, int) {
var (
defList int
optinList int
@@ -77,6 +118,7 @@ func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempo
models.ListTypePrivate,
models.ListOptinSingle,
pq.StringArray{"test"},
+ "",
); err != nil {
lo.Fatalf("error creating list: %v", err)
}
@@ -86,17 +128,22 @@ func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempo
models.ListTypePublic,
models.ListOptinDouble,
pq.StringArray{"test"},
+ "",
); err != nil {
lo.Fatalf("error creating list: %v", err)
}
+ return defList, optinList
+}
+
+func installSubs(defListID, optinListID int, q *models.Queries) {
// Sample subscriber.
if _, err := q.UpsertSubscriber.Exec(
uuid.Must(uuid.NewV4()),
"john@example.com",
"John Doe",
`{"type": "known", "good": true, "city": "Bengaluru"}`,
- pq.Int64Array{int64(defList)},
+ pq.Int64Array{int64(defListID)},
models.SubscriptionStatusUnconfirmed,
true); err != nil {
lo.Fatalf("Error creating subscriber: %v", err)
@@ -106,29 +153,53 @@ func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempo
"anon@example.com",
"Anon Doe",
`{"type": "unknown", "good": true, "city": "Bengaluru"}`,
- pq.Int64Array{int64(optinList)},
+ pq.Int64Array{int64(optinListID)},
models.SubscriptionStatusUnconfirmed,
true); err != nil {
lo.Fatalf("error creating subscriber: %v", err)
}
+}
- // Default template.
- tplBody, err := fs.Get("/static/email-templates/default.tpl")
+func installTemplates(q *models.Queries) (int, int) {
+ // Default campaign template.
+ campTpl, err := fs.Get("/static/email-templates/default.tpl")
if err != nil {
lo.Fatalf("error reading default e-mail template: %v", err)
}
- var tplID int
- if err := q.CreateTemplate.Get(&tplID,
- "Default template",
- string(tplBody.ReadBytes()),
- ); err != nil {
- lo.Fatalf("error creating default template: %v", err)
+ var campTplID int
+ if err := q.CreateTemplate.Get(&campTplID, "Default campaign template", models.TemplateTypeCampaign, "", campTpl.ReadBytes()); err != nil {
+ lo.Fatalf("error creating default campaign template: %v", err)
}
- if _, err := q.SetDefaultTemplate.Exec(tplID); err != nil {
+ if _, err := q.SetDefaultTemplate.Exec(campTplID); err != nil {
lo.Fatalf("error setting default template: %v", err)
}
+ // Default campaign archive template.
+ archiveTpl, err := fs.Get("/static/email-templates/default-archive.tpl")
+ if err != nil {
+ lo.Fatalf("error reading default archive template: %v", err)
+ }
+
+ var archiveTplID int
+ if err := q.CreateTemplate.Get(&archiveTplID, "Default archive template", models.TemplateTypeCampaign, "", archiveTpl.ReadBytes()); err != nil {
+ lo.Fatalf("error creating default campaign template: %v", err)
+ }
+
+ // Sample tx template.
+ txTpl, err := fs.Get("/static/email-templates/sample-tx.tpl")
+ if err != nil {
+ lo.Fatalf("error reading default e-mail template: %v", err)
+ }
+
+ if _, err := q.CreateTemplate.Exec("Sample transactional template", models.TemplateTypeTx, "Welcome {{ .Subscriber.Name }}", txTpl.ReadBytes()); err != nil {
+ lo.Fatalf("error creating sample transactional template: %v", err)
+ }
+
+ return campTplID, archiveTplID
+}
+
+func installCampaign(campTplID, archiveTplID int, q *models.Queries) {
// Sample campaign.
if _, err := q.CreateCampaign.Exec(uuid.Must(uuid.NewV4()),
models.CampaignTypeRegular,
@@ -136,35 +207,30 @@ func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempo
"Welcome to listmonk",
"No Reply This is a test e-mail campaign. Your second name is {{ .Subscriber.LastName }} and you are from {{ .Subscriber.Attribs.city }}.
+Here is a tracked link.
+Use the link icon in the editor toolbar or when writing raw HTML or Markdown, + simply suffix @TrackLink to the end of a URL to turn it into a tracking link. Example:
+<a href="https://listmonk.app@TrackLink"></a>+
For help, refer to the documentation.
+ `, nil, "richtext", nil, + json.RawMessage("[]"), pq.StringArray{"test-campaign"}, emailMsgr, - 1, + campTplID, pq.Int64Array{1}, + false, + "welcome-to-listmonk", + archiveTplID, + `{"name": "Subscriber"}`, + nil, ); err != nil { lo.Fatalf("error creating sample campaign: %v", err) } - lo.Printf("setup complete") - lo.Printf(`run the program and access the dashboard at %s`, ko.MustString("app.address")) -} - -// installSchema executes the SQL schema and creates the necessary tables and types. -func installSchema(curVer string, db *sqlx.DB, fs stuffbin.FileSystem) error { - q, err := fs.Read("/schema.sql") - if err != nil { - return err - } - - if _, err := db.Exec(string(q)); err != nil { - return err - } - - // Insert the current migration version. - return recordMigrationVersion(curVer, db) } // recordMigrationVersion inserts the given version (of DB migration) into the @@ -189,14 +255,7 @@ func newConfigFile(path string) error { return fmt.Errorf("error reading sample config (is binary stuffed?): %v", err) } - // Generate a random admin password. - pwd, err := generateRandomString(16) - if err == nil { - b = regexp.MustCompile(`admin_password\s+?=\s+?(.*)`). - ReplaceAll(b, []byte(fmt.Sprintf(`admin_password = "%s"`, pwd))) - } - - return ioutil.WriteFile(path, b, 0644) + return os.WriteFile(path, b, 0644) } // checkSchema checks if the DB schema is installed. @@ -209,3 +268,21 @@ func checkSchema(db *sqlx.DB) (bool, error) { } return true, nil } + +func installUser(username, password string, q *models.Queries) { + consts := initConstants() + + // Super admin role. + perms := []string{} + for p := range consts.Permissions { + perms = append(perms, p) + } + + if _, err := q.CreateRole.Exec("Super Admin", "user", pq.Array(perms)); err != nil { + lo.Fatalf("error creating super admin role: %v", err) + } + + if _, err := q.CreateUser.Exec(username, true, password, username+"@listmonk", username, "user", 1, nil, "enabled"); err != nil { + lo.Fatalf("error creating superadmin user: %v", err) + } +} diff --git a/cmd/lists.go b/cmd/lists.go index d965b24ec..8b84da002 100644 --- a/cmd/lists.go +++ b/cmd/lists.go @@ -1,84 +1,89 @@ package main import ( - "fmt" "net/http" "strconv" + "strings" - "github.com/gofrs/uuid" + "github.com/knadh/listmonk/internal/auth" "github.com/knadh/listmonk/models" - "github.com/lib/pq" - - "github.com/labstack/echo" + "github.com/labstack/echo/v4" ) -type listsWrap struct { - Results []models.List `json:"results"` - - Total int `json:"total"` - PerPage int `json:"per_page"` - Page int `json:"page"` -} - -var ( - listQuerySortFields = []string{"name", "type", "subscriber_count", "created_at", "updated_at"} -) - -// handleGetLists handles retrieval of lists. +// handleGetLists retrieves lists with additional metadata like subscriber counts. func handleGetLists(c echo.Context) error { var ( - app = c.Get("app").(*App) - out listsWrap - - pg = getPagination(c.QueryParams(), 20) - orderBy = c.FormValue("order_by") - order = c.FormValue("order") - listID, _ = strconv.Atoi(c.Param("id")) - single = false + app = c.Get("app").(*App) + user = c.Get(auth.UserKey).(models.User) + pg = app.paginator.NewFromURL(c.Request().URL.Query()) + + query = strings.TrimSpace(c.FormValue("query")) + tags = c.QueryParams()["tag"] + orderBy = c.FormValue("order_by") + typ = c.FormValue("type") + optin = c.FormValue("optin") + order = c.FormValue("order") + minimal, _ = strconv.ParseBool(c.FormValue("minimal")) + + out models.PageResults ) - // Fetch one list. - if listID > 0 { - single = true + var ( + permittedIDs []int + getAll = false + ) + if _, ok := user.PermissionsMap[models.PermListGetAll]; ok { + getAll = true + } else { + permittedIDs = user.GetListIDs } - // Sort params. - if !strSliceContains(orderBy, listQuerySortFields) { - orderBy = "created_at" - } - if order != sortAsc && order != sortDesc { - order = sortAsc - } + // Minimal query simply returns the list of all lists without JOIN subscriber counts. This is fast. + if minimal { + res, err := app.core.GetLists("", getAll, permittedIDs) + if err != nil { + return err + } + if len(res) == 0 { + return c.JSON(http.StatusOK, okResp{[]struct{}{}}) + } - if err := db.Select(&out.Results, fmt.Sprintf(app.queries.QueryLists, orderBy, order), listID, pg.Offset, pg.Limit); err != nil { - app.log.Printf("error fetching lists: %v", err) - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("globals.messages.errorFetching", - "name", "{globals.terms.lists}", "error", pqErrMsg(err))) - } - if single && len(out.Results) == 0 { - return echo.NewHTTPError(http.StatusBadRequest, - app.i18n.Ts("globals.messages.notFound", "name", "{globals.terms.list}")) - } - if len(out.Results) == 0 { - return c.JSON(http.StatusOK, okResp{[]struct{}{}}) - } + // Meta. + out.Results = res + out.Total = len(res) + out.Page = 1 + out.PerPage = out.Total - // Replace null tags. - for i, v := range out.Results { - if v.Tags == nil { - out.Results[i].Tags = make(pq.StringArray, 0) - } + return c.JSON(http.StatusOK, okResp{out}) } - if single { - return c.JSON(http.StatusOK, okResp{out.Results[0]}) + // Full list query. + res, total, err := app.core.QueryLists(query, typ, optin, tags, orderBy, order, getAll, permittedIDs, pg.Offset, pg.Limit) + if err != nil { + return err } - // Meta. - out.Total = out.Results[0].Total + out.Query = query + out.Results = res + out.Total = total out.Page = pg.Page out.PerPage = pg.PerPage + + return c.JSON(http.StatusOK, okResp{out}) +} + +// handleGetList retrieves a single list by id. +func handleGetList(c echo.Context) error { + var ( + app = c.Get("app").(*App) + listID, _ = strconv.Atoi(c.Param("id")) + ) + + out, err := app.core.GetList(listID, "") + if err != nil { + return err + } + return c.JSON(http.StatusOK, okResp{out}) } @@ -86,44 +91,24 @@ func handleGetLists(c echo.Context) error { func handleCreateList(c echo.Context) error { var ( app = c.Get("app").(*App) - o = models.List{} + l = models.List{} ) - if err := c.Bind(&o); err != nil { + if err := c.Bind(&l); err != nil { return err } // Validate. - if !strHasLen(o.Name, 1, stdInputMaxLen) { + if !strHasLen(l.Name, 1, stdInputMaxLen) { return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("lists.invalidName")) } - uu, err := uuid.NewV4() + out, err := app.core.CreateList(l) if err != nil { - app.log.Printf("error generating UUID: %v", err) - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("globals.messages.errorUUID", "error", err.Error())) - } - - // Insert and read ID. - var newID int - o.UUID = uu.String() - if err := app.queries.CreateList.Get(&newID, - o.UUID, - o.Name, - o.Type, - o.Optin, - pq.StringArray(normalizeTags(o.Tags))); err != nil { - app.log.Printf("error creating list: %v", err) - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("globals.messages.errorCreating", - "name", "{globals.terms.list}", "error", pqErrMsg(err))) + return err } - // Hand over to the GET handler to return the last insertion. - return handleGetLists(copyEchoCtx(c, map[string]string{ - "id": fmt.Sprintf("%d", newID), - })) + return c.JSON(http.StatusOK, okResp{out}) } // handleUpdateList handles list modification. @@ -138,26 +123,22 @@ func handleUpdateList(c echo.Context) error { } // Incoming params. - var o models.List - if err := c.Bind(&o); err != nil { + var l models.List + if err := c.Bind(&l); err != nil { return err } - res, err := app.queries.UpdateList.Exec(id, - o.Name, o.Type, o.Optin, pq.StringArray(normalizeTags(o.Tags))) - if err != nil { - app.log.Printf("error updating list: %v", err) - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("globals.messages.errorUpdating", - "name", "{globals.terms.list}", "error", pqErrMsg(err))) + // Validate. + if !strHasLen(l.Name, 1, stdInputMaxLen) { + return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("lists.invalidName")) } - if n, _ := res.RowsAffected(); n == 0 { - return echo.NewHTTPError(http.StatusBadRequest, - app.i18n.Ts("globals.messages.notFound", "name", "{globals.terms.list}")) + out, err := app.core.UpdateList(id, l) + if err != nil { + return err } - return handleGetLists(c) + return c.JSON(http.StatusOK, okResp{out}) } // handleDeleteLists handles list deletion, either a single one (ID in the URI), or a list. @@ -165,7 +146,7 @@ func handleDeleteLists(c echo.Context) error { var ( app = c.Get("app").(*App) id, _ = strconv.ParseInt(c.Param("id"), 10, 64) - ids pq.Int64Array + ids []int ) if id < 1 && len(ids) == 0 { @@ -173,15 +154,46 @@ func handleDeleteLists(c echo.Context) error { } if id > 0 { - ids = append(ids, id) + ids = append(ids, int(id)) } - if _, err := app.queries.DeleteLists.Exec(ids); err != nil { - app.log.Printf("error deleting lists: %v", err) - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("globals.messages.errorDeleting", - "name", "{globals.terms.list}", "error", pqErrMsg(err))) + if err := app.core.DeleteLists(ids); err != nil { + return err } return c.JSON(http.StatusOK, okResp{true}) } + +// listPerm is a middleware for wrapping /list/* API calls that take a +// list :id param for validating the list ID against the user's list perms. +func listPerm(next echo.HandlerFunc) echo.HandlerFunc { + return func(c echo.Context) error { + var ( + app = c.Get("app").(*App) + user = c.Get(auth.UserKey).(models.User) + id, _ = strconv.Atoi(c.Param("id")) + ) + + // Define permissions based on HTTP read/write. + var ( + permAll = models.PermListManageAll + perm = models.PermListManage + ) + if c.Request().Method == http.MethodGet { + permAll = models.PermListGetAll + perm = models.PermListGet + } + + // Check if the user has permissions for all lists or the specific list. + if _, ok := user.PermissionsMap[permAll]; ok { + return next(c) + } + if id > 0 { + if _, ok := user.ListPermissionsMap[id][perm]; ok { + return next(c) + } + } + + return echo.NewHTTPError(http.StatusBadRequest, app.i18n.Ts("globals.messages.permissionDenied", "name", "list")) + } +} diff --git a/cmd/main.go b/cmd/main.go index 8c7274059..e7de40926 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -3,7 +3,6 @@ package main import ( "context" "fmt" - "html/template" "io" "log" "os" @@ -14,15 +13,20 @@ import ( "time" "github.com/jmoiron/sqlx" - "github.com/knadh/koanf" "github.com/knadh/koanf/providers/env" + "github.com/knadh/koanf/v2" + "github.com/knadh/listmonk/internal/auth" "github.com/knadh/listmonk/internal/bounce" "github.com/knadh/listmonk/internal/buflog" + "github.com/knadh/listmonk/internal/captcha" + "github.com/knadh/listmonk/internal/core" + "github.com/knadh/listmonk/internal/events" "github.com/knadh/listmonk/internal/i18n" "github.com/knadh/listmonk/internal/manager" "github.com/knadh/listmonk/internal/media" - "github.com/knadh/listmonk/internal/messenger" "github.com/knadh/listmonk/internal/subimporter" + "github.com/knadh/listmonk/models" + "github.com/knadh/paginator" "github.com/knadh/stuffbin" ) @@ -33,27 +37,37 @@ const ( // App contains the "global" components that are // passed around, especially through HTTP handlers. type App struct { - fs stuffbin.FileSystem - db *sqlx.DB - queries *Queries - constants *constants - manager *manager.Manager - importer *subimporter.Importer - messengers map[string]messenger.Messenger - media media.Store - i18n *i18n.I18n - bounce *bounce.Manager - notifTpls *template.Template - log *log.Logger - bufLog *buflog.BufLog + core *core.Core + fs stuffbin.FileSystem + db *sqlx.DB + queries *models.Queries + constants *constants + manager *manager.Manager + importer *subimporter.Importer + messengers []manager.Messenger + emailMessenger manager.Messenger + auth *auth.Auth + media media.Store + i18n *i18n.I18n + bounce *bounce.Manager + paginator *paginator.Paginator + captcha *captcha.Captcha + events *events.Events + notifTpls *notifTpls + about about + log *log.Logger + bufLog *buflog.BufLog // Channel for passing reload signals. - sigChan chan os.Signal + chReload chan os.Signal // Global variable that stores the state indicating that a restart is required // after a settings update. needsRestart bool + // First time installation with no user records in the DB. Needs user setup. + needsUserSetup bool + // Global state that stores data on an available remote update. update *AppUpdate sync.Mutex @@ -61,14 +75,15 @@ type App struct { var ( // Buffered log writer for storing N lines of log entries for the UI. - bufLog = buflog.New(5000) - lo = log.New(io.MultiWriter(os.Stdout, bufLog), "", - log.Ldate|log.Ltime|log.Lshortfile) + evStream = events.New() + bufLog = buflog.New(5000) + lo = log.New(io.MultiWriter(os.Stdout, bufLog, evStream.ErrWriter()), "", + log.Ldate|log.Ltime|log.Lmicroseconds|log.Lshortfile) ko = koanf.New(".") fs stuffbin.FileSystem db *sqlx.DB - queries *Queries + queries *models.Queries // Compile-time variables. buildString string @@ -78,7 +93,7 @@ var ( // are not embedded (in make dist), these paths are looked up. The default values before, when not // overridden by build flags, are relative to the CWD at runtime. appDir string = "." - frontendDir string = "frontend" + frontendDir string = "frontend/dist" ) func init() { @@ -141,11 +156,16 @@ func init() { // Before the queries are prepared, see if there are pending upgrades. checkUpgrade(db) - // Load the SQL queries from the filesystem. - _, queries := initQueries(queryFilePath, db, fs, true) + // Read the SQL queries from the queries file. + qMap := readQueries(queryFilePath, db, fs) // Load settings from DB. - initSettings(queries.GetSettings) + if q, ok := qMap["get-settings"]; ok { + initSettings(q.Query, db, ko) + } + + // Prepare queries. + queries = prepareQueries(qMap, db, ko) } func main() { @@ -156,40 +176,88 @@ func main() { db: db, constants: initConstants(), media: initMediaStore(), - messengers: make(map[string]messenger.Messenger), + messengers: []manager.Messenger{}, log: lo, bufLog: bufLog, + captcha: initCaptcha(), + events: evStream, + + paginator: paginator.New(paginator.Opt{ + DefaultPerPage: 20, + MaxPerPage: 50, + NumPageNums: 10, + PageParam: "page", + PerPageParam: "per_page", + AllowAll: true, + }), } // Load i18n language map. app.i18n = initI18n(app.constants.Lang, fs) + cOpt := &core.Opt{ + Constants: core.Constants{ + SendOptinConfirmation: app.constants.SendOptinConfirmation, + CacheSlowQueries: ko.Bool("app.cache_slow_queries"), + }, + Queries: queries, + DB: db, + I18n: app.i18n, + Log: lo, + } - _, app.queries = initQueries(queryFilePath, db, fs, true) + if err := ko.Unmarshal("bounce.actions", &cOpt.Constants.BounceActions); err != nil { + lo.Fatalf("error unmarshalling bounce config: %v", err) + } + + app.core = core.New(cOpt, &core.Hooks{ + SendOptinConfirmation: sendOptinConfirmationHook(app), + }) + + app.queries = queries app.manager = initCampaignManager(app.queries, app.constants, app) - app.importer = initImporter(app.queries, db, app) + app.importer = initImporter(app.queries, db, app.core, app) + + hasUsers, auth := initAuth(db.DB, ko, app.core) + app.auth = auth + // If there are are no users in the DB who can login, the app has to prompt + // for new user setup. + app.needsUserSetup = !hasUsers + app.notifTpls = initNotifTemplates("/email-templates/*.html", fs, app.i18n, app.constants) + initTxTemplates(app.manager, app) if ko.Bool("bounce.enabled") { app.bounce = initBounceManager(app) go app.bounce.Run() } - // Initialize the default SMTP (`email`) messenger. - app.messengers[emailMsgr] = initSMTPMessenger(app.manager) + // Initialize the SMTP messengers. + app.messengers = initSMTPMessengers() + for _, m := range app.messengers { + if m.Name() == emailMsgr { + app.emailMessenger = m + } + } // Initialize any additional postback messengers. - for _, m := range initPostbackMessengers(app.manager) { - app.messengers[m.Name()] = m - } + app.messengers = append(app.messengers, initPostbackMessengers()...) // Attach all messengers to the campaign manager. for _, m := range app.messengers { app.manager.AddMessenger(m) } + // Load system information. + app.about = initAbout(queries, db) + + // Start cronjobs. + if cOpt.Constants.CacheSlowQueries { + initCron(app.core) + } + // Start the campaign workers. The campaign batches (fetch from DB, push out // messages) get processed at the specified interval. - go app.manager.Run(time.Second * 5) + go app.manager.Run() // Start the app server. srv := initHTTPServer(app) @@ -202,11 +270,11 @@ func main() { // Wait for the reload signal with a callback to gracefully shut down resources. // The `wait` channel is passed to awaitReload to wait for the callback to finish // within N seconds, or do a force reload. - app.sigChan = make(chan os.Signal) - signal.Notify(app.sigChan, syscall.SIGHUP) + app.chReload = make(chan os.Signal) + signal.Notify(app.chReload, syscall.SIGHUP) closerWait := make(chan bool) - <-awaitReload(app.sigChan, closerWait, func() { + <-awaitReload(app.chReload, closerWait, func() { // Stop the HTTP server. ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second) defer cancel() diff --git a/cmd/maintenance.go b/cmd/maintenance.go new file mode 100644 index 000000000..c11e3a854 --- /dev/null +++ b/cmd/maintenance.go @@ -0,0 +1,92 @@ +package main + +import ( + "net/http" + "time" + + "github.com/labstack/echo/v4" +) + +// handleGCSubscribers garbage collects (deletes) orphaned or blocklisted subscribers. +func handleGCSubscribers(c echo.Context) error { + var ( + app = c.Get("app").(*App) + typ = c.Param("type") + ) + + var ( + n int + err error + ) + + switch typ { + case "blocklisted": + n, err = app.core.DeleteBlocklistedSubscribers() + case "orphan": + n, err = app.core.DeleteOrphanSubscribers() + default: + err = echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidData")) + } + + if err != nil { + return err + } + + return c.JSON(http.StatusOK, okResp{struct { + Count int `json:"count"` + }{n}}) +} + +// handleGCSubscriptions garbage collects (deletes) orphaned or blocklisted subscribers. +func handleGCSubscriptions(c echo.Context) error { + var ( + app = c.Get("app").(*App) + ) + + t, err := time.Parse(time.RFC3339, c.FormValue("before_date")) + if err != nil { + return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidData")) + } + + n, err := app.core.DeleteUnconfirmedSubscriptions(t) + if err != nil { + return err + } + + return c.JSON(http.StatusOK, okResp{struct { + Count int `json:"count"` + }{n}}) +} + +// handleGCCampaignAnalytics garbage collects (deletes) campaign analytics. +func handleGCCampaignAnalytics(c echo.Context) error { + var ( + app = c.Get("app").(*App) + typ = c.Param("type") + ) + + t, err := time.Parse(time.RFC3339, c.FormValue("before_date")) + if err != nil { + return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidData")) + } + + switch typ { + case "all": + if err := app.core.DeleteCampaignViews(t); err != nil { + return err + } + err = app.core.DeleteCampaignLinkClicks(t) + case "views": + err = app.core.DeleteCampaignViews(t) + case "clicks": + err = app.core.DeleteCampaignLinkClicks(t) + default: + err = echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidData")) + } + + if err != nil { + return err + } + + return c.JSON(http.StatusOK, okResp{true}) +} diff --git a/cmd/manager_store.go b/cmd/manager_store.go index 5ff5e55d5..b1d67ef8c 100644 --- a/cmd/manager_store.go +++ b/cmd/manager_store.go @@ -1,27 +1,47 @@ package main import ( - "github.com/gofrs/uuid" + "net/http" + + "github.com/gofrs/uuid/v5" + "github.com/knadh/listmonk/internal/core" + "github.com/knadh/listmonk/internal/manager" + "github.com/knadh/listmonk/internal/media" "github.com/knadh/listmonk/models" "github.com/lib/pq" ) -// runnerDB implements runner.DataSource over the primary +// store implements DataSource over the primary // database. -type runnerDB struct { - queries *Queries +type store struct { + queries *models.Queries + core *core.Core + media media.Store + h *http.Client +} + +type runningCamp struct { + CampaignID int `db:"campaign_id"` + CampaignType string `db:"campaign_type"` + LastSubscriberID int `db:"last_subscriber_id"` + MaxSubscriberID int `db:"max_subscriber_id"` + ListID int `db:"list_id"` } -func newManagerStore(q *Queries) *runnerDB { - return &runnerDB{ +func newManagerStore(q *models.Queries, c *core.Core, m media.Store) *store { + return &store{ queries: q, + core: c, + media: m, } } -// NextCampaigns retrieves active campaigns ready to be processed. -func (r *runnerDB) NextCampaigns(excludeIDs []int64) ([]*models.Campaign, error) { +// NextCampaigns retrieves active campaigns ready to be processed excluding +// campaigns that are also being processed. Additionally, it takes a map of campaignID:sentCount +// of campaigns that are being processed and updates them in the DB. +func (s *store) NextCampaigns(currentIDs []int64, sentCounts []int64) ([]*models.Campaign, error) { var out []*models.Campaign - err := r.queries.NextCampaigns.Select(&out, pq.Int64Array(excludeIDs)) + err := s.queries.NextCampaigns.Select(&out, pq.Int64Array(currentIDs), pq.Int64Array(sentCounts)) return out, err } @@ -29,27 +49,66 @@ func (r *runnerDB) NextCampaigns(excludeIDs []int64) ([]*models.Campaign, error) // Since batches are processed sequentially, the retrieval is ordered by ID, // and every batch takes the last ID of the last batch and fetches the next // batch above that. -func (r *runnerDB) NextSubscribers(campID, limit int) ([]models.Subscriber, error) { +func (s *store) NextSubscribers(campID, limit int) ([]models.Subscriber, error) { + var camps []runningCamp + if err := s.queries.GetRunningCampaign.Select(&camps, campID); err != nil { + return nil, err + } + + var listIDs []int + for _, c := range camps { + listIDs = append(listIDs, c.ListID) + } + + if len(listIDs) == 0 { + return nil, nil + } + var out []models.Subscriber - err := r.queries.NextCampaignSubscribers.Select(&out, campID, limit) + err := s.queries.NextCampaignSubscribers.Select(&out, camps[0].CampaignID, camps[0].CampaignType, camps[0].LastSubscriberID, camps[0].MaxSubscriberID, pq.Array(listIDs), limit) return out, err } // GetCampaign fetches a campaign from the database. -func (r *runnerDB) GetCampaign(campID int) (*models.Campaign, error) { +func (s *store) GetCampaign(campID int) (*models.Campaign, error) { var out = &models.Campaign{} - err := r.queries.GetCampaign.Get(out, campID, nil) + err := s.queries.GetCampaign.Get(out, campID, nil, nil, "default") return out, err } // UpdateCampaignStatus updates a campaign's status. -func (r *runnerDB) UpdateCampaignStatus(campID int, status string) error { - _, err := r.queries.UpdateCampaignStatus.Exec(campID, status) +func (s *store) UpdateCampaignStatus(campID int, status string) error { + _, err := s.queries.UpdateCampaignStatus.Exec(campID, status) + return err +} + +// UpdateCampaignCounts updates a campaign's status. +func (s *store) UpdateCampaignCounts(campID int, toSend int, sent int, lastSubID int) error { + _, err := s.queries.UpdateCampaignCounts.Exec(campID, toSend, sent, lastSubID) return err } +// GetAttachment fetches a media attachment blob. +func (s *store) GetAttachment(mediaID int) (models.Attachment, error) { + m, err := s.core.GetMedia(mediaID, "", s.media) + if err != nil { + return models.Attachment{}, err + } + + b, err := s.media.GetBlob(m.URL) + if err != nil { + return models.Attachment{}, err + } + + return models.Attachment{ + Name: m.Filename, + Content: b, + Header: manager.MakeAttachmentHeader(m.Filename, "base64", m.ContentType), + }, nil +} + // CreateLink registers a URL with a UUID for tracking clicks and returns the UUID. -func (r *runnerDB) CreateLink(url string) (string, error) { +func (s *store) CreateLink(url string) (string, error) { // Create a new UUID for the URL. If the URL already exists in the DB // the UUID in the database is returned. uu, err := uuid.NewV4() @@ -58,7 +117,7 @@ func (r *runnerDB) CreateLink(url string) (string, error) { } var out string - if err := r.queries.CreateLink.Get(&out, uu, url); err != nil { + if err := s.queries.CreateLink.Get(&out, uu, url); err != nil { return "", err } @@ -66,13 +125,13 @@ func (r *runnerDB) CreateLink(url string) (string, error) { } // RecordBounce records a bounce event and returns the bounce count. -func (r *runnerDB) RecordBounce(b models.Bounce) (int64, int, error) { +func (s *store) RecordBounce(b models.Bounce) (int64, int, error) { var res = struct { SubscriberID int64 `db:"subscriber_id"` Num int `db:"num"` }{} - err := r.queries.UpdateCampaignStatus.Select(&res, + err := s.queries.UpdateCampaignStatus.Select(&res, b.SubscriberUUID, b.Email, b.CampaignUUID, @@ -83,12 +142,12 @@ func (r *runnerDB) RecordBounce(b models.Bounce) (int64, int, error) { return res.SubscriberID, res.Num, err } -func (r *runnerDB) BlocklistSubscriber(id int64) error { - _, err := r.queries.BlocklistSubscribers.Exec(pq.Int64Array{id}) +func (s *store) BlocklistSubscriber(id int64) error { + _, err := s.queries.BlocklistSubscribers.Exec(pq.Int64Array{id}) return err } -func (r *runnerDB) DeleteSubscriber(id int64) error { - _, err := r.queries.DeleteSubscribers.Exec(pq.Int64Array{id}) +func (s *store) DeleteSubscriber(id int64) error { + _, err := s.queries.DeleteSubscribers.Exec(pq.Int64Array{id}) return err } diff --git a/cmd/media.go b/cmd/media.go index b1c61bc05..1885d7aa1 100644 --- a/cmd/media.go +++ b/cmd/media.go @@ -6,22 +6,21 @@ import ( "net/http" "path/filepath" "strconv" + "strings" "github.com/disintegration/imaging" - "github.com/gofrs/uuid" - "github.com/knadh/listmonk/internal/media" - "github.com/labstack/echo" + "github.com/knadh/listmonk/models" + "github.com/labstack/echo/v4" ) const ( thumbPrefix = "thumb_" - thumbnailSize = 90 + thumbnailSize = 250 ) -// validMimes is the list of image types allowed to be uploaded. var ( - validMimes = []string{"image/jpg", "image/jpeg", "image/png", "image/gif"} - validExts = []string{".jpg", ".jpeg", ".png", ".gif"} + vectorExts = []string{"svg"} + imageExts = []string{"gif", "png", "jpg", "jpeg"} ) // handleUploadMedia handles media file uploads. @@ -36,23 +35,6 @@ func handleUploadMedia(c echo.Context) error { app.i18n.Ts("media.invalidFile", "error", err.Error())) } - // Validate file extension. - ext := filepath.Ext(file.Filename) - if ok := inArray(ext, validExts); !ok { - return echo.NewHTTPError(http.StatusBadRequest, - app.i18n.Ts("media.unsupportedFileType", "type", ext)) - } - - // Validate file's mime. - typ := file.Header.Get("Content-type") - if ok := inArray(typ, validMimes); !ok { - return echo.NewHTTPError(http.StatusBadRequest, - app.i18n.Ts("media.unsupportedFileType", "type", typ)) - } - - // Generate filename - fName := makeFilename(file.Filename) - // Read file contents in memory src, err := file.Open() if err != nil { @@ -61,76 +43,127 @@ func handleUploadMedia(c echo.Context) error { } defer src.Close() + var ( + // Naive check for content type and extension. + ext = strings.TrimPrefix(strings.ToLower(filepath.Ext(file.Filename)), ".") + contentType = file.Header.Get("Content-Type") + ) + if !isASCII(file.Filename) { + return echo.NewHTTPError(http.StatusUnprocessableEntity, + app.i18n.Ts("media.invalidFileName", "name", file.Filename)) + } + + // Validate file extension. + if !inArray("*", app.constants.MediaUpload.Extensions) { + if ok := inArray(ext, app.constants.MediaUpload.Extensions); !ok { + return echo.NewHTTPError(http.StatusBadRequest, + app.i18n.Ts("media.unsupportedFileType", "type", ext)) + } + } + + // Sanitize filename. + fName := makeFilename(file.Filename) + + // Add a random suffix to the filename to ensure uniqueness. + suffix, _ := generateRandomString(6) + fName = appendSuffixToFilename(fName, suffix) + // Upload the file. - fName, err = app.media.Put(fName, typ, src) + fName, err = app.media.Put(fName, contentType, src) if err != nil { app.log.Printf("error uploading file: %v", err) - cleanUp = true return echo.NewHTTPError(http.StatusInternalServerError, app.i18n.Ts("media.errorUploading", "error", err.Error())) } + var ( + thumbfName = "" + width = 0 + height = 0 + ) defer func() { // If any of the subroutines in this function fail, // the uploaded image should be removed. if cleanUp { app.media.Delete(fName) - app.media.Delete(thumbPrefix + fName) + + if thumbfName != "" { + app.media.Delete(thumbfName) + } } }() - // Create thumbnail from file. - thumbFile, err := createThumbnail(file) - if err != nil { - cleanUp = true - app.log.Printf("error resizing image: %v", err) - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("media.errorResizing", "error", err.Error())) + // Create thumbnail from file for non-vector formats. + isImage := inArray(ext, imageExts) + if isImage { + thumbFile, w, h, err := processImage(file) + if err != nil { + cleanUp = true + app.log.Printf("error resizing image: %v", err) + return echo.NewHTTPError(http.StatusInternalServerError, + app.i18n.Ts("media.errorResizing", "error", err.Error())) + } + width = w + height = h + + // Upload thumbnail. + tf, err := app.media.Put(thumbPrefix+fName, contentType, thumbFile) + if err != nil { + cleanUp = true + app.log.Printf("error saving thumbnail: %v", err) + return echo.NewHTTPError(http.StatusInternalServerError, + app.i18n.Ts("media.errorSavingThumbnail", "error", err.Error())) + } + thumbfName = tf } - - // Upload thumbnail. - thumbfName, err := app.media.Put(thumbPrefix+fName, typ, thumbFile) - if err != nil { - cleanUp = true - app.log.Printf("error saving thumbnail: %v", err) - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("media.errorSavingThumbnail", "error", err.Error())) - } - - uu, err := uuid.NewV4() - if err != nil { - app.log.Printf("error generating UUID: %v", err) - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("globals.messages.errorUUID", "error", err.Error())) + if inArray(ext, vectorExts) { + thumbfName = fName } // Write to the DB. - if _, err := app.queries.InsertMedia.Exec(uu, fName, thumbfName, app.constants.MediaProvider); err != nil { + meta := models.JSON{} + if isImage { + meta = models.JSON{ + "width": width, + "height": height, + } + } + m, err := app.core.InsertMedia(fName, thumbfName, contentType, meta, app.constants.MediaUpload.Provider, app.media) + if err != nil { cleanUp = true - app.log.Printf("error inserting uploaded file to db: %v", err) - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("globals.messages.errorCreating", - "name", "{globals.terms.media}", "error", pqErrMsg(err))) + return err } - return c.JSON(http.StatusOK, okResp{true}) + return c.JSON(http.StatusOK, okResp{m}) } // handleGetMedia handles retrieval of uploaded media. func handleGetMedia(c echo.Context) error { var ( - app = c.Get("app").(*App) - out = []media.Media{} + app = c.Get("app").(*App) + pg = app.paginator.NewFromURL(c.Request().URL.Query()) + query = c.FormValue("query") + id, _ = strconv.Atoi(c.Param("id")) ) - if err := app.queries.GetMedia.Select(&out, app.constants.MediaProvider); err != nil { - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("globals.messages.errorFetching", - "name", "{globals.terms.media}", "error", pqErrMsg(err))) + // Fetch one list. + if id > 0 { + out, err := app.core.GetMedia(id, "", app.media) + if err != nil { + return err + } + return c.JSON(http.StatusOK, okResp{out}) } - for i := 0; i < len(out); i++ { - out[i].URL = app.media.Get(out[i].Filename) - out[i].ThumbURL = app.media.Get(out[i].Thumb) + res, total, err := app.core.QueryMedia(app.constants.MediaUpload.Provider, app.media, query, pg.Offset, pg.Limit) + if err != nil { + return err + } + + out := models.PageResults{ + Results: res, + Total: total, + Page: pg.Page, + PerPage: pg.PerPage, } return c.JSON(http.StatusOK, okResp{out}) @@ -147,29 +180,29 @@ func handleDeleteMedia(c echo.Context) error { return echo.NewHTTPError(http.StatusBadRequest, app.i18n.T("globals.messages.invalidID")) } - var m media.Media - if err := app.queries.DeleteMedia.Get(&m, id); err != nil { - return echo.NewHTTPError(http.StatusInternalServerError, - app.i18n.Ts("globals.messages.errorDeleting", - "name", "{globals.terms.media}", "error", pqErrMsg(err))) + fname, err := app.core.DeleteMedia(id) + if err != nil { + return err } - app.media.Delete(m.Filename) - app.media.Delete(thumbPrefix + m.Filename) + app.media.Delete(fname) + app.media.Delete(thumbPrefix + fname) + return c.JSON(http.StatusOK, okResp{true}) } -// createThumbnail reads the file object and returns a smaller image -func createThumbnail(file *multipart.FileHeader) (*bytes.Reader, error) { +// processImage reads the image file and returns thumbnail bytes and +// the original image's width, and height. +func processImage(file *multipart.FileHeader) (*bytes.Reader, int, int, error) { src, err := file.Open() if err != nil { - return nil, err + return nil, 0, 0, err } defer src.Close() img, err := imaging.Decode(src) if err != nil { - return nil, err + return nil, 0, 0, err } // Encode the image into a byte slice as PNG. @@ -178,7 +211,9 @@ func createThumbnail(file *multipart.FileHeader) (*bytes.Reader, error) { out bytes.Buffer ) if err := imaging.Encode(&out, thumb, imaging.PNG); err != nil { - return nil, err + return nil, 0, 0, err } - return bytes.NewReader(out.Bytes()), nil + + b := img.Bounds().Max + return bytes.NewReader(out.Bytes()), b.X, b.Y, nil } diff --git a/cmd/notifications.go b/cmd/notifications.go index 3fd0406c1..eece80ed2 100644 --- a/cmd/notifications.go +++ b/cmd/notifications.go @@ -2,8 +2,11 @@ package main import ( "bytes" + "net/textproto" + "regexp" + "strings" - "github.com/knadh/listmonk/internal/manager" + "github.com/knadh/listmonk/models" ) const ( @@ -13,6 +16,10 @@ const ( notifSubscriberData = "subscriber-data" ) +var ( + reTitle = regexp.MustCompile(`(?s)Hi there
+Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis et elit ac elit sollicitudin condimentum non a magna. + Sed tempor mauris in facilisis vehicula. Aenean nisl urna, accumsan ac tincidunt vitae, interdum cursus massa. + Interdum et malesuada fames ac ante ipsum primis in faucibus. Aliquam varius turpis et turpis lacinia placerat. + Aenean id ligula a orci lacinia blandit at eu felis. Phasellus vel lobortis lacus. Suspendisse leo elit, luctus sed + erat ut, venenatis fermentum ipsum. Donec bibendum neque quis.
+ +Nam luctus dui non placerat mattis. Morbi non accumsan orci, vel interdum urna. Duis faucibus id nunc ut euismod. + Curabitur et eros id erat feugiat fringilla in eget neque. Aliquam accumsan cursus eros sed faucibus.
+ +Here is a link to listmonk.
+``` + +______________________________________________________________________ + +#### POST /api/templates + +Create a template. + +##### Parameters + +| Name | Type | Required | Description | +|:--------|:----------|:---------|:----------------------------------------------| +| name | string | Yes | Name of the template | +| type | string | Yes | Type of the template (`campaign` or `tx`) | +| subject | string | | Subject line for the template (only for `tx`) | +| body | string | Yes | HTML body of the template | + +##### Example Request + +```shell +curl -u "api_user:token" -X POST 'http://localhost:9000/api/templates' \ +-H 'Content-Type: application/json' \ +-d '{ + "name": "New template", + "type": "campaign", + "subject": "Your Weekly Newsletter", + "body": "Content goes here
" +}' +``` + +##### Example Response + +```json +{ + "data": [ + { + "id": 1, + "created_at": "2020-03-14T17:36:41.288578+01:00", + "updated_at": "2020-03-14T17:36:41.288578+01:00", + "name": "Default template", + "body": "{{ template \"content\" . }}", + "type": "campaign", + "is_default": true + } + ] +} +``` + +______________________________________________________________________ + +#### PUT /api/templates/{template_id} + +Update a template. + +> Refer to parameters from [POST /api/templates](#post-apitemplates) + +______________________________________________________________________ + +#### PUT /api/templates/{template_id}/default + +Set a template as the default. + +##### Parameters + +| Name | Type | Required | Description | +|:------------|:----------|:---------|:-------------------------------------| +| template_id | number | Yes | ID of the template to set as default | + +##### Example Request + +```shell +curl -u "api_user:token" -X PUT 'http://localhost:9000/api/templates/1/default' +``` + +##### Example Response + +```json +{ + "data": { + "id": 1, + "created_at": "2020-03-14T17:36:41.288578+01:00", + "updated_at": "2020-03-14T17:36:41.288578+01:00", + "name": "Default template", + "body": "{{ template \"content\" . }}", + "type": "campaign", + "is_default": true + } +} +``` + +______________________________________________________________________ + +#### DELETE /api/templates/{template_id} + +Delete a template. + +##### Parameters + +| Name | Type | Required | Description | +|:------------|:----------|:---------|:-----------------------------| +| template_id | number | Yes | ID of the template to delete | + +##### Example Request + +```shell +curl -u "api_user:token" -X DELETE 'http://localhost:9000/api/templates/35' +``` + +##### Example Response + +```json +{ + "data": true +} +``` diff --git a/docs/docs/content/apis/transactional.md b/docs/docs/content/apis/transactional.md new file mode 100644 index 000000000..80207ec73 --- /dev/null +++ b/docs/docs/content/apis/transactional.md @@ -0,0 +1,65 @@ +# API / Transactional + +| Method | Endpoint | Description | +|:-------|:---------|:-------------------------------| +| POST | /api/tx | Send transactional messages | + +______________________________________________________________________ + +#### POST /api/tx + +Allows sending transactional messages to one or more subscribers via a preconfigured transactional template. + +##### Parameters + +| Name | Type | Required | Description | +|:------------------|:----------|:---------|:---------------------------------------------------------------------------| +| subscriber_email | string | | Email of the subscriber. Can substitute with `subscriber_id`. | +| subscriber_id | number | | Subscriber's ID can substitute with `subscriber_email`. | +| subscriber_emails | string\[\] | | Multiple subscriber emails as alternative to `subscriber_email`. | +| subscriber_ids | number\[\] | | Multiple subscriber IDs as an alternative to `subscriber_id`. | +| template_id | number | Yes | ID of the transactional template to be used for the message. | +| from_email | string | | Optional sender email. | +| data | JSON | | Optional nested JSON map. Available in the template as `{{ .Tx.Data.* }}`. | +| headers | JSON\[\] | | Optional array of email headers. | +| messenger | string | | Messenger to send the message. Default is `email`. | +| content_type | string | | Email format options include `html`, `markdown`, and `plain`. | + +##### Example + +```shell +curl -u "api_user:token" "http://localhost:9000/api/tx" -X POST \ + -H 'Content-Type: application/json; charset=utf-8' \ + --data-binary @- << EOF + { + "subscriber_email": "user@test.com", + "template_id": 2, + "data": {"order_id": "1234", "date": "2022-07-30", "items": [1, 2, 3]}, + "content_type": "html" + } +EOF +``` + +##### Example response + +```json +{ + "data": true +} +``` + +______________________________________________________________________ + +#### File Attachments + +To include file attachments in a transactional message, use the `multipart/form-data` Content-Type. Use `data` param for the parameters described above as a JSON object. Include any number of attachments via the `file` param. + +```shell +curl -u "api_user:token" "http://localhost:9000/api/tx" -X POST \ +-F 'data=\"{ + \"subscriber_email\": \"user@test.com\", + \"template_id\": 4 +}"' \ +-F 'file=@"/path/to/attachment.pdf"' \ +-F 'file=@"/path/to/attachment2.pdf"' +``` diff --git a/docs/docs/content/archives.md b/docs/docs/content/archives.md new file mode 100644 index 000000000..2c2b639b2 --- /dev/null +++ b/docs/docs/content/archives.md @@ -0,0 +1,32 @@ +# Archives + +A global public archive is maintained on the public web interface. It can be +enabled under Settings -> Settings -> General -> Enable public mailing list +archive. + +To make a campaign available in the public archive (provided it has been +enabled in the settings as described above), enable the option +'Publish to public archive' under Campaigns -> Create new -> Archive. + +When using template variables that depend on subscriber data (such as any +template variable referencing `.Subscriber`), such data must be supplied +as 'Campaign metadata', which is a JSON object that will be used in place +of `.Subscriber` when rendering the archive template and content. + +When individual subscriber tracking is enabled, TrackLink requires that a UUID +of an existing user is provided as part of the campaign metadata. Any clicks on +a TrackLink from the archived campaign will be counted towards that subscriber. + +As an example: + +```json +{ + "UUID": "5a837423-a186-5623-9a87-82691cbe3631", + "email": "example@example.com", + "name": "Reader", + "attribs": {} +} +``` + + + diff --git a/docs/docs/content/bounces.md b/docs/docs/content/bounces.md new file mode 100644 index 000000000..84d76418e --- /dev/null +++ b/docs/docs/content/bounces.md @@ -0,0 +1,106 @@ +# Bounce processing + +Enable bounce processing in Settings -> Bounces. POP3 bounce scanning and APIs only become available once the setting is enabled. + +## POP3 bounce mailbox +Configure the bounce mailbox in Settings -> Bounces. Either the "From" e-mail that is set on a campaign (or in settings) should have a POP3 mailbox behind it to receive bounce e-mails, or you should configure a dedicated POP3 mailbox and add that address as the `Return-Path` (envelope sender) header in Settings -> SMTP -> Custom headers box. For example: + +``` +[ + {"Return-Path": "your-bounce-inbox@site.com"} +] + +``` + +Some mail servers may also return the bounce to the `Reply-To` address, which can also be added to the header settings. + +## Webhook API +The bounce webhook API can be used to record bounce events with custom scripting. This could be by reading a mailbox, a database, or mail server logs. + +| Method | Endpoint | Description | +| ------ | ---------------- | ---------------------- | +| `POST` | /webhooks/bounce | Record a bounce event. | + + +| Name | Type | Required | Description | +| ----------------| --------- | -----------| ------------------------------------------------------------------------------------ | +| subscriber_uuid | string | | The UUID of the subscriber. Either this or `email` is required. | +| email | string | | The e-mail of the subscriber. Either this or `subscriber_uuid` is required. | +| campaign_uuid | string | | UUID of the campaign for which the bounce happened. | +| source | string | Yes | A string indicating the source, eg: `api`, `my_script` etc. | +| type | string | Yes | `hard` or `soft` bounce. Currently, this has no effect on how the bounce is treated. | +| meta | string | | An optional escaped JSON string with arbitrary metadata about the bounce event. | + + +```shell +curl -u 'api_username:access_token' -X POST 'http://localhost:9000/webhooks/bounce' \ + -H "Content-Type: application/json" \ + --data '{"email": "user1@mail.com", "campaign_uuid": "9f86b50d-5711-41c8-ab03-bc91c43d711b", "source": "api", "type": "hard", "meta": "{\"additional\": \"info\"}}' + +``` + +## External webhooks +listmonk supports receiving bounce webhook events from the following SMTP providers. + +| Endpoint | Description | More info | +|:--------------------------------------------------------------|:---------------------------------------|:----------------------------------------------------------------------------------------------------------------------| +| `https://listmonk.yoursite.com/webhooks/service/ses` | Amazon (AWS) SES | See below | +| `https://listmonk.yoursite.com/webhooks/service/sendgrid` | Sendgrid / Twilio Signed event webhook | [More info](https://docs.sendgrid.com/for-developers/tracking-events/getting-started-event-webhook-security-features) | +| `https://listmonk.yoursite.com/webhooks/service/postmark` | Postmark webhook | [More info](https://postmarkapp.com/developer/webhooks/webhooks-overview) | +| `https://listmonk.yoursite.com/webhooks/service/forwardemail` | Forward Email webhook | [More info](https://forwardemail.net/en/faq#do-you-support-bounce-webhooks) | + +## Amazon Simple Email Service (SES) + +If using SES as your SMTP provider, automatic bounce processing is the recommended way to maintain your [sender reputation](https://docs.aws.amazon.com/ses/latest/dg/monitor-sender-reputation.html). The settings below are based on Amazon's [recommendations](https://docs.aws.amazon.com/ses/latest/dg/send-email-concepts-deliverability.html). Please note that your sending domain must be verified in SES before proceeding. + +1. In listmonk settings, go to the "Bounces" tab and configure the following: + - Enable bounce processing: `Enabled` + - Soft: + - Bounce count: `2` + - Action: `None` + - Hard: + - Bounce count: `1` + - Action: `Blocklist` + - Complaint: + - Bounce count: `1` + - Action: `Blocklist` + - Enable bounce webhooks: `Enabled` + - Enable SES: `Enabled` +2. In the AWS console, go to [Simple Notification Service](https://console.aws.amazon.com/sns/) and create a new topic with the following settings: + - Type: `Standard` + - Name: `ses-bounces` (or any other name) +3. Create a new subscription to that topic with the following settings: + - Protocol: `HTTPS` + - Endpoint: `https://listmonk.yoursite.com/webhooks/service/ses` + - Enable raw message delivery: `Disabled` (unchecked) +4. SES will then make a request to your listmonk instance to confirm the subscription. After a page refresh, the subscription should have a status of "Confirmed". If not, your endpoint may be incorrect or not publicly accessible. +5. In the AWS console, go to [Simple Email Service](https://console.aws.amazon.com/ses/) and click "Verified identities" in the left sidebar. +6. Click your domain and go to the "Notifications" tab. +7. Next to "Feedback notifications", click "Edit". +8. For both "Bounce feedback" and "Complaint feedback", use the following settings: + - SNS topic: `ses-bounces` (or whatever you named it) + - Include original email headers: `Enabled` (checked) +9. Repeat steps 6-8 for any `Email address` identities you send from using listmonk +10. Bounce processing should now be working. You can test it with [SES simulator addresses](https://docs.aws.amazon.com/ses/latest/dg/send-an-email-from-console.html#send-email-simulator). Add them as subscribers, send them campaign previews, and ensure that the appropriate action was taken after the configured bounce count was reached. + - Soft bounce: `ooto@simulator.amazonses.com` + - Hard bounce: `bounce@simulator.amazonses.com` + - Complaint: `complaint@simulator.amazonses.com` +11. You can optionally [disable email feedback forwarding](https://docs.aws.amazon.com/ses/latest/dg/monitor-sending-activity-using-notifications-email.html#monitor-sending-activity-using-notifications-email-disabling). + +## Exporting bounces + +Bounces can be exported via the JSON API: +```shell +curl -u 'username:passsword' 'http://localhost:9000/api/bounces' +``` + +Or by querying the database directly: +```sql +SELECT bounces.created_at, + bounces.subscriber_id, + subscribers.uuid AS subscriber_uuid, + subscribers.email AS email +FROM bounces +LEFT JOIN subscribers ON (subscribers.id = bounces.subscriber_id) +ORDER BY bounces.created_at DESC LIMIT 1000; +``` diff --git a/docs/docs/content/concepts.md b/docs/docs/content/concepts.md new file mode 100644 index 000000000..2064962cb --- /dev/null +++ b/docs/docs/content/concepts.md @@ -0,0 +1,72 @@ +# Concepts + +## Subscriber + +A subscriber is a recipient identified by an e-mail address and name. Subscribers receive e-mails that are sent from listmonk. A subscriber can be added to any number of lists. Subscribers who are not a part of any lists are considered *orphan* records. + +### Attributes + +Attributes are arbitrary properties attached to a subscriber in addition to their e-mail and name. They are represented as a JSON map. It is not necessary for all subscribers to have the same attributes. Subscribers can be [queried and segmented](querying-and-segmentation.md) into lists based on their attributes, and the attributes can be inserted into the e-mails sent to them. For example: + +```json +{ + "city": "Bengaluru", + "likes_tea": true, + "spoken_languages": ["English", "Malayalam"], + "projects": 3, + "stack": { + "frameworks": ["echo", "go"], + "languages": ["go", "python"], + "preferred_language": "go" + } +} +``` + +### Subscription statuses + +A subscriber can be added to one or more lists, and each such relationship can have one of these statuses. + +| Status | Description | +| ------------- | --------------------------------------------------------------------------------- | +| `unconfirmed` | The subscriber was added to the list directly without their explicit confirmation. Nonetheless, the subscriber will receive campaign messages sent to single optin campaigns. | +| `confirmed` | The subscriber confirmed their subscription by clicking on 'accept' in the confirmation e-mail. Only confirmed subscribers in opt-in lists will receive campaign messages send to the list. | +| `unsubscribed` | The subscriber is unsubscribed from the list and will not receive any campaign messages sent to the list. + + +### Segmentation + +Segmentation is the process of filtering a large list of subscribers into a smaller group based on arbitrary conditions, primarily based on their attributes. For instance, if an e-mail needs to be sent subscribers who live in a particular city, given their city is described in their attributes, it's possible to quickly filter them out into a new list and e-mail them. [Learn more](querying-and-segmentation.md). + +## List + +A list (or a _mailing list_) is a collection of subscribers grouped under a name, for instance, _clients_. Lists are used to organise subscribers and send e-mails to specific groups. A list can be single optin or double optin. Subscribers added to double optin lists have to explicitly accept the subscription by clicking on the confirmation e-mail they receive. Until then, they do not receive campaign messages. + +## Campaign + +A campaign is an e-mail (or any other kind of messages) that is sent to one or more lists. + + +## Transactional message + +A transactional message is an arbitrary message sent to a subscriber using the transactional message API. For example a welcome e-mail on signing up to a service; an order confirmation e-mail on purchasing an item; a password reset e-mail when a user initiates an online account recovery process. + + +## Template + +A template is a re-usable HTML design that can be used across campaigns and when sending arbitrary transactional messages. Most commonly, templates have standard header and footer areas with logos and branding elements, where campaign content is inserted in the middle. listmonk supports [Go template](https://gowebexamples.com/templates/) expressions that lets you create powerful, dynamic HTML templates. [Learn more](templating.md). + +## Messenger + +listmonk supports multiple custom messaging backends in additional to the default SMTP e-mail backend, enabling not just e-mail campaigns, but arbitrary message campaigns such as SMS, FCM notifications etc. A *Messenger* is a web service that accepts a campaign message pushed to it as a JSON request, which the service can in turn broadcast as SMS, FCM etc. [Learn more](messengers.md). + +## Tracking pixel + +The tracking pixel is a tiny, invisible image that is inserted into an e-mail body to track e-mail views. This allows measuring the read rate of e-mails. While this is exceedingly common in e-mail campaigns, it carries privacy implications and should be used in compliance with rules and regulations such as GDPR. It is possible to track reads anonymously without associating an e-mail read to a subscriber. + +## Click tracking + +It is possible to track the clicks on every link that is sent in an e-mail. This allows measuring the clickthrough rates of links in e-mails. While this is exceedingly common in e-mail campaigns, it carries privacy implications and should be used in compliance with rules and regulations such as GDPR. It is possible to track link clicks anonymously without associating an e-mail read to a subscriber. + +## Bounce + +A bounce occurs when an e-mail that is sent to a recipient "bounces" back for one of many reasons including the recipient address being invalid, their mailbox being full, or the recipient's e-mail service provider marking the e-mail as spam. listmonk can automatically process such bounce e-mails that land in a configured POP mailbox, or via APIs of SMTP e-mail providers such as AWS SES and Sengrid. Based on settings, subscribers returning bounced e-mails can either be blocklisted or deleted automatically. [Learn more](bounces.md). diff --git a/docs/docs/content/configuration.md b/docs/docs/content/configuration.md new file mode 100644 index 000000000..11e7244fd --- /dev/null +++ b/docs/docs/content/configuration.md @@ -0,0 +1,150 @@ +# Configuration + +### TOML Configuration file +One or more TOML files can be read by passing `--config config.toml` multiple times. Apart from a few low level configuration variables and the database configuration, all other settings can be managed from the `Settings` dashboard on the admin UI. + +To generate a new sample configuration file, run `--listmonk --new-config` + +### Environment variables +Variables in config.toml can also be provided as environment variables prefixed by `LISTMONK_` with periods replaced by `__` (double underscore). To start listmonk purely with environment variables without a configuration file, set the environment variables and pass the config flag as `--config=""`. + +Example: + +| **Environment variable** | Example value | +| ------------------------------ | -------------- | +| `LISTMONK_app__address` | "0.0.0.0:9000" | +| `LISTMONK_db__host` | db | +| `LISTMONK_db__port` | 9432 | +| `LISTMONK_db__user` | listmonk | +| `LISTMONK_db__password` | listmonk | +| `LISTMONK_db__database` | listmonk | +| `LISTMONK_db__ssl_mode` | disable | + + +### Customizing system templates +See [system templates](templating.md#system-templates). + + +### HTTP routes +When configuring auth proxies and web application firewalls, use this table. + +#### Private admin endpoints. + +| Methods | Route | Description | +| ------- | ------------------ | ----------------------- | +| `*` | `/api/*` | Admin APIs | +| `GET` | `/admin/*` | Admin UI and HTML pages | +| `POST` | `/webhooks/bounce` | Admin bounce webhook | + + +#### Public endpoints to expose to the internet. + +| Methods | Route | Description | +| ----------- | --------------------- | --------------------------------------------- | +| `GET, POST` | `/subscription/*` | HTML subscription pages | +| `GET, ` | `/link/*` | Tracked link redirection | +| `GET` | `/campaign/*` | Pixel tracking image | +| `GET` | `/public/*` | Static files for HTML subscription pages | +| `POST` | `/webhooks/service/*` | Bounce webhook endpoints for AWS and Sendgrid | +| `GET` | `/uploads/*` | The file upload path configured in media settings | + + +## Media uploads + +#### Using filesystem + +When configuring `docker` volume mounts for using filesystem media uploads, you can follow either of two approaches. [The second option may be necessary if](https://github.com/knadh/listmonk/issues/1169#issuecomment-1674475945) your setup requires you to use `sudo` for docker commands. + +After making any changes you will need to run `sudo docker compose stop ; sudo docker compose up`. + +And under `https://listmonk.mysite.com/admin/settings` you put `/listmonk/uploads`. + +#### Using volumes + +Using `docker volumes`, you can specify the name of volume and destination for the files to be uploaded inside the container. + + +```yml +app: + volumes: + - type: volume + source: listmonk-uploads + target: /listmonk/uploads + +volumes: + listmonk-uploads: +``` + +!!! note + + This volume is managed by `docker` itself, and you can see find the host path with `docker volume inspect listmonk_listmonk-uploads`. + +#### Using bind mounts + +```yml + app: + volumes: + - ./path/on/your/host/:/path/inside/container +``` +Eg: +```yml + app: + volumes: + - ./data/uploads:/listmonk/uploads +``` +The files will be available inside `/data/uploads` directory on the host machine. + +To use the default `uploads` folder: +```yml + app: + volumes: + - ./uploads:/listmonk/uploads +``` + +## Logs + +### Docker + +https://docs.docker.com/engine/reference/commandline/logs/ +``` +sudo docker logs -f +sudo docker logs listmonk_app -t +sudo docker logs listmonk_db -t +sudo docker logs --help +``` +Container info: `sudo docker inspect listmonk_listmonk` + +Docker logs to `/dev/stdout` and `/dev/stderr`. The logs are collected by the docker daemon and stored in your node's host path (by default). The same can be configured (/etc/docker/daemon.json) in your docker daemon settings to setup other logging drivers, logrotate policy and more, which you can read about [here](https://docs.docker.com/config/containers/logging/configure/). + +### Binary + +listmonk logs to `stdout`, which is usually not saved to any file. To save listmonk logs to a file use `./listmonk > listmonk.log`. + +Settings -> Logs in admin shows the last 1000 lines of the standard log output but gets erased when listmonk is restarted. + +For the [service file](https://github.com/knadh/listmonk/blob/master/listmonk%40.service), you can use `ExecStart=/bin/bash -ce "exec /usr/bin/listmonk --config /etc/listmonk/config.toml --static-dir /etc/listmonk/static >>/etc/listmonk/listmonk.log 2>&1"` to create a log file that persists after restarts. [More info](https://github.com/knadh/listmonk/issues/1462#issuecomment-1868501606). + + +## Time zone + +To change listmonk's time zone (logs, etc.) edit `docker-compose.yml`: +``` +environment: + - TZ=Etc/UTC +``` +with any Timezone listed [here](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). Then run `sudo docker-compose stop ; sudo docker-compose up` after making changes. + +## SMTP + +### Retries +The `Settings -> SMTP -> Retries` denotes the number of times a message that fails at the moment of sending is retried silently using different connections from the SMTP pool. The messages that fail even after retries are the ones that are logged as errors and ignored. + +## SMTP ports +Some server hosts block outgoing SMTP ports (25, 465). You may have to contact your host to unblock them before being able to send e-mails. Eg: [Hetzner](https://docs.hetzner.com/cloud/servers/faq/#why-can-i-not-send-any-mails-from-my-server). + + +## Performance + +### Batch size + +The batch size parameter is useful when working with very large lists with millions of subscribers for maximising throughput. It is the number of subscribers that are fetched from the database sequentially in a single cycle (~5 seconds) when a campaign is running. Increasing the batch size uses more memory, but reduces the round trip to the database. diff --git a/docs/docs/content/developer-setup.md b/docs/docs/content/developer-setup.md new file mode 100644 index 000000000..b0e9658da --- /dev/null +++ b/docs/docs/content/developer-setup.md @@ -0,0 +1,42 @@ +# Developer setup +The app has two distinct components, the Go backend and the VueJS frontend. In the dev environment, both are run independently. + + +### Pre-requisites +- `go` +- `nodejs` (if you are working on the frontend) and `yarn` +- Postgres database. If there is no local installation, the demo docker DB can be used for development (`docker compose up demo-db`) + + +### First time setup +`git clone https://github.com/knadh/listmonk.git`. The project uses go.mod, so it's best to clone it outside the Go src path. + +1. Copy `config.toml.sample` as `config.toml` and add your config. +2. `make dist` to build the listmonk binary. Once the binary is built, run `./listmonk --install` to run the DB setup. For subsequent dev runs, use `make run`. + +> [mailhog](https://github.com/mailhog/MailHog) is an excellent standalone mock SMTP server (with a UI) for testing and dev. + + +### Running the dev environment +You can run your dev environment locally or inside containers. + +After setting up the dev environment, you can visit `http://localhost:8080`. + + +1. Locally +- Run `make run` to start the listmonk dev server on `:9000`. +- Run `make run-frontend` to start the Vue frontend in dev mode using yarn on `:8080`. All `/api/*` calls are proxied to the app running on `:9000`. Refer to the [frontend README](https://github.com/knadh/listmonk/blob/master/frontend/README.md) for an overview on how the frontend is structured. + +2. Inside containers (Using Makefile) +- Run `make init-dev-docker` to setup container for db. +- Run `make dev-docker` to setup docker container suite. +- Run `make rm-dev-docker` to clean up docker container suite. + +3. Inside containers (Using devcontainer) +- Open repo in vscode, open command palette, and select "Dev Containers: Rebuild and Reopen in Container". + +It will set up db, and start frontend/backend for you. + + +# Production build +Run `make dist` to build the Go binary, build the Javascript frontend, and embed the static assets producing a single self-contained binary, `listmonk` diff --git a/docs/docs/content/external-integration.md b/docs/docs/content/external-integration.md new file mode 100644 index 000000000..9ced11c0c --- /dev/null +++ b/docs/docs/content/external-integration.md @@ -0,0 +1,11 @@ +# Integrating with external systems + +In many environments, a mailing list manager's subscriber database is not run independently but as a part of an existing customer database or a CRM. There are multiple ways of keeping listmonk in sync with external systems. + +## Using APIs + +The [subscriber APIs](apis/subscribers.md) offers several APIs to manipulate the subscribers database, like addition, updation, and deletion. For bulk synchronisation, a CSV can be generated (and optionally zipped) and posted to the import API. + +## Interacting directly with the DB + +listmonk uses tables with simple schemas to represent subscribers (`subscribers`), lists (`lists`), and subscriptions (`subscriber_lists`). It is easy to add, update, and delete subscriber information directly with the database tables for advanced usecases. See the [table schemas](https://github.com/knadh/listmonk/blob/master/schema.sql) for more information. diff --git a/docs/docs/content/i18n.md b/docs/docs/content/i18n.md new file mode 100644 index 000000000..e8829e54d --- /dev/null +++ b/docs/docs/content/i18n.md @@ -0,0 +1,35 @@ +# Internationalization (i18n) + +listmonk comes available in multiple languages thanks to language packs contributed by volunteers. A language pack is a JSON file with a map of keys and corresponding translations. The bundled languages can be [viewed here](https://github.com/knadh/listmonk/tree/master/i18n). + +## Additional language packs +These additional language packs can be downloaded and passed to listmonk with the `--i18n-dir` flag as described in the next section. + +| Language | Description | +|------------------|--------------------------------------| +| [Deutsch (formal)](https://raw.githubusercontent.com/SvenPe/listmonk/4bbb2e5ebb2314b754cb2318f4f6683a0f854d43/i18n/de.json) | German language with formal pronouns | + + +## Customizing languages + +To customize an existing language or to load a new language, put one or more `.json` language files in a directory, and pass the directory path to listmonk with the
+
+
+
+
+
+
+## Tutorials
+
+* [Informal step-by-step on how to get started with listmonk using *Railway*](https://github.com/knadh/listmonk/issues/120#issuecomment-1421838533)
+* [Step-by-step tutorial for installation and all basic functions. *Amazon EC2, SES, docker & binary*](https://gist.github.com/MaximilianKohler/e5158fcfe6de80a9069926a67afcae11)
+* [Step-by-step guide on how to install and set up listmonk on *AWS Lightsail with docker* (rameerez)](https://github.com/knadh/listmonk/issues/1208)
+* [Quick setup on any cloud server using *docker and caddy*](https://github.com/samyogdhital/listmonk-caddy-reverse-proxy)
+* [*Binary* install on Ubuntu 22.04 as a service](https://mumaritc.hashnode.dev/how-to-install-listmonk-using-binary-on-ubuntu-2204)
+* [*Binary* install on Ubuntu 18.04 as a service (Apache & Plesk)](https://devgypsy.com/post/2020-08-18-installing-listmonk-newsletter-manager/)
+* [*Binary and docker* on linux (techviewleo)](https://techviewleo.com/manage-mailing-list-and-newsletter-using-listmonk/)
+* [*Binary* install on your PC](https://www.youtube.com/watch?v=fAOBqgR9Yfo). Discussions of limitations: [[1](https://github.com/knadh/listmonk/issues/862#issuecomment-1307328228)][[2](https://github.com/knadh/listmonk/issues/248#issuecomment-1320806990)].
+* [*Docker on Rocky Linux 8* (nginx, Let's Encrypt SSL)](https://wiki.crowncloud.net/?How_to_Install_Listmonk_with_Docker_on_Rocky_Linux_8)
+* [*Docker* with nginx reverse proxy, certbot SSL, and Gmail SMTP](https://www.maketecheasier.com/create-own-newsletter-with-listmonk/)
+* [Install Listmonk on Self-hosting with *Pre-Configured AMI Package at AWS* by Single Click](https://meetrix.io/articles/how-to-install-llama-2-on-aws-with-pre-configured-ami-package/)
+* [Tutorial for deploying on *Fly.io*](https://github.com/paulrudy/listmonk-on-fly) -- Currently [not working](https://github.com/knadh/listmonk/issues/984#issuecomment-1694545255)
diff --git a/docs/docs/content/maintenance/performance.md b/docs/docs/content/maintenance/performance.md
new file mode 100644
index 000000000..fc012e5ab
--- /dev/null
+++ b/docs/docs/content/maintenance/performance.md
@@ -0,0 +1,18 @@
+# Performance
+
+listmonk is built to be highly performant and can handle millions of subscribers with minimal system resources.
+
+However, as the Postgres database grows—with a large number of subscribers, campaign views, and click records—it can significantly slow down certain aspects of the program, particularly in counting records and aggregating various statistics. For instance, loading admin pages that do these aggregations can take tens of seconds if the database has millions of subscribers.
+
+- Aggregate counts, statistics, and charts on the landing dashboard.
+- Subscriber count beside every list on the Lists page.
+- Total subscriber count on the Subscribers page.
+
+However, at that scale, viewing the exact number of subscribers or statistics every time the admin panel is accessed becomes mostly unnecessary. On installations with millions of subscribers, where the above pages do not load instantly, it is highly recommended to turn on the `Settings -> Performance -> Cache slow database queries` option.
+
+## Slow query caching
+
+When this option is enabled, the subscriber counts on the Lists page, the Subscribers page, and the statistics on the dashboard, etc., are no longer counted in real-time in the database. Instead, they are updated periodically and cached, resulting in a massive performance boost. The periodicity can be configured on the Settings -> Performance page using a standard crontab expression (default: `0 3 * * *`, which means 3 AM daily). Use a tool like [crontab.guru](https://crontab.guru) for easily generating a desired crontab expression.
+
+## VACUUM-ing
+Running [`VACUUM ANALYZE`](https://www.postgresql.org/docs/current/sql-vacuum.html) on large Postgres databases at regular intervals (for instance, once a week), is recommended. It reclaims disk space and improves Postgres' query performance. Do note that this is a blocking operation and all database queries can come to a stand-still on a large database while the operation is running (generally only a few seconds).
diff --git a/docs/docs/content/messengers.md b/docs/docs/content/messengers.md
new file mode 100644
index 000000000..fbbeeb0c2
--- /dev/null
+++ b/docs/docs/content/messengers.md
@@ -0,0 +1,47 @@
+# Messengers
+
+listmonk supports multiple custom messaging backends in additional to the default SMTP e-mail backend, enabling not just e-mail campaigns, but arbitrary message campaigns such as SMS, FCM notifications etc.
+
+A *Messenger* is a web service that accepts a campaign message pushed to it as a JSON request, which the service can in turn broadcast as SMS, FCM etc. Messengers are registered in the *Settings -> Messengers* UI, and can be selected on individual campaigns.
+
+Messengers support optional BasicAuth authentication. `Plain text` format for campaign content is ideal for messengers such as SMS and FCM.
+
+When a campaign starts, listmonk POSTs messages in the following format to the selected messenger's endpoint. The endpoint should return a `200 OK` response in case of a successful request.
+
+The address required to broadcast the message, for instance, a phone number or an FCM ID, is expected to be stored and relayed as [subscriber attributes](concepts.md/#attributes).
+
+```json
+{
+ "subject": "Welcome to listmonk",
+ "body": "The message body",
+ "content_type": "plain",
+ "recipients": [{
+ "uuid": "e44b4135-1e1d-40c5-8a30-0f9a886c2884",
+ "email": "anon@example.com",
+ "name": "Anon Doe",
+ "attribs": {
+ "phone": "123123123",
+ "fcm_id": "2e7e4b512e7e4b512e7e4b51",
+ "city": "Bengaluru"
+ },
+ "status": "enabled"
+ }],
+ "campaign": {
+ "uuid": "2e7e4b51-f31b-418a-a120-e41800cb689f",
+ "name": "Test campaign",
+ "tags": ["test-campaign"]
+ }
+}
+```
+
+## Messenger implementations
+
+Following is a list of HTTP messenger servers that connect to various backends.
+
+| Name | Backend |
+|:-------------------------------------------------------------------------------------|:-----------------|
+| [listmonk-messenger](https://github.com/joeirimpan/listmonk-messenger) | AWS Pinpoint SMS |
+| [listmonk-verimor-gateway](https://github.com/antandros/listmonk-verimor-gateway) | Verimor |
+| [listmonk-mailersend](https://github.com/tkawczynski/listmonk-mailersend) | Mailersend |
+| [listmonk-novu-messenger](https://github.com/Codepowercode/listmonk-novu-messenger) | Novu |
+| [listmonk-push-messenger](https://github.com/shyamkrishna21/listmonk-push-messenger) | Google FCM |
diff --git a/docs/docs/content/querying-and-segmentation.md b/docs/docs/content/querying-and-segmentation.md
new file mode 100644
index 000000000..ea79ae381
--- /dev/null
+++ b/docs/docs/content/querying-and-segmentation.md
@@ -0,0 +1,102 @@
+# Querying and segmenting subscribers
+
+listmonk allows the writing of partial Postgres SQL expressions to query, filter, and segment subscribers.
+
+## Database fields
+
+These are the fields in the subscriber database that can be queried.
+
+| Field | Description |
+| ------------------------ | --------------------------------------------------------------------------------------------------- |
+| `subscribers.uuid` | The randomly generated unique ID of the subscriber |
+| `subscribers.email` | E-mail ID of the subscriber |
+| `subscribers.name` | Name of the subscriber |
+| `subscribers.status` | Status of the subscriber (enabled, disabled, blocklisted) |
+| `subscribers.attribs` | Map of arbitrary attributes represented as JSON. Accessed via the `->` and `->>` Postgres operator. |
+| `subscribers.created_at` | Timestamp when the subscriber was first added |
+| `subscribers.updated_at` | Timestamp when the subscriber was modified |
+
+## Sample attributes
+
+Here's a sample JSON map of attributes assigned to an imaginary subscriber.
+
+```json
+{
+ "city": "Bengaluru",
+ "likes_tea": true,
+ "spoken_languages": ["English", "Malayalam"],
+ "projects": 3,
+ "stack": {
+ "frameworks": ["echo", "go"],
+ "languages": ["go", "python"],
+ "preferred_language": "go"
+ }
+}
+```
+
+
+
+## Sample SQL query expressions
+
+
+
+#### Find a subscriber by e-mail
+
+```sql
+-- Exact match
+subscribers.email = 'some@domain.com'
+
+-- Partial match to find e-mails that end in @domain.com.
+subscribers.email LIKE '%@domain.com'
+
+```
+
+#### Find a subscriber by name
+
+```sql
+-- Find all subscribers whose name start with John.
+subscribers.email LIKE 'John%'
+
+```
+
+#### Multiple conditions
+
+```sql
+-- Find all Johns who have been blocklisted.
+subscribers.email LIKE 'John%' AND status = 'blocklisted'
+```
+
+#### Querying subscribers who viewed the campaign email
+
+```sql
+-- Find all subscribers who viewed the campaign email.
+EXISTS(SELECT 1 FROM campaign_views WHERE campaign_views.subscriber_id=subscribers.id AND campaign_views.campaign_id=+ Changes are stored in the browser's localStorage until the cache is cleared. + To edit an existing language, load it and edit the fields. + To create a new language, load the default language and edit the fields. + Once done, copy the raw JSON and send a PR to the + repo. +
+ +
+ + Live demo +
+
+ 
+ 
+ 
+ + The latest version is {{ .Page.Site.Data.github.version }} + released on {{ .Page.Site.Data.github.date | dateFormat "02 Jan 2006" }}. + See release notes. +
./listmonk --new-config to generate config.toml. Edit it.
+ ./listmonk --install to setup the Postgres DB or --upgrade to upgrade an existing DB../listmonk and visit http://localhost:9000
+
+
+
+ *listmonk has no affiliation with any of these providers.
++ Download and use the sample docker-compose.yml +
+ ++# Download the compose file to the current directory. +curl -LO https://github.com/knadh/listmonk/raw/master/docker-compose.yml + +# Run the services in the background. +docker compose up -d ++
Visit http://localhost:9000
+ + Manage millions of subscribers across many single and double opt-in one-way mailing lists + with custom JSON attributes for each subscriber. + Query and segment subscribers with SQL expressions. +
+Use the fast bulk importer (~10k records per second) or use HTTP/JSON APIs or interact with the simple + table schema to integrate external CRMs and subscriber databases. +
+
+ + Simple API to send arbitrary transactional messages to subscribers + using pre-defined templates. Send messages as e-mail, SMS, Whatsapp messages or any medium via Messenger interfaces. +
+
+ + Simple analytics and visualizations. Connect external visualization programs to the database easily with the simple table structure. +
+
+ + Create powerful, dynamic e-mail templates with the Go templating language. + Use template expressions, logic, and 100+ functions in subject lines and content. + Write HTML e-mails in a WYSIWYG editor, Markdown, raw syntax-highlighted HTML, or just plain text. +
+
+
+ + Multi-threaded, high-throughput, multi-SMTP e-mail queues. + Throughput and sliding window rate limiting for fine grained control. + Single binary application with nominal CPU and memory footprint that runs everywhere. + The only dependency is a Postgres (⩾ 12) database. +
+
+ Use the media manager to upload images for e-mail campaigns + on the server's filesystem, Amazon S3, or any S3 compatible (Minio) backend.
+
+ + More than just e-mail campaigns. Connect HTTP webhooks to send SMS, + Whatsapp, FCM notifications, or any type of messages. +
+
+ + Allow subscribers to permanently blocklist themselves, export all their data, + and to wipe all their data in a single click. +
+
+ + listmonk is free and open source software licensed under AGPLv3. + If you are interested in contributing, check out the GitHub repository + and refer to the developer setup. + The backend is written in Go and the frontend is Vue with Buefy for UI. +
+