Skip to content

Commit

Permalink
- updated README.md
Browse files Browse the repository at this point in the history
- insignificant changes in code
  • Loading branch information
OndrejValenta committed Apr 5, 2024
1 parent 916a4ac commit 8d9c485
Show file tree
Hide file tree
Showing 4 changed files with 67 additions and 27 deletions.
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -199,4 +199,4 @@
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
limitations under the License.
86 changes: 62 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,68 +1,108 @@
# LokiLogger

LokiLogger is an Elixir logger backend providing support for Logging to [Grafana Loki](https://github.com/grafana/loki)
KeenLokiLogger is an Elixir logger backend, based on original [LokiLogger](https://github.com/wardbekker/LokiLogger), which development is dead from 2019, providing support for Logging to [Grafana Loki](https://github.com/grafana/loki)

[![Hex.pm Version](http://img.shields.io/hexpm/v/loki_logger.svg?style=flat)](https://hex.pm/packages/loki_logger)
[![Hex.pm Version](http://img.shields.io/hexpm/v/keen_loki_logger.svg?style=flat)](https://hex.pm/packages/keen_loki_logger)

## Known issues

* "works-on-my-machine" level of quality. Love to get your feedback in the repo's Github issues
* "works-on-my-machine" level of quality, but we use it in a big enterprise company. Post a feedback in the repo's Github issues.

## Features (and TODO)

Some of the features were reworked, some implemented and some are still missing. The main part is there though.

* [x] Elixir Logger formatting
* [x] Elixir Logger metadata
* [x] Loki Scope-Org-Id header for multi-tenancy
* [x] Loki Scope-Org-Id header for multi-tenancy
* [x] Timezone aware
* [X] Snappy compressed proto format in the HTTP Body
* [ ] Async http call to backend.
* [ ] Proper unit tests.
* [ ] HTTP post retry strategy on temporary loki backend failure or network hiccups.
* [ ] Authentication with basic auth and/or Oauth2 reverse proxy.
* [ ] CI build integration (e.g. TravisCI)
* [X] Authentication with basic auth.
* [ ] Authentication with Oauth2 reverse proxy


## Installation

The package can be installed by adding `loki_logger` to your list of dependencies in `mix.exs`:
The package can be installed by adding `keen_loki_logger` to your list of dependencies in `mix.exs`:

```elixir
def deps do
[
{:loki_logger, "~> 0.3.0"}
{:keen_loki_logger, "~> 0.5.1"}
]
end
```

### WINDOWS MACHINES STRUGGLES

On Windows machines you might have issues compiling your project because of `snappyer` package that requires C/++ code compilation.
Use these steps to be able to compile it without issues:

- Use command line and not PowerShell!
- Install Microsoft C++ toolset, you don't have to go full C++ development, just the compilation toolset is fine
- Add this c:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\Hostx64\x64\ or similar path (based on your installed version) to PATH environment variable, this will allow add cl.exe to path
- Before compilation run "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat" which you should have by now. This setups all environment variables needed for compilation of C/++ code, including standard .h files

We have this script in our project to setup the environment.

`setup.bat`
```bat
call "c:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"
mix deps.get && mix deps.compile
```

## Configuration

### Elixir Project

Loki Logger's behavior is controlled using the application configuration environment:

* __loki_host__ : the hostname of the syslog server e.g. http://localhost:3100
* __loki_labels__ : the Loki log labels used to select the log stream in e.g. Grafana
* __loki_scope_org_id__: optional tenant ID for multitenancy. Currently not (yet?) supported in Grafana when enforced with `auth_enabled: true` in Loki config
* __level__: logging threshold. Messages "above" this threshold will be discarded. The supported levels, ordered by precedence are :debug, :info, :warn, :error.
* __format__: the format message used to print logs. Defaults to: "$metadata level=$level $levelpad$message". It may also be a {module, function} tuple that is invoked with the log level, the message, the current timestamp and the metadata.
* __metadata__: the metadata to be printed by $metadata. Defaults to to :all, which prints all metadata.
* __max_buffer__: the amount of entries to buffer before posting to the Loki REST api. Defaults to 32.
* __loki_host__ : [OPTIONAL] the hostname of the syslog server, default `http://localhost:3100`
* __loki_path__ : [OPTIONAL] path to the endpoint, default `/loki/api/v1/push`
* __loki_labels__ : [OPTIONAL] the Loki log labels used to select the log stream in, default `%{application: "loki_logger_library"}`
* __loki_scope_org_id__: [OPTIONAL] optional tenant ID for multitenancy, default `fake`, which is a standard Loki value when you have just one tenant
* __level__: [OPTIONAL] logging threshold. Messages "above" this threshold will be discarded. The supported levels, ordered by precedence are :debug, :info, :warn, :error.
* __format__: [OPTIONAL] the format message used to print logs. Defaults to: "$metadata level=$level $levelpad$message". It may also be a {module, function} tuple that is invoked with the log level, the message, the current timestamp and the metadata.
* __metadata__: [OPTIONAL] the metadata to be printed by $metadata. Defaults to :all, which prints all metadata to the message.
* __max_buffer__: [OPTIONAL] the amount of entries to buffer before posting to the Loki REST api. Defaults to 32.

* __finch_protocols__: [OPTIONAL] the HTTP protocol to use when communicating with Loki, supported values are `:http1` and `:http2`, default `:http1`.
* __finch_pool_size__: [OPTIONAL] the amount of HTTP connection pools to create, applicable only for HTTP1, for HTTP2 the value is always set to 1, because HTTP2 is by itself multiplexed, default 16.
* __finch_pool_count__: [OPTIONAL] the amount of HTTP connection pools to create, default `4`.
* __finch_pool_max_idle_time__: [OPTIONAL] I have no idea what this does :-)
* __mint_conn_opts__: [OPTIONAL] Mint is the low-level HTTP client library and this is the way to set it up. We use it, for example, to setup CA certificate for internal servers.


For example, the following `config/config.exs` file sets up Loki Logger using
level debug, with `application` label `loki_logger_library`.

`config.exs`
```elixir
use Mix.Config

config :logger,
backends: [LokiLogger]

config :logger, :loki_logger,
level: :debug,
format: "$metadata level=$level $levelpad$message",
metadata: :all,
max_buffer: 300,
loki_labels: %{application: "loki_logger_library", elixir_node: node()},
loki_host: "http://localhost:3100"
config :logger, :keen_loki_logger,
level: :debug,
format: "$metadata level=$level $message",
metadata: [:erl_level, :application, :file, :module, :function],
max_buffer: 300,
loki_host: "https://1.2.3.4:3100",
basic_auth_user: "lokiuser",
basic_auth_password: "lokipassword",
finch_protocols: [:http2],
mint_conn_opts: [transport_opts: [cacerts: :public_key.cacerts_get()]]
```

`runtime.exs`
```elixir
config :logger, :keen_loki_logger,
loki_labels: %{app: "dummy_logging_service", env: System.get_env("APP_ENV"), host: System.get_env("COMPUTER_NAME")}
```

# Protobuff lib regeneration
Expand All @@ -75,8 +115,6 @@ protoc --proto_path=./lib/proto --elixir_out=./lib lib/proto/loki.proto

## License

Loki Logger is copyright (c) 2019 Ward Bekker

The source code is released under the Apache v2.0 License.

Check [LICENSE](LICENSE) for more information.
Expand Down
5 changes: 3 additions & 2 deletions lib/loki_logger.ex
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ defmodule LokiLogger do
finch_pool_max_idle_time = Keyword.get(config, :finch_pool_max_idle_time, 10_000)
mint_conn_opts = Keyword.get(config, :mint_conn_opts)

loki_labels = Keyword.get(config, :loki_labels, %{application: "loki_logger_library"})

%{
state
| format: format,
Expand Down Expand Up @@ -121,8 +123,7 @@ defmodule LokiLogger do
},
{Task.Supervisor, name: LokiLogger.TaskSupervisor},
{LokiLogger.Exporter,
loki_labels:
Keyword.get(config, :loki_labels, %{application: "loki_logger_library"}),
loki_labels: loki_labels,
loki_url: loki_url,
tesla_client: config |> tesla_client()}
],
Expand Down
1 change: 1 addition & 0 deletions lib/loki_logger/exporter.ex
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ defmodule LokiLogger.Exporter do
|> Enum.join(",")

labels = "{" <> labels <> "}"

# sort entries on epoch seconds as first element of tuple, to prevent out-of-order entries
sorted_entries =
output
Expand Down

0 comments on commit 8d9c485

Please sign in to comment.