Skip to content

Commit

Permalink
Prepare v1.0.0 (#51)
Browse files Browse the repository at this point in the history
Prepare v1.0.0
- [#52] Make shards a library to work locally and move shards_dist to a separate repo
- [#53] Support named and unnamed tables
- [#54] Refactor shards to use TIDs instead of names (atoms) for handling the partitions
- [#55] Use the metadata ETS table for storing the partitions TIDs, instead of building atoms in runtime for table names
- [#56] Create shards_group as a dynamic supervisor for creating/deleting tables dynamically as part of a supervision tree
- Improve docs
  • Loading branch information
cabol authored Oct 12, 2020
1 parent 15fff93 commit f79d021
Show file tree
Hide file tree
Showing 37 changed files with 3,361 additions and 5,092 deletions.
3 changes: 3 additions & 0 deletions .codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
comment:
behaviour: new
require_changes: yes
37 changes: 32 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,43 @@ on:

jobs:
test:
name: make ci (OTP ${{ matrix.otp }})
runs-on: ubuntu-latest
name: 'Shards Test (OTP ${{ matrix.otp }})'

strategy:
matrix:
otp: [22.3, 21.3]
otp: [23.1, 22.3]

steps:
- uses: actions/checkout@v2
- uses: gleam-lang/[email protected]
with:
otp-version: ${{ matrix.otp }}
- name: Starting EPMD
run: epmd -daemon

- name: Running Tests
run: make ci
run: |
make test
make covertool
- name: Running Edoc
run: make docs

- name: Running Xref
run: make xref

- uses: actions/cache@v1
with:
path: priv/plts
key: ${{ runner.os }}-plt-v2-${{ hashFiles(format('{0}{1}', github.workspace, '/rebar.lock')) }}
restore-keys: |
${{ runner.os }}-plt-v2-
- name: Running Dialyzer
run: make dialyzer

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
with:
file: _build/test/covertool/shards.covertool.xml
flags: otp-${{ matrix.otp }}
fail_ci_if_error: true
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,4 @@ priv/*.so
*_plt
doc
docs
/priv/plts
36 changes: 11 additions & 25 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@ REBAR = $(shell which rebar3)

EPMD_PROC_NUM = $(shell ps -ef | grep epmd | grep -v "grep")

LOCAL_SUITES = "test/shards_local_SUITE,test/shards_state_SUITE,test/shards_lib_SUITE,test/shards_task_SUITE"

.PHONY: all check_rebar compile clean distclean dialyzer test shell doc
.PHONY: all compile clean distclean test test_suite covertool dialyzer xref check shell docs

all: check_rebar compile

Expand All @@ -25,24 +23,24 @@ dialyzer: check_rebar
xref: check_rebar
$(REBAR) xref

ci: check_rebar check_epmd check_plt xref
test: check_rebar
$(REBAR) do ct, cover

test: check_rebar check_epmd check_plt
$(REBAR) do ct, cover
test_suite: check_rebar
$(REBAR) do ct --suite=test/$(SUITE)_SUITE, cover

covertool: check_rebar
$(REBAR) as test covertool generate

local_test: check_rebar check_epmd
$(REBAR) do ct --suite=$(LOCAL_SUITES), cover
docs: check_rebar
$(REBAR) edoc

dist_test: check_rebar check_epmd
$(REBAR) do ct --suite=test/shards_dist_SUITE, cover
check: test covertool dialyzer xref docs
@echo "OK!"

shell: check_rebar
$(REBAR) shell

doc: check_rebar
$(REBAR) edoc

check_rebar:
ifeq ($(REBAR),)
ifeq ($(wildcard rebar3),)
Expand All @@ -52,18 +50,6 @@ else
endif
endif

check_plt:
ifeq (,$(wildcard ./*_plt))
@echo " ---> Running dialyzer ..."
$(REBAR) dialyzer
endif

check_epmd:
ifeq ($(EPMD_PROC_NUM),)
epmd -daemon
@echo " ---> Started epmd!"
endif

define get_rebar
curl -O https://s3.amazonaws.com/rebar3/rebar3
chmod a+x rebar3
Expand Down
93 changes: 53 additions & 40 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,38 @@
<img src="http://38.media.tumblr.com/db32471b7c8870cbb0b2cc173af283bb/tumblr_inline_nm9x9u6u261rw7ney_540.gif" height="170" width="100%" />
<h1>
<img align="left" height="60px" src="guides/images/shards.png">
Shards
</h1>

# Shards
> ### ETS tables on steroids!
> Sharding support for ETS tables out-of-box.
> Sharding for ETS tables out-of-box.
[![Build Status](https://github.com/cabol/shards/workflows/CI/badge.svg)](https://github.com/cabol/shards/actions)
![CI](https://github.com/cabol/shards/workflows/CI/badge.svg)
[![Codecov](https://codecov.io/gh/cabol/shards/branch/master/graphs/badge.svg)](https://codecov.io/gh/cabol/shards/branch/master/graphs/badge.svg)
[![Hex Version](https://img.shields.io/hexpm/v/shards.svg)](https://hex.pm/packages/shards)


Why might we need **Sharding** on ETS tables? Well, the main reason is
to keep the lock contention under control, in order to scale-out ETS tables
(linearly) and support higher levels of concurrency without lock issues;
specially write-locks, which most of the cases might cause significant
performance degradation.
Why might we need **Sharding/Partitioning** for the ETS tables? The main reason
is to keep the lock contention under control enabling ETS tables to scale out
and support higher levels of concurrency without lock issues; specially
write-locks, which most of the cases might cause significant performance
degradation.

Therefore, one of the most common and proven strategies to deal with these
problems is [Sharding][sharding] or [Partitioning][partitioning]; the principle
is pretty similar to [DHTs][dht].

This is where **Shards** comes in. **Shards** is an **Erlang/Elixir** library
compatible with the current [ETS API][ets_api], which implements
[Sharding][sharding] or [Partitioning][partitioning] on top of ETS tables,
completely transparent and out-of-box.

See the [getting started][getting_started] guide
and the [online documentation](https://hexdocs.pm/shards/).
This is where [shards][shards] comes in. [shards][shards] is an Erlang/Elixir
library fully compatible with the [ETS API][ets_api], but it implements sharding
or partitioning on top of the ETS tables, completely transparent and out-of-box.

> [List of compatible ETS functions](https://github.com/cabol/shards/issues/1)
See the **[getting started][getting_started]** guide
and the **[online documentation](https://hexdocs.pm/shards/)**.

[ets_api]: http://erlang.org/doc/man/ets.html
[sharding]: https://en.wikipedia.org/wiki/Shard_(database_architecture)
[partitioning]: https://en.wikipedia.org/wiki/Partition_(database)
[dht]: https://en.wikipedia.org/wiki/Distributed_hash_table
[getting_started]: https://github.com/cabol/shards/blob/master/guides/getting-started.md
[shards]: https://hexdocs.pm/shards/shards.html
[getting_started]: guides/getting-started.md

## Installation

Expand All @@ -42,7 +42,7 @@ In your `rebar.config`:

```erlang
{deps, [
{shards, "0.6.1"}
{shards, "0.6.2"}
]}.
```

Expand All @@ -56,27 +56,25 @@ def deps do
end
```

> Check out the [getting started][getting_started] guide to learn
more about it.
> For more information and examples, see the [getting started][getting_started]
guide.

## Important links

* [Documentation](https://hexdocs.pm/shards) - Hex Docs.

* [Blog Post](http://cabol.github.io/posts/2016/04/14/sharding-support-for-ets.html) -
Transparent and out-of-box sharding support for ETS tables in Erlang/Elixir.

* [ExShards](https://github.com/cabol/ex_shards) – Elixir wrapper for
`shards`; with extra and nicer functions.

* [Nebulex](https://github.com/cabol/nebulex) – Distributed Caching
framework for Elixir.
* [Documentation](https://hexdocs.pm/shards) - Hex Docs.

* [KVX](https://github.com/cabol/kvx) – Simple Elixir in-memory Key/Value
Store using `shards` (default adapter).
* [Blog Post](http://cabol.github.io/posts/2016/04/14/sharding-support-for-ets.html) -
Transparent and out-of-box sharding support for ETS tables in Erlang/Elixir.

* [Cacherl](https://github.com/ferigis/cacherl) Distributed Cache
using `shards`.
* Projects using **shards**:
* [ExShards](https://github.com/cabol/ex_shards) – Elixir wrapper for
`shards`; with extra and nicer functions.
* [Nebulex](https://github.com/cabol/nebulex) – Distributed Caching
framework for Elixir.
* [KVX](https://github.com/cabol/kvx) – Simple Elixir in-memory Key/Value
Store using `shards` (default adapter).
* [Cacherl](https://github.com/ferigis/cacherl) Distributed Cache
using `shards`.

## Testing

Expand All @@ -91,14 +89,29 @@ You can find tests results in `_build/test/logs`, and coverage in
wrapper on top of `rebar3`, therefore, you can do everything using `rebar3`
directly as well (e.g.: `rebar3 do ct, cover`).

## Building Edoc
## Generating Edoc

```
$ make doc
$ make docs
```

> **NOTE:** Once you run the previous command, a new folder `doc` is created,
and you'll have a pretty nice HTML documentation.
> **NOTE:** Once you run the previous command, you will find the generated HTML
documentation within `doc` folder; open `doc/index.html`.

## Contributing

Contributions to **shards** are very welcome and appreciated!

Use the [issue tracker](https://github.com/cabol/shards/issues) for bug reports
or feature requests. Open a [pull request](https://github.com/cabol/shards/pulls)
when you are ready to contribute.

When submitting a pull request you should not update the [CHANGELOG.md](CHANGELOG.md),
and also make sure you test your changes thoroughly, include unit tests
alongside new or changed code.

Before to submit a PR it is highly recommended to run `make check` before and
ensure all checks run successfully.

## Copyright and License

Expand Down
29 changes: 0 additions & 29 deletions elvis.config

This file was deleted.

Loading

0 comments on commit f79d021

Please sign in to comment.