Skip to content

Commit

Permalink
updating documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
gonzalo committed May 31, 2017
1 parent ad98f5a commit d9df031
Show file tree
Hide file tree
Showing 6 changed files with 208 additions and 38 deletions.
80 changes: 68 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,75 @@
# Piton

**TODO: Add description**

## Installation
[![hex.pm](https://img.shields.io/hexpm/v/piton.svg?style=flat-square)](https://hex.pm/packages/piton) [![hexdocs.pm](https://img.shields.io/badge/docs-latest-green.svg?style=flat-square)](https://hexdocs.pm/piton/) [![Build Status](https://travis-ci.org/mendrugory/piton.svg?branch=master)](https://travis-ci.org/mendrugory/piton)

`Piton` is a library which will help you to run your Python code.

If [available in Hex](https://hex.pm/docs/publish), the package can be installed
by adding `piton` to your list of dependencies in `mix.exs`:
You can implement your own `Piton.Port` and run your python code but I highly recommend to use `Piton.Pool`,
a pool which will allow to run Python code in parallel, a way of avoiding the GIL, and it will protect you from
python exceptions.

## Installation
Add `piton` to your list of dependencies in `mix.exs`:

```elixir
def deps do
[{:piton, "~> 0.1.0"}]
end
```
```elixir
def deps do
[{:piton, "~> 0.1.0"}]
end
```


Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc)
and published on [HexDocs](https://hexdocs.pm). Once published, the docs can
be found at [https://hexdocs.pm/piton](https://hexdocs.pm/piton).
## How to use it
Define your own `port`

* The Easiest one
```elixir
defmodule MySimplePort do
use Piton.Port
end
```

* A port with some wrapper functions which will help you to call the python function:
*YOUR_MODULE.execute(pid, python_module, python_function, list_of_arguments)*
```elixir
defmodule MyCustomPort do
use Piton.Port
def start_link(), do: MyCustomPort.start_link([path: Path.expand("python_folder"), python: "python"], [name: __MODULE__])
def fun(n), do: MyCustomPort.execute(__MODULE__, :functions, :fun, [n])
end
```

* A port prepared to be run by `Piton.Pool`
They have to have a function *start()* and it has not to be linked.
```elixir
defmodule MyPoolPort do
use Piton.Port
def start(), do: MyPoolPort.start([path: Path.expand("python_folder"), python: "python"], [])
def fun(n), do: MyPoolPort.execute(__MODULE__, :functions, :fun, [n])
end
```

### Run a Pool
Pay attention to the number of Pythons you want to run in parallel. It does not exist an optimal number, maybe it is the
number of cores, maybe half or maybe double. Test it with your application.
```elixir
{:ok, pool} = Piton.Pool.start_link([module: MyPoolPort, pool_number: pool_number], [])
```
### Call a Port (No pool)
```elixir
iex> MyCustomPort.execute(pid_of_the_port, python_module, python_function, list_of_arguments_of_python_function)
```

### Call a Pool
```elixir
iex> Piton.Pool.execute(pid_of_the_pool, elixir_function, list_of_arguments_of_elixir_function)
```

## Test
Run the tests.
```bash
mix test
```

## Name
Pitón is only Python in Spanish :stuck_out_tongue_winking_eye: :snake:
74 changes: 73 additions & 1 deletion lib/piton.ex
Original file line number Diff line number Diff line change
@@ -1,6 +1,78 @@
defmodule Piton do
@moduledoc """
Documentation for Piton.
# Piton
`Piton` is a library which will help you to run your Python code.
You can implement your own `Piton.Port` and run your python code but I highly recommend to use `Piton.Pool`,
a pool which will allow to run Python code in parallel, a way of avoiding the GIL, and it will protect you from
python exceptions.
## Installation
Add `piton` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[{:piton, "~> 0.1.0"}]
end
```
## How to use it
Define your own `port`
* The Easiest one
```elixir
defmodule MySimplePort do
use Piton.Port
end
```
* A port with some wrapper functions which will help you to call the python function:
*YOUR_MODULE.execute(pid, python_module, python_function, list_of_arguments)*
```elixir
defmodule MyCustomPort do
use Piton.Port
def start_link(), do: MyCustomPort.start_link([path: Path.expand("python_folder"), python: "python"], [name: __MODULE__])
def fun(n), do: MyCustomPort.execute(__MODULE__, :functions, :fun, [n])
end
```
* A port prepared to be run by `Piton.Pool`
They have to have a function *start()* and it has not to be linked.
```elixir
defmodule MyPoolPort do
use Piton.Port
def start(), do: MyPoolPort.start([path: Path.expand("python_folder"), python: "python"], [])
def fun(n), do: MyPoolPort.execute(__MODULE__, :functions, :fun, [n])
end
```
### Run a Pool
Pay attention to the number of Pythons you want to run in parallel. It does not exist an optimal number, maybe it is the
number of cores, maybe half or maybe double. Test it with your application.
```elixir
{:ok, pool} = Piton.Pool.start_link([module: MyPoolPort, pool_number: pool_number], [])
```
### Call a Port (No pool)
```elixir
iex> MyCustomPort.execute(pid_of_the_port, python_module, python_function, list_of_arguments_of_python_function)
```
### Call a Pool
```elixir
iex> Piton.Pool.execute(pid_of_the_pool, elixir_function, list_of_arguments_of_elixir_function)
```
## Test
Run the tests.
```bash
mix test
```
## Name
Pitón is only Python in Spanish :stuck_out_tongue_winking_eye: :snake:
"""

end
32 changes: 25 additions & 7 deletions lib/piton/pool.ex
Original file line number Diff line number Diff line change
@@ -1,10 +1,25 @@
defmodule Piton.Pool do
@moduledoc"""
Piton.Pool is a `GenServer` which will be on charge of a pool of Python Ports.
`args` arguments is a Keyword List and has to contain:
* module: Module which has to `use Piton.Port`
* pool_number: number of available Pythons.
`Piton.Pool` is a `GenServer` which will be on charge of a pool of `Piton.Port`s.
`Piton.Pool` will launch as many Python processes as you define in `pool_number` and it will share them between all the request (executions)
it receives. It is also protected from Python exceptions, therefore, if a Python code raises an exception that can close the port, a new one
will be opened and added it to the pool.
## Start a Pool
```elixir
{:ok, pool} = Piton.Pool.start_link([module: MyPoolPort, pool_number: pool_number], [])
```
The arguments has to be in a Keyword List and it has to contain:
module: Module which has to `use Piton.Port`
pool_number: number of available Pythons.
## Run a Python code using the pool
```elixir
Piton.Pool.execute(pid_of_the_pool, elixir_function, list_of_arguments_of_elixir_function)
```
### Timeout
`Piton.Port.execution` function has a `timeout`, this timeout will be passes as timeout to the `Piton.Port.execution` function.
"""

@timeout 5000
Expand All @@ -20,18 +35,21 @@ defmodule Piton.Pool do
@doc """
It will execute the arguments in the given function of the given module using the given pool of ports.
"""
@spec execute(pid, atom, list, timeout) :: {:ok, any} | {:error, any}
def execute(pid, python_function, python_arguments, timeout \\ @timeout) do
GenServer.call(pid, {:execute, python_function, python_arguments, timeout}, timeout)
end

@doc """
It will return the number of available ports
It will return the number of available ports.
"""
@spec get_number_of_available_ports(pid) :: integer
def get_number_of_available_ports(pid), do: GenServer.call(pid, :number_of_available_ports)

@doc """
It will return the number of processes that are waiting for an available port
It will return the number of processes that are waiting for an available port.
"""
@spec get_number_of_waiting_processes(pid) :: integer
def get_number_of_waiting_processes(pid), do: GenServer.call(pid, :number_of_waiting_processes)

def init([module: module, pool_number: pool_number]) do
Expand Down
33 changes: 26 additions & 7 deletions lib/piton/port.ex
Original file line number Diff line number Diff line change
@@ -1,18 +1,35 @@
defmodule Piton.Port do
@moduledoc"""
Piton.Port is a `GenServer` which will be on charge of the open Python Port.
`Piton.Port` is a `GenServer` which will be on charge of the Python Port.
`args` arguments is a Keyword List and has to contain:
* path: Path to the folder where the python scripts are.
* python: python executable
It is prepared to be the base of your own Port:
It is prepared to be the base of your own Port.
## Make your own Port
```elixir
defmodule MyPort do
use Piton.PythonPort
# rest of the code if it is need.
end
```
The arguments has to be in a Keyword List and it has to contain:
path: Path to the folder where the python scripts are.
python: python executable
If your port is going to run in a `Piton.Pool` (highly recommended) it has to have a *start()* function
and it has not to be linked.
```elixir
defmodule MyPoolPort do
use Piton.Port
def start(), do: MyPoolPort.start([path: Path.expand("python_folder"), python: "python"], [])
def fun(n), do: MyPoolPort.execute(__MODULE__, :functions, :fun, [n])
end
```
## Run a Python code using directly the port (no pool)
```elixir
iex> MyPort.execute(pid_of_the_port, python_module, python_function, list_of_arguments_of_python_function)
```
"""

defmacro __using__(_) do
Expand All @@ -32,11 +49,13 @@ defmodule Piton.Port do
@doc """
It will return the erl port
"""
@spec get_port(pid) :: pid
def get_port(pid), do: GenServer.call(pid, :get_python_port)

@doc """
It will execute the arguments in the given function of the given module using the given port.
"""
@spec execute(pid, atom, atom, list, timeout) :: any
def execute(pid, python_module, python_function, python_arguments, timeout \\ @timeout) do
GenServer.call(pid, {:execute, python_module, python_function, python_arguments}, timeout)
end
Expand Down
23 changes: 13 additions & 10 deletions mix.exs
Original file line number Diff line number Diff line change
Expand Up @@ -7,23 +7,26 @@ defmodule Piton.Mixfile do
[app: :piton,
version: @version,
elixir: "~> 1.4",
package: package(),
description: "Run your Python algorithms in parallel and forget the GIL",
build_embedded: Mix.env == :prod,
start_permanent: Mix.env == :prod,
deps: deps()]
deps: deps(),
docs: [main: "Piton", source_ref: "v#{@version}",
source_url: "https://github.com/mendrugory/piton"]]
end

# Configuration for the OTP application
#
# Type "mix help compile.app" for more information
def application do
# Specify extra applications you'll use from Erlang/Elixir
[extra_applications: [:logger]]
end

defp deps do
defp deps() do
[{:erlport, "~> 0.9.8"},
{:earmark, ">= 0.0.0", only: :dev},
{:ex_doc, ">= 0.0.0", only: :dev}
]
end

defp package() do
%{licenses: ["MIT"],
maintainers: ["Gonzalo Jiménez Fuentes"],
links: %{"GitHub" => "https://github.com/mendrugory/piton"}}
end

end
4 changes: 3 additions & 1 deletion mix.lock
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
%{"erlport": {:hex, :erlport, "0.9.8", "b7dc57eb87f215a671926bfbcd23e6e9c76f8653b0d072627b41431ef51c4d20", [:rebar3], []},
%{"earmark": {:hex, :earmark, "1.2.2", "f718159d6b65068e8daeef709ccddae5f7fdc770707d82e7d126f584cd925b74", [:mix], []},
"erlport": {:hex, :erlport, "0.9.8", "b7dc57eb87f215a671926bfbcd23e6e9c76f8653b0d072627b41431ef51c4d20", [:rebar3], []},
"ex_doc": {:hex, :ex_doc, "0.16.1", "b4b8a23602b4ce0e9a5a960a81260d1f7b29635b9652c67e95b0c2f7ccee5e81", [:mix], [{:earmark, "~> 1.1", [hex: :earmark, optional: false]}]},
"flow": {:hex, :flow, "0.12.0", "32c5a5f3ff6693e004b6c17a8c64dce2f8cdaf9564912d79427176013a586ab6", [:mix], [{:gen_stage, "~> 0.12.0", [hex: :gen_stage, optional: false]}]},
"gen_stage": {:hex, :gen_stage, "0.12.0", "74ec6f84ea0e0e81aa26b745870495094de1c59bfdbd012ae3103208ea124623", [:mix], []}}

0 comments on commit d9df031

Please sign in to comment.