Skip to content

Commit

Permalink
Added info about caching to the documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
maximecb committed Aug 29, 2017
1 parent da1105f commit 7c316f7
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 0 deletions.
18 changes: 18 additions & 0 deletions docs/caching.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
Caching datasets locally
========================

In some use cases, it may be desirable to set Fuel's ``data_path`` to
point to a shared network drive. For example, when configuring multiple
machines in a cluster to work on the same data in parallel.
However, this can easily cause network bandwidth to become saturated.

To avoid this problem, Fuel provides a second configuration variable
named ``local_data_path``, which can be set in ``~/.fuelrc``. This
variable points to a filesystem directory to be used to act as a local
cache for datasets.

This variable can also be set through an environment variable as follows:

.. code-block:: bash
$ export FUEL_LOCAL_DATA_PATH="/home/username/my_local_cache"
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Welcome to Fuel's documentation!
new_dataset
extending_fuel
server
caching
api/index

Fuel is a data pipeline framework which provides your machine learning models
Expand Down

0 comments on commit 7c316f7

Please sign in to comment.