PyWren for IBM Cloud

What is PyWren

PyWren is an open source project whose goals are massively scaling the execution of Python code and its dependencies on serverless computing platforms and monitoring the results. PyWren delivers the user’s code into the serverless platform without requiring knowledge of how functions are invoked and run.

PyWren and IBM Cloud

This repository is based on PyWren main branch and adapted for IBM Cloud Functions and IBM Cloud Object Storage. PyWren for IBM Cloud is not, however, just a mere reimplementation of PyWren’s API atop IBM Cloud Functions. Rather, it is must be viewed as an advanced extension of PyWren. See the complete design overview.

PyWren for IBM Cloud provides great value for the variety of uses cases, like processing data in object storage, running embarrassingly parallel compute jobs (e.g. Monte-Carlo simulations), enriching data with additional attributes and many more. In extending PyWren to work with IBM Cloud Object Storage, we also added a partition discovery component that allows PyWren to process large amounts of data stored in the IBM Cloud Object Storage. See changelog for more details.

IBM Cloud for Academic institutions

IBM Academic Initiative is a special program that allows free trial of IBM Cloud for Academic institutions. This program is provided for students and faculty staff members, and allow up to 12 months of free usage. You can register your university email and get a free of charge account.

Initial Requirements

• IBM Cloud Functions account
• IBM Cloud Object Storage account
• Python >=3.5, <=3.8

PyWren Setup

1. Install PyWren from the PyPi repository:

   $ pip install pywren-ibm-cloud

2. Navigate into config/ and follow the instructions to configure PyWren.

3. Test PyWren by simply running the next command:

    $ pywren-ibm-cloud test

   or by running the next code:

   import pywren_ibm_cloud as pywren
   
   def hello(name):
       return 'Hello {}!'.format(name)
   
   pw = pywren.ibm_cf_executor()
   pw.call_async(hello, 'World')
   print(pw.get_result())

How to use PyWren for IBM Cloud

The primary object in PyWren is the executor. The standard way to get everything set up is to import pywren_ibm_cloud, and call one of the available functions to get a ready-to-use executor.

The available executors are:

• ibm_cf_executor(): IBM Cloud Functions executor.
• knative_executor(): Knative executor. See additional information.
• openwhisk_executor(): Vanilla OpenWhisk executor. See additional information.
• function_executor(): Generic executor based on the compute backend specified in configuration.
• local_executor(): Localhost executor to run functions by using local processes.

The available calls within an executor are:

API Call	Type	Description
call_async()	Async.	Method used to spawn one function activation
map()	Async.	Method used to spawn multiple function activations
map_reduce()	Async.	Method used to spawn multiple function activations with one (or multiple) reducers
wait()	Sync.	Wait for the function activations to complete. It blocks the local execution until all the function activations finished their execution (configurable)
get_result()	Sync.	Method used to retrieve the results of all function activations. The results are returned within an ordered list, where each element of the list is the result of one activation
plot()	Sync.	Method used to create execution plots
clean()	Async.	Method used to clean the temporary data generated by PyWren

Additional information and examples:

• Functions: PyWren functions and parameters
• Runtimes: Building and managing PyWren runtimes to run the functions
• Data processing: Using PyWren to process data from IBM Cloud Object Storage and public URLs
• Notebooks: PyWren on IBM Watson Studio and Jupyter notebooks

Verify - Unit Testing

To test that all is working, use the command:

$ pywren-ibm-cloud verify

or

$ python -m pywren_ibm_cloud.tests

Notice that if you didn't set a local PyWren's config file, you need to provide it as a json file path by -c <CONFIG> flag.

Alternatively, for debugging purposes, you can run specific tests by -t <TESTNAME>. use --help flag to get more information about the test script.

Additional resources

• Your easy move to serverless computing and radically simplified data processing Strata Data Conference, NY 2019
  • See video of PyWren-IBM usage here and the example of Monte Carlo here
• Ants, serverless computing, and simplified data processing
• Speed up data pre-processing with PyWren in deep learning
• Predicting the future with Monte Carlo simulations over IBM Cloud Functions
• Process large data sets at massive scale with PyWren over IBM Cloud Functions
• PyWren for IBM Cloud on CODAIT
• Industrial project in Technion on PyWren-IBM
• Serverless data analytics in the IBM Cloud - Proceedings of the 19th International Middleware Conference (Industry)