Runnables¶
Runnables are top level objects that flambé is able to run.
Experiment
is an example of a Runnable
.
Implementing Runnables is as easy as implementing the Runnable interface.
Esentially, it requires a single method run()
.
Hint
A Runnable
object can be executed by flambé:
flambe runnable.yaml
For example, let’s imagine we want to implement an S3Pusher that takes
an Experiment
output folder and uploads the content to a specific S3 bucket:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | from flambe.runnable import Runnable
class S3Pusher(Runnable):
def __init__(self, experiment_path: str, bucket_name: str) -> None:
super().__init__()
self.experiment_path = experiment_path
self.bucket_name = bucket_name
def run(self, **kwargs) -> None:
"""Upload a local folder to a S3 bucket"""
# Code to upload to S3 bucket
S3_client = boto3.client("s3")
for root, dirs, files in os.walk(self.experiment_path):
for f in files:
s3C.upload_file(os.path.join(root, f), self.bucketname, f)
|
This class definition can now be included in an extension (read more about extensions in Extensions) and used as a top level object in a YAML file.
ext: /path/to/S3Pusher/extension:
---
!ext.S3Pusher
experiment_path: /path/to/my/experiment
bucket_name: my-bucket
Then, simply execute:
flambe s3pusher.yaml
Providing secrets¶
All Runnables
have access to secret information that the users can share via
an ini file.
By executing the Runnable
with a --secrets
parameter, then the Runnable
can
access the secrets through the config
attribute:
Let’s say that our S3Pusher
needs to access the AWS_SECRET_TOKEN
. Then:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | from flambe.runnable import Runnable
class S3Pusher(Runnable):
def __init__(self, experiment_path: str, bucket_name: str) -> None:
# Same as before
...
def run(self, **kwargs) -> None:
"""Upload a local folder to a S3 bucket"""
# Code to upload to S3 bucket
S3_client = boto3.client("s3", token=self.config['AWS']['AWS_SECRET_TOKEN'])
for root, dirs, files in os.walk(self.experiment_path):
for file in files:
s3C.upload_file(os.path.join(root, file), self.bucketname, file)
|
Then if secrets.ini
contains:
[AWS]
AWS_SECRET_TOKEN = ABCDEFGHI123456789
We can execute:
flambe s3pusher.yaml --secrets secret.ini
Automatic extensions installation¶
Important
To understand this section you should be familiar with extensions. For information about extensions, go to Extensions.
When executing a Runnable
, it’s possible that extensions are being involved. For example:
ext: /path/to/extension
other_ext: http://github.com/user/some_extension
---
!ext.CustomRunnable
...
param: !other_ext.CustomComponent
Flambé provides a -i / --install-extensions
flag to automatically “pip” installs the extensions:
flambe custom_runnable.yaml -i
By default, this is not activated and the user needs to install the extensions beforehand.
Warning
Installing extensions automatically could possibly update libraries in the your environment because of a version reequirement. Flambé will output all libraries that are being updated.