Pipeline API

pydantic model code_genie.pipeline.PipelineStep[source]
Fields:
field additional_inputs: Optional[Dict[str, Union[Source, GenieResult, Argument]]] = None

Set this value for each additional input to the genie. The dictionary key should be the name of the input and the value could be one of the 3 things: 1. A genie data source 2. A genie result from a previous step 3. A constant value to be passed as an argument to the pipeline

field data: Optional[Union[Source, GenieResult]] = None

Data to be passed to the genie for computation. This could be either a data source or a previous genie result

field genie_result: GenieResult [Required]

Result of the genie which should be run in this step

field sink: Optional[Sink] = None

If the output of this step needs to be exported, then a sink can be provided here

pydantic model code_genie.pipeline.GeniePipeline[source]
Fields:
Validators:

  • _validate_steps » steps

field cache_dir: str [Required]

Directory where the genies being used in this pipeline are cached. The pipeline will also be cached at the same location

field name: str [Required]

Name of the pipeline

field steps: List[PipelineStep] [Required]

List of steps in the pipeline

Validated by:

  • _validate_steps

field version: str [Required]

Version of the pipeline

add(step: PipelineStep)[source]

Add a step to the pipeline

export(filename: str)[source]

Export the pipeline as a json file

classmethod load(filepath: str)[source]

Load the pipeline from a json file

run(args: Dict[str, Any])[source]

Run the pipeline using the value of the arguments passed. Note that all arguments which do not have a default value needs to be passed here for the pipeline to run.