zoo.pipeline.api.net package

Submodules

zoo.pipeline.api.net.graph_net module

class zoo.pipeline.api.net.graph_net.GraphNet(input, output, jvalue=None, bigdl_type='float', **kwargs)

Bases: bigdl.nn.layer.Model

flattened_layers(include_container=False)

freeze_up_to(names)

Freeze the model from the bottom up to the layers specified by names (inclusive). This is useful for finetuning a model

Parameters:names – A list of module names to be Freezed Returns:current graph model

static from_jvalue(jvalue, bigdl_type='float')

Create a Python Model base on the given java value

Parameters:jvalue – Java object create by Py4j Returns:A Python Model

layers

new_graph(outputs)

Specify a list of nodes as output and return a new graph using the existing nodes

Parameters:outputs – A list of nodes specified Returns:A graph model

predict(x, batch_per_thread=4, distributed=True)

Use a model to do prediction.

# Arguments x: Prediction data. A Numpy array or RDD of Sample or ImageSet. batch_per_thread: The default value is 4. When distributed is True,the total batch size is batch_per_thread * rdd.getNumPartitions. When distributed is False the total batch size is batch_per_thread * numOfCores. distributed: Boolean. Whether to do prediction in distributed mode or local mode. Default is True. In local mode, x must be a Numpy array.

to_keras()

unfreeze(names=None)

“unfreeze” module, i.e. make the module parameters(weight/bias, if exists) to be trained(updated) in training process. If ‘names’ is a non-empty list, unfreeze layers that match given names

Parameters:names – list of module names to be unFreezed. Default is None. Returns:current graph model

zoo.pipeline.api.net.net_load module

class zoo.pipeline.api.net.net_load.JavaToPython(jvalue, bigdl_type='float')

Bases: object

get_python_class()

Redirect the jvalue to the proper python class. :param jvalue: Java object create by Py4j :return: A proper Python wrapper which would be a Model, Sequential…

class zoo.pipeline.api.net.net_load.Net

Bases: object

static from_jvalue(jvalue, bigdl_type='float')

static load(model_path, weight_path=None, bigdl_type='float')

Load an existing Analytics Zoo model defined in Keras-style(with weights).

Parameters:model_path – The path to load the saved model.

Local file system, HDFS and Amazon S3 are supported. HDFS path should be like ‘hdfs://[host]:[port]/xxx’. Amazon S3 path should be like ‘s3a://bucket/xxx’. :param weight_path: The path for pre-trained weights if any. Default is None. :return: An Analytics Zoo model.

static load_bigdl(model_path, weight_path=None, bigdl_type='float')

Load a pre-trained BigDL model.

Parameters:

• model_path – The path to the pre-trained model.
• weight_path – The path to the weights of the pre-trained model. Default is None.

Returns:

A pre-trained model.

static load_caffe(def_path, model_path, bigdl_type='float')

Load a pre-trained Caffe model.

Parameters:

• def_path – The path containing the caffe model definition.
• model_path – The path containing the pre-trained caffe model.

Returns:

A pre-trained model.

static load_keras(json_path=None, hdf5_path=None, by_name=False)

Load a pre-trained Keras model.

Parameters:

• json_path – The json path containing the keras model definition. Default is None.
• hdf5_path – The HDF5 path containing the pre-trained keras model weights

with or without the model architecture. Default is None. :param by_name: by default the architecture should be unchanged. If set as True, only layers with the same name will be loaded. :return: A BigDL model.

static load_torch(path, bigdl_type='float')

Load a pre-trained Torch model.

Parameters:path – The path containing the pre-trained model. Returns:A pre-trained model.

zoo.pipeline.api.net.torch_criterion module

class zoo.pipeline.api.net.torch_criterion.LossWrapperModule(lossFunc)

Bases: torch.nn.modules.module.Module

forward(x, y)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class zoo.pipeline.api.net.torch_criterion.TorchCriterion(path, bigdl_type='float')

Bases: bigdl.nn.criterion.Criterion

TorchCriterion wraps a loss function for distributed inference or training. Use TorchCriterion.from_pytorch to initialize.

static from_pytorch(loss, input, label=None)

Create a TorchCriterion directly from PyTorch function. We need users to provide example input and label (or just their sizes) to trace the loss function.

Parameters:loss – this can be a torch loss (e.g. nn.MSELoss()) or

a function that takes two Tensor parameters: input and label. E.g. def lossFunc(input, target): return nn.CrossEntropyLoss().forward(input, target.flatten().long()) :param input: example input. It can be: 1. a torch tensor, or tuple of torch tensors for multi-input models 2. list of integers, or tuple of int list for multi-input models. E.g. For ResNet, this can be [1, 3, 224, 224]. A random tensor with the specified size will be used as the example input. :param label: example label. It can be: 1. a torch tensor, or tuple of torch tensors for multi-input models 2. list of integers, or tuple of int list for multi-input models. E.g. For ResNet, this can be [1, 3, 224, 224]. A random tensor with the specified size will be used as the example input. When label is None, input will also be used as label.

zoo.pipeline.api.net.torch_net module

class zoo.pipeline.api.net.torch_net.TorchNet(path, bigdl_type='float')

Bases: bigdl.nn.layer.Layer

TorchNet wraps a TorchScript model as a single layer, thus the Pytorch model can be used for distributed inference or training. :param path: path to the TorchScript model.

static from_pytorch(module, input, check_trace=True)

Create a TorchNet directly from PyTorch model, e.g. model in torchvision.models. Users need to provide an example input or the input tensor shape. :param module: a PyTorch model :param input: To trace the tensor operations, torch jit trace requires users to provide an example input. Here the input parameter can be: 1. a torch tensor, or tuple of torch tensors for multi-input models 2. list of integers, or tuple of int list for multi-input models. E.g. For ResNet, this can be [1, 3, 224, 224]. A random tensor with the specified size will be used as the example input. :param check_trace: boolean value, optional. check if the same inputs run through traced module produce the same outputs. Default: True. You might want to disable this if, for example, your network contains non-deterministic ops or if you are sure that the network is correct despite a checker failure.

static get_sample_input(input)

predict(x, batch_per_thread=1, distributed=True)

Use a model to do prediction.

savePytorch(path)

save the model as a torch script module

zoo.pipeline.api.net.utils module

zoo.pipeline.api.net.utils.find_placeholders(grads)

zoo.pipeline.api.net.utils.find_tensors(sources, predicate)

find all the tensors that are used for computing grads and has been computed during forward :param grads: :param forward_ops: :return:

zoo.pipeline.api.net.utils.to_bigdl_optim_method(koptim_method)

Module contents