zoo.tfpark package

Subpackages

• zoo.tfpark.gan package
  • Submodules
  • zoo.tfpark.gan.common module
  • zoo.tfpark.gan.gan_estimator module
  • Module contents
• zoo.tfpark.text package
  • Subpackages
    • zoo.tfpark.text.estimator package
      • Submodules
      • zoo.tfpark.text.estimator.bert_base module
      • zoo.tfpark.text.estimator.bert_classifier module
      • zoo.tfpark.text.estimator.bert_ner module
      • zoo.tfpark.text.estimator.bert_squad module
      • Module contents
    • zoo.tfpark.text.keras package
      • Submodules
      • zoo.tfpark.text.keras.intent_extraction module
      • zoo.tfpark.text.keras.ner module
      • zoo.tfpark.text.keras.pos_tagging module
      • zoo.tfpark.text.keras.text_model module
      • Module contents
  • Module contents

Submodules

zoo.tfpark.estimator module

class zoo.tfpark.estimator.TFEstimator(estimator)

Bases: object

evaluate(input_fn, eval_methods, steps=None, checkpoint_path=None)

Evaluates the model given evaluation data input_fn.

Parameters:input_fn – A function that constructs the input data for evaluation. The

function should construct and return one of the following: * A TFDataset object, each elements of which is a tuple (features, labels). * A tf.data.Dataset object: Outputs of Dataset object must be a tuple (features, labels) with same constraints as below. * A tuple (features, labels): Where features is a tf.Tensor or a dictionary of string feature name to Tensor and labels is a Tensor or a dictionary of string label name to Tensor. Both features and labels are consumed by model_fn. They should satisfy the expectation of model_fn from inputs. :param eval_methods: a list of strings to specify the evaluation metrics to be used in this model :param steps: Number of steps for which to evaluate model. :param checkpoint_path: Path of a specific checkpoint to evaluate. If None, the latest checkpoint in model_dir is used. If there are no checkpoints in model_dir, evaluation is run with newly initialized Variables instead of ones restored from checkpoint.

Returns: A dict containing the evaluation metrics specified in model_fn keyed by name.

classmethod from_model_fn(model_fn, model_dir=None, config=None, params=None, warm_start_from=None)

Parameters:model_fn – Model function. Follows the signature:

• Args:
• features: This is the first item returned from the input_fn

passed to train, evaluate, and predict. This should be a single tf.Tensor or dict of same. * labels: This is the second item returned from the input_fn passed to train, evaluate, and predict. This should be a single tf.Tensor or dict of same (for multi-head models). If mode is tf.estimator.ModeKeys.PREDICT, labels=None will be passed. If the model_fn’s signature does not accept mode, the model_fn must still be able to handle labels=None. * mode: Optional. Specifies if this training, evaluation or prediction. See tf.estimator.ModeKeys. * params: Optional dict of hyperparameters. Will receive what is passed to Estimator in params parameter. This allows to configure Estimators from hyper parameter tuning. * config: Optional estimator.RunConfig object. Will receive what is passed to Estimator as its config parameter, or a default value. Allows setting up things in your model_fn based on configuration such as num_ps_replicas, or model_dir.

• Returns:

tf.estimator.EstimatorSpec

For the train_op in tf.estimator.EstimatorSpec, it derive from and only from zoo.tfpark.zoo_optimizer.ZooOptimizer :param model_dir: Directory to save model parameters, graph and etc. This can also be used to load checkpoints from the directory into an estimator to continue training a previously saved model. If PathLike object, the path will be resolved. If None, the model_dir in config will be used if set. If both are set, they must be same. If both are None, a temporary directory will be used. :param config: estimator.RunConfig configuration object. :param params: dict of hyper parameters that will be passed into model_fn. Keys are names of parameters, values are basic python types. :param warm_start_from: Optional string filepath to a checkpoint or SavedModel to warm-start from, or a tf.estimator.WarmStartSettings object to fully configure warm-starting. If the string filepath is provided instead of a tf.estimator.WarmStartSettings, then all variables are warm-started, and it is assumed that vocabularies and tf.Tensor names are unchanged.

predict(input_fn, predict_keys=None, checkpoint_path=None)

Outputs predictions for given features.

Parameters:input_fn – A function that constructs the features.

• A TFDataset object, each elements of which is a tuple (features, None).
• A tf.data.Dataset object: Outputs of Dataset object must have

same constraints as below. * features: A tf.Tensor or a dictionary of string feature name to Tensor. features are consumed by model_fn. They should satisfy the expectation of model_fn from inputs. * A tuple, in which case the first item is extracted as features.

Parameters:checkpoint_path – Path of a specific checkpoint to predict. If None, the

latest checkpoint in model_dir is used. If there are no checkpoints in model_dir, prediction is run with newly initialized Variables instead of ones restored from checkpoint.

Return: Evaluated values of predictions tensors.

train(input_fn, steps=None, session_config=None)

Trains a model given training data input_fn.

Parameters:input_fn – A function that constructs the input data for evaluation. The

function should construct and return one of the following: * A TFDataset object, each elements of which is a tuple (features, labels). * A tf.data.Dataset object: Outputs of Dataset object must be a tuple (features, labels) with same constraints as below. * A tuple (features, labels): Where features is a tf.Tensor or a dictionary of string feature name to Tensor and labels is a Tensor or a dictionary of string label name to Tensor. Both features and labels are consumed by model_fn. They should satisfy the expectation of model_fn from inputs. :param steps: Number of steps for which to train the model.

Returns: self, for chaining.

zoo.tfpark.model module

class zoo.tfpark.model.KerasModel(model, model_dir=None)

Bases: object

add_metric(tensor, name)

evaluate(x=None, y=None, batch_per_thread=None, distributed=False)

Evaluate a model on a given dataset

Parameters:x – Input data. It could be:

• a TFDataset object
• A Numpy array (or array-like), or a list of arrays

(in case the model has multiple inputs). - A dict mapping input names to the corresponding array/tensors, if the model has named inputs. :param y: Target data. Like the input data x, It should be consistent with x (you cannot have Numpy inputs and tensor targets, or inversely). If x is a TFDataset, y should not be specified (since targets will be obtained from x). :param batch_per_thread: The default value is 1. When distributed is True,the total batch size is batch_per_thread * rdd.getNumPartitions. When distributed is False the total batch size is batch_per_thread * numOfCores. :param distributed: Boolean. Whether to do prediction in distributed mode or local mode. Default is True. In local mode, x must be a Numpy array.

fit(x=None, y=None, batch_size=None, epochs=1, validation_data=None, distributed=False, **kwargs)

Train the model for a fixed num of epochs

Arguments: :param x: Input data. It could be: - a TFDataset object - A Numpy array (or array-like), or a list of arrays (in case the model has multiple inputs). - A dict mapping input names to the corresponding array/tensors, if the model has named inputs. :param y: Target data. Like the input data x, It should be consistent with x (you cannot have Numpy inputs and tensor targets, or inversely). If x is a TFDataset, y should not be specified (since targets will be obtained from x). :param batch_size: Integer or None. Number of samples per gradient update. If x is a TFDataset, you do not need to specify batch_size. :param epochs: Integer. Number of epochs to train the model. An epoch is an iteration over the entire x and y data provided. :param validation_data: Data on which to evaluate the loss and any model metrics at the end of each epoch. The model will not be trained on this data. validation_data could be: - tuple (x_val, y_val) of Numpy arrays or tensors :param distributed: Boolean. Whether to do prediction in distributed mode or local mode. Default is True. In local mode, x must be a Numpy array.

get_weights()

static load_model(path)

Load an existing keras model (with weights) from HDF5 file.

Parameters:path – String. The path to the pre-defined model. Returns:KerasModel.

load_weights(filepath, by_name=False)

metrics_names

predict(x, batch_per_thread=None, distributed=False)

Use a model to do prediction.

Parameters:x – Input data. It could be:

• a TFDataset object
• A Numpy array (or array-like), or a list of arrays

(in case the model has multiple inputs). - A dict mapping input names to the corresponding array/tensors, if the model has named inputs. :param batch_per_thread: The default value is 1. When distributed is True,the total batch size is batch_per_thread * rdd.getNumPartitions. When distributed is False the total batch size is batch_per_thread * numOfCores. :param distributed: Boolean. Whether to do prediction in distributed mode or local mode. Default is True. In local mode, x must be a Numpy array.

predict_on_batch(x)

save_model(path, overwrite=True)

Save the model to a single HDF5 file.

Parameters:

• path – String. The path to save the model.
• overwrite – Boolean. Whether to silently overwrite any existing file at the target

location

save_weights(filepath, overwrite=True, save_format=None)

set_weights(weights)

test_on_batch(x, y=None, sample_weight=None, reset_metrics=True)

train_on_batch(x, y=None, sample_weight=None, class_weight=None, reset_metrics=True)

zoo.tfpark.tf_dataset module

class zoo.tfpark.tf_dataset.DataFrameDataset(df, feature_cols, labels_cols=None, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, validation_df=None, sequential_order=False, shuffle=True)

Bases: zoo.tfpark.tf_dataset.TFNdarrayDataset

static df_datatype_to_tf(dtype)

static is_scalar_type(dtype)

class zoo.tfpark.tf_dataset.MergeFeatureLabelFeatureTransformer(bigdl_type='float')

Bases: bigdl.transform.vision.image.FeatureTransformer

class zoo.tfpark.tf_dataset.MergeFeatureLabelImagePreprocessing(bigdl_type='float')

Bases: zoo.feature.image.imagePreprocessing.ImagePreprocessing

class zoo.tfpark.tf_dataset.TFBytesDataset(string_rdd, batch_size, batch_per_thread, hard_code_batch_size=False, validation_string_rdd=None, sequential_order=False, shuffle=True)

Bases: zoo.tfpark.tf_dataset.TFDataset

get_num_partitions()

Returns:the num of partitions of the underlying RDD

class zoo.tfpark.tf_dataset.TFDataDataset(tf_data_dataset, batch_size, batch_per_thread, hard_code_batch_size=False, validation_dataset=None, sequential_order=False, shuffle=True, remove_checking=False, batch_outside=False, inter_threads=None, intra_threads=None)

Bases: zoo.tfpark.tf_dataset.TFDataset

static check_rules(dataset, rules, is_training)

get_num_partitions()

Returns:the num of partitions of the underlying RDD

class zoo.tfpark.tf_dataset.TFDataset(tensor_structure, batch_size, batch_per_thread, hard_code_batch_size=False)

Bases: object

feature_tensors

static from_bytes_rdd(bytes_rdd, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, validation_bytes_rdd=None)

Create a TFDataset from a RDD of bytes. Each element is the RDD should be a bytes object. The returning TFDataset’s feature_tensors has only one Tensor. the type of the Tensor is tf.string, and the shape is (None,). The returning don’t have label_tensors. If the dataset is used for training, the label should be encoded in the bytes. :param bytes_rdd: the RDD of bytes :param batch_size: the batch size, used for training, should be a multiple of total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param validation_bytes_rdd: the RDD of bytes to be used in validation :return: a TFDataset

static from_dataframe(df, feature_cols, labels_cols=None, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, validation_df=None, sequential_order=False, shuffle=True)

Create a TFDataset from a pyspark.sql.DataFrame.

Parameters:

• df – the DataFrame for the dataset
• feature_cols – a list of string, indicating which columns are used as features.

Currently supported types are FloatType, DoubleType, IntegerType, LongType, ArrayType (value should be numbers), DenseVector and SparseVector. For ArrayType, DenseVector and SparseVector, the sizes are assume to the same. :param labels_cols: a list of string, indicating which columns are used as labels. Currently supported types are FloatType, DoubleType, IntegerType, LongType, ArrayType (value should be numbers), DenseVector and SparseVector. For ArrayType, DenseVector and SparseVector, the sizes are assume to the same. :param batch_size: the batch size, used for training, should be a multiple of total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param validation_df: the DataFrame used for validation :return: a TFDataset

static from_feature_set(dataset, features, labels=None, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, validation_dataset=None)

Create a TFDataset from a FeatureSet. Currently, the element in this Feature set must be a Sample, i.e. the result of ImageFeatureToSample transformer :param dataset: the feature set used to create this TFDataset :param features: a tuple of two, the first element is the type of this input feature, the second element is the shape of this element, i.e. (tf.float32, [224, 224, 3])). text can also be nested structure of this tuple of two. :param labels: a tuple of two, the first element is the type of label, the second element is the shape of this element, i.e. (tf.int32, [1])). label can also be nested structure of this tuple of two. :param batch_size: the batch size, used for training, should be a multiple of total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param validation_dataset: The FeatureSet used for validation during training :return: a TFDataset

static from_image_set(image_set, image, label=None, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, validation_image_set=None, sequential_order=False, shuffle=True)

Create a TFDataset from a ImagetSet. Each ImageFeature in the ImageSet should already has the “sample” field, i.e. the result of ImageSetToSample transformer

Parameters:

• image_set – the ImageSet used to create this TFDataset
• image – a tuple of two, the first element is the type of image, the second element

is the shape of this element, i.e. (tf.float32, [224, 224, 3])) :param label: a tuple of two, the first element is the type of label, the second element is the shape of this element, i.e. (tf.int32, [1])) :param batch_size: the batch size, used for training, should be a multiple of total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param validation_image_set: the ImageSet used for validation during training :param sequential_order: whether to iterate the elements in the Dataset in sequential order when training. :param shuffle: whether to shuffle the elements in each partition before each epoch when training :return: a TFDataset

static from_ndarrays(*args, **kwargs)

Create a TFDataset from a nested structure of numpy ndarrays. Each element in the resulting TFDataset has the same structure of the argument tensors and is created by indexing on the first dimension of each ndarray in the tensors argument.

This method is equivalent to sc.parallize the tensors and call TFDataset.from_rdd

Parameters:

• tensors – the numpy ndarrays
• batch_size – the batch size, used for training, should be a multiple of

total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param val_tensors: the numpy ndarrays used for validation during training :param sequential_order: whether to iterate the elements in the Dataset in sequential order when training. :param shuffle: whether to shuffle the elements in each partition before each epoch when training :return: a TFDataset

static from_rdd(*args, **kwargs)

Create a TFDataset from a rdd.

For training and evaluation, both features and labels arguments should be specified. The element of the rdd should be a tuple of two, (features, labels), each has the same structure of numpy.ndarrays of the argument features, labels.

E.g. if features is [(tf.float32, [10]), (tf.float32, [20])], and labels is {“label1”:(tf.float32, [10]), “label2”: (tf.float32, [20])} then a valid element of the rdd could be

( [np.zeros(dtype=float, shape=(10,), np.zeros(dtype=float, shape=(10,)))], {“label1”: np.zeros(dtype=float, shape=(10,)), “label2”:np.zeros(dtype=float, shape=(10,))))} )

If labels is not specified, then the above element should be changed to [np.zeros(dtype=float, shape=(10,), np.zeros(dtype=float, shape=(10,)))]

For inference, labels can be not specified. The element of the rdd should be some ndarrays of the same structure of the features argument.

A note on the legacy api: if you are using names, shapes, types arguments, each element of the rdd should be a list of numpy.ndarray.

Parameters:rdd – a rdd containing the numpy.ndarrays to be used

for training/evaluation/inference :param features: the structure of input features, should one the following: - a tuple (dtype, shape), e.g. (tf.float32, [28, 28, 1]) - a list of such tuple [(dtype1, shape1), (dtype2, shape2)], e.g. [(tf.float32, [10]), (tf.float32, [20])], - a dict of such tuple, mapping string names to tuple {“name”: (dtype, shape}, e.g. {“input1”:(tf.float32, [10]), “input2”: (tf.float32, [20])}

Parameters:

• labels – the structure of input labels, format is the same as features
• batch_size – the batch size, used for training, should be a multiple of

total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param val_rdd: validation data with the same structure of rdd :param sequential_order: whether to iterate the elements in the Dataset in sequential order when training. :param shuffle: whether to shuffle the elements in each partition before each epoch when training :return: a TFDataset

static from_string_rdd(string_rdd, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, validation_string_rdd=None)

Create a TFDataset from a RDD of strings. Each element is the RDD should be a single string. The returning TFDataset’s feature_tensors has only one Tensor. the type of the Tensor is tf.string, and the shape is (None,). The returning don’t have label_tensors. If the dataset is used for training, the label should be encoded in the string. :param string_rdd: the RDD of strings :param batch_size: the batch size, used for training, should be a multiple of total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param validation_string_rdd: the RDD of strings to be used in validation :return: a TFDataset

static from_text_set(text_set, text, label=None, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, validation_image_set=None, sequential_order=False, shuffle=True)

Create a TFDataset from a TextSet. The TextSet must be transformed to Sample, i.e. the result of TextFeatureToSample transformer. :param text_set: the TextSet used to create this TFDataset :param text: a tuple of two, the first element is the type of this input feature, the second element is the shape of this element, i.e. (tf.float32, [10, 100, 4])). text can also be nested structure of this tuple of two. :param label: a tuple of two, the first element is the type of label, the second element is the shape of this element, i.e. (tf.int32, [1])). label can also be nested structure of this tuple of two. :param batch_size: the batch size, used for training, should be a multiple of total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param validation_image_set: The TextSet used for validation during training :param sequential_order: whether to iterate the elements in the Dataset in sequential order when training. :param shuffle: whether to shuffle the elements in each partition before each epoch when training :return: a TFDataset

static from_tf_data_dataset(dataset, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, validation_dataset=None, sequential_order=False, shuffle=True, remove_checking=False, batch_outside=False, inter_threads=None, intra_threads=None)

Create a TFDataset from a tf.data.Dataset.

The recommended way to create the dataset is to reading files in a shared file system (e.g. HDFS) that is accessible from every executor of this Spark Application.

If the dataset is created by reading files in the local file system, then the files must exist in every executor in the exact same path. The path should be absolute path and relative path is not supported.

A few kinds of dataset is not supported for now: 1. dataset created from tf.data.Dataset.from_generators 2. dataset with Dataset.batch operation. 3. dataset with Dataset.repeat operation 4. dataset contains tf.py_func, tf.py_function or tf.numpy_function

Parameters:

• dataset – the tf.data.Dataset
• batch_size – the batch size, used for training, should be a multiple of

total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param validation_dataset: the dataset used for validation :return: a TFDataset

static from_tfrecord_file(sc, file_path, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, validation_file_path=None, sequential_order=False, shuffle=True)

Create a TFDataset from tfrecord files. :param sc: The SparkContext :param file_path: comma seperated tfrecord file(s) path :param batch_size: the batch size, used for training, should be a multiple of total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param validation_file_path: The tfrecord files used for validation :param sequential_order: whether to iterate the elements in the Dataset in sequential order when training. :param shuffle: whether to shuffle the elements in each partition before each epoch when training :return: a TFDataset

get_evaluation_data()

Returns:an object that can be used for TFNet.evaluate,

e.g. an RDD of Sample or a ImageSet

get_num_partitions()

Returns:the num of partitions of the underlying RDD

get_prediction_data()

Returns:an object that can be used for TFNet.predict

e.g. an RDD of Sample or a ImageSet

get_training_data()

Returns:an object that can be used to create a BigDL optimizer,

e.g. an RDD of Sample or a DataSet

get_validation_data()

Returns:an object that can be used to set validation in a BigDL optimizer,

e.g. an RDD of Sample or a DataSet

label_tensors

tensors

a nested structure of TensorFlow tensor object in TensorFlow graph. The elements of this dataset will be fed into these tensors on each iteration. :return: the nested structure of TensorFlow tensor object

class zoo.tfpark.tf_dataset.TFFeatureDataset(dataset, tensor_structure, batch_size, batch_per_thread, hard_code_batch_size=False, validation_dataset=None)

Bases: zoo.tfpark.tf_dataset.TFDataset

get_num_partitions()

Returns:the num of partitions of the underlying RDD

class zoo.tfpark.tf_dataset.TFImageDataset(image_set, tensor_structure, batch_size, batch_per_thread, hard_code_batch_size=False, validation_image_set=None, sequential_order=False, shuffle=True)

Bases: zoo.tfpark.tf_dataset.TFDataset

get_num_partitions()

Returns:the num of partitions of the underlying RDD

class zoo.tfpark.tf_dataset.TFNdarrayDataset(rdd, tensor_structure, batch_size, batch_per_thread, hard_code_batch_size=False, val_rdd=None, sequential_order=True, shuffle=False)

Bases: zoo.tfpark.tf_dataset.TFDataset

static from_ndarrays(tensors, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, val_tensors=None, sequential_order=False, shuffle=True)

Create a TFDataset from a nested structure of numpy ndarrays. Each element in the resulting TFDataset has the same structure of the argument tensors and is created by indexing on the first dimension of each ndarray in the tensors argument.

This method is equivalent to sc.parallize the tensors and call TFDataset.from_rdd

Parameters:

• tensors – the numpy ndarrays
• batch_size – the batch size, used for training, should be a multiple of

total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param val_tensors: the numpy ndarrays used for validation during training :param sequential_order: whether to iterate the elements in the Dataset in sequential order when training. :param shuffle: whether to shuffle the elements in each partition before each epoch when training :return: a TFDataset

static from_rdd(rdd, names=None, shapes=None, types=None, batch_size=-1, batch_per_thread=-1, hard_code_batch_size=False, val_rdd=None, features=None, labels=None, sequential_order=False, shuffle=True)

Create a TFDataset from a rdd.

For training and evaluation, both features and labels arguments should be specified. The element of the rdd should be a tuple of two, (features, labels), each has the same structure of numpy.ndarrays of the argument features, labels.

E.g. if features is [(tf.float32, [10]), (tf.float32, [20])], and labels is {“label1”:(tf.float32, [10]), “label2”: (tf.float32, [20])} then a valid element of the rdd could be

( [np.zeros(dtype=float, shape=(10,), np.zeros(dtype=float, shape=(10,)))], {“label1”: np.zeros(dtype=float, shape=(10,)), “label2”:np.zeros(dtype=float, shape=(10,))))} )

If labels is not specified, then the above element should be changed to [np.zeros(dtype=float, shape=(10,), np.zeros(dtype=float, shape=(10,)))]

For inference, labels can be not specified. The element of the rdd should be some ndarrays of the same structure of the features argument.

A note on the legacy api: if you are using names, shapes, types arguments, each element of the rdd should be a list of numpy.ndarray.

Parameters:rdd – a rdd containing the numpy.ndarrays to be used

for training/evaluation/inference :param features: the structure of input features, should one the following: - a tuple (dtype, shape), e.g. (tf.float32, [28, 28, 1]) - a list of such tuple [(dtype1, shape1), (dtype2, shape2)], e.g. [(tf.float32, [10]), (tf.float32, [20])], - a dict of such tuple, mapping string names to tuple {“name”: (dtype, shape}, e.g. {“input1”:(tf.float32, [10]), “input2”: (tf.float32, [20])}

Parameters:

• labels – the structure of input labels, format is the same as features
• batch_size – the batch size, used for training, should be a multiple of

total core num :param batch_per_thread: the batch size for each thread, used for inference or evaluation :param hard_code_batch_size: whether to hard code the batch_size into tensorflow graph, if True, the static size of the first dimension of the resulting tensors is batch_size/total_core_num (training) or batch_per_thread for inference; if False, it is None. :param val_rdd: validation data with the same structure of rdd :param sequential_order: whether to iterate the elements in the Dataset in sequential order when training. :param shuffle: whether to shuffle the elements in each partition before each epoch when training :return: a TFDataset

get_num_partitions()

Returns:the num of partitions of the underlying RDD

class zoo.tfpark.tf_dataset.TFParkSampleToMiniBatch(batch_size, drop_remainder, bigdl_type='float')

Bases: zoo.feature.common.Preprocessing

a Transformer that converts Feature to (Feature, None).

class zoo.tfpark.tf_dataset.TFTextDataset(text_set, tensor_structure, batch_size, batch_per_thread, hard_code_batch_size=False, validation_text_set=None, sequential_order=False, shuffle=True)

Bases: zoo.tfpark.tf_dataset.TFDataset

get_num_partitions()

Returns:the num of partitions of the underlying RDD

class zoo.tfpark.tf_dataset.TensorMeta(dtype, name=None, shape=None)

Bases: object

zoo.tfpark.tf_optimizer module

class zoo.tfpark.tf_optimizer.BigDLMetric(val_method, outputs, labels)

Bases: object

class zoo.tfpark.tf_optimizer.IdentityCriterion

Bases: bigdl.nn.criterion.Criterion

class zoo.tfpark.tf_optimizer.StatelessMetric(metric_name, idx, count_idx)

Bases: bigdl.util.common.JavaValue

class zoo.tfpark.tf_optimizer.TFModel(training_helper_layer, criterion, val_methods)

Bases: object

static create(loss_tensor, sess, inputs, labels, predictions, grads, variables, graph, tensors_with_value, session_config, metrics, updates, model_dir, train_op=None)

static export(model_dir, loss_tensor, sess, inputs, labels, predictions, grads, variables, graph, tensors_with_value, metrics, updates, train_op=None)

class zoo.tfpark.tf_optimizer.TFOptimizer(tf_model, optim_method, sess=None, dataset=None, clip_norm=None, clip_value=None, model_dir=None)

Bases: object

static export_training_model(export_dir, loss, sess, inputs, labels=None, predictions=None, metrics=None, tensor_with_value=None, updates=None)

classmethod from_keras(keras_model, dataset, session_config=None, model_dir=None, metrics=None)

Create a TFOptimizer from a tensorflow.keras model. The model must be compiled. :param keras_model: the tensorflow.keras model, which must be compiled. :param dataset: a TFDataset :return:

classmethod from_loss(loss, optim_method, session=None, inputs=None, dataset=None, val_outputs=None, val_labels=None, val_method=None, clip_norm=None, clip_value=None, metrics=None, tensor_with_value=None, session_config=None, model_dir=None, updates=None)

Create a TFOptimizer from a TensorFlow loss tensor. The loss tensor must come from a TensorFlow graph that only takes TFDataset.tensors and the tensors in tensor_with_value as inputs. :param loss: The loss tensor of the TensorFlow model, should be a scalar :param optim_method: the optimization method to be used, such as bigdl.optim.optimizer.Adam :param session: the current TensorFlow Session, if you want to used a pre-trained model, you should use the Session to load the pre-trained variables and pass it to TFOptimizer. :param val_outputs: the validation output TensorFlow tensor to be used by val_methods :param val_labels: the validation label TensorFlow tensor to be used by val_methods :param val_method: the BigDL val_method(s) to be used. :param clip_norm: float >= 0. Gradients will be clipped when their L2 norm exceeds this value. :param clip_value: float >= 0. Gradients will be clipped when their absolute value exceeds this value. :param metrics: a dictionary. The key should be a string representing the metric’s name and the value should be the corresponding TensorFlow tensor, which should be a scalar. :param tensor_with_value: a dictionary. The key is TensorFlow tensor, usually a placeholder, the value of the dictionary is a tuple of two elements. The first one of the tuple is the value to feed to the tensor in training phase and the second one is the value to feed to the tensor in validation phase. :return: a TFOptimizer

classmethod from_train_op(train_op, loss, *, inputs=None, labels=None, metrics=None, updates=None, sess=None, dataset=None, tensor_with_value=None, session_config=None, model_dir=None)

load_checkpoint(path, version)

optimize(end_trigger=None, checkpoint_trigger=None)

Run the training loop of the this optimizer :param end_trigger: BigDL’s Trigger to indicate when to stop the training. :param checkpoint_trigger: When to save a checkpoint and evaluate model.

set_constant_gradient_clipping(min_value, max_value)

Configure constant clipping settings.

Parameters:

• min_value – the minimum value to clip by
• max_value – the maxmimum value to clip by

set_gradient_clipping_by_l2_norm(clip_norm)

Configure L2 norm clipping settings. :param clip_norm: gradient L2-Norm threshold

class zoo.tfpark.tf_optimizer.TFTrainingHelper(path, config_proto, saver, meta, sess)

Bases: bigdl.nn.layer.Layer

get_weights_to_python()

load_checkpoint(path)

save_checkpoint()

class zoo.tfpark.tf_optimizer.TFValidationMethod(val_method, name, output_indices, label_indices)

Bases: bigdl.util.common.JavaValue

zoo.tfpark.tf_predictor module

class zoo.tfpark.tf_predictor.TFPredictor(sess, outputs, inputs=None, dataset=None)

Bases: object

classmethod from_keras(keras_model, dataset)

classmethod from_outputs(sess, outputs)

predict()

zoo.tfpark.tfnet module

class zoo.tfpark.tfnet.TFNet(path, input_names=None, output_names=None, tf_session_config=None, jvalue=None, bigdl_type='float')

Bases: bigdl.nn.layer.Layer

static check_input(input)

Parameters:input – ndarray or list of ndarray or JTensor or list of JTensor. Returns:(list of JTensor, isTable)

evaluate(dataset, batch_size, val_methods)

No argument passed in: Evaluate the model to set train = false, useful when doing test/forward :return: layer itself

Three arguments passed in: A method to benchmark the model quality.

Parameters:

• dataset – the input data
• batch_size – batch size
• val_methods – a list of validation methods. i.e: Top1Accuracy,Top5Accuracy and Loss.

Returns:

a list of the metrics result

static from_export_folder(folder, tf_session_config=None)

Create a TFNet from an exported folder produced by export_tf :param folder: the folder the TensorFlow model exported to :param tf_session_config: an optional tf.ConfigProto object to set the session config in java side. This config does not necessarily be the same with your current session. E.g. sess_config = tf.ConfigProto(inter_op_parallelism_threads=1, intra_op_parallelism_threads=1) net = TFNet.from_session(sess, inputs, outputs, sess_config) :return: a TFNet

static from_saved_model(model_path, tag=None, signature=None, inputs=None, outputs=None, tf_session_config=None, init_op=None)

Create a TFNet from an TensorFlow saved model :param model_path: the path to the SavedModel path :param tag: the tag to load in the saved model, default to “serve” :param signature: The signature of the SignatureDef that defines inputs and outputs of the graph. TFNet assumes inputs is sorted by their corresponding key in SignatureDef. :param inputs: a list input tensor names of this model, you may want to use TensorFlow’s command line tool to inspect the saved model to find the input tensor names e.g. saved_model_cli show –dir {saved_model_path} –all :param outputs: a list output tensor names of this model, you may want to use TensorFlow’s command line tool to inspect the saved model to find the output tensor names e.g. saved_model_cli show –dir {saved_model_path} –all :param tf_session_config: an optional tf.ConfigProto object to set the session config in java side. This config does not necessarily be the same with your current session. E.g. sess_config = tf.ConfigProto(inter_op_parallelism_threads=1, intra_op_parallelism_threads=1) net = TFNet.from_session(sess, inputs, outputs, sess_config) :return: a TFNet

static from_session(sess, inputs, outputs, generate_backward=False, allow_non_differentiable_input=True, tf_session_config=None)

Create a TFNet from an a session and the inputs and outpus endpoints of the TensorFlow graph. :param sess: the TensorFlow session contain all the variables :param inputs: a list of TensorFlow Tensor represents the input endpoints of the TensorFlow graph :param outputs: a list of TensorFlow Tensor represents the output endpoints of the TensorFlow graph :param tf_session_config: an optional tf.ConfigProto object to set the session config in java side. This config does not necessarily be the same with your current session. E.g. sess_config = tf.ConfigProto(inter_op_parallelism_threads=1, intra_op_parallelism_threads=1) net = TFNet.from_session(sess, inputs, outputs, sess_config) :return a TFNet

predict(x, batch_per_thread=1, distributed=True, mini_batch=False)

Use a model to do prediction.

zoo.tfpark.utils module

zoo.tfpark.utils.evaluate_metrics(inputs, sess, dataset, metrics)

zoo.tfpark.utils.evaluate_string_metrics(*, sess, string_metrics, dataset, inputs, targets=None, outputs=None, loss=None)

zoo.tfpark.utils.to_bigdl_metric(metric)

zoo.tfpark.zoo_optimizer module

class zoo.tfpark.zoo_optimizer.FakeOptimMethod

Bases: bigdl.optim.optimizer.OptimMethod

class zoo.tfpark.zoo_optimizer.ZooOptimizer(optimizer, name=None)

Bases: tensorflow.python.training.optimizer.Optimizer

An optimizer that wraps another tf.Optimizer, using an allreduce to combine gradient values before applying gradients to model weights.

apply_gradients(*args, **kwargs)

Calls this same method on the underlying optimizer.

compute_gradients(*args, **kwargs)

Compute gradients of all trainable variables. See Optimizer.compute_gradients() for more info. In DistributedOptimizer, compute_gradients() is overriden to also allreduce the gradients before returning them.

get_slot(*args, **kwargs)

Calls this same method on the underlying optimizer.

get_slot_names(*args, **kwargs)

Calls this same method on the underlying optimizer.

variables(*args, **kwargs)

Calls this same method on the underlying optimizer.

zoo.tfpark.zoo_optimizer.get_gradients_for_keras(optimizer, loss, params)

Module contents

zoo.tfpark.check_tf_version()