zoo.orca.learn.tf package

Submodules

zoo.orca.learn.tf.estimator module

class zoo.orca.learn.tf.estimator.Estimator

Bases: object

evaluate(data, **kwargs)

fit(data, epochs, **kwargs)

static from_graph(*, inputs, outputs=None, labels=None, loss=None, optimizer=None, clip_norm=None, clip_value=None, metrics=None, updates=None, sess=None, model_dir=None, backend='bigdl')

Create an Estimator for tesorflow graph. :param inputs: input tensorflow tensors. :param outputs: output tensorflow tensors. :param labels: label tensorflow tensors. :param loss: The loss tensor of the TensorFlow model, should be a scalar :param optimizer: tensorflow optimization method. :param clip_norm: float >= 0. Gradients will be clipped when their L2 norm exceeds this value. :param clip_value: a float >= 0 or a tuple of two floats. If clip_value is a float, gradients will be clipped when their absolute value exceeds this value. If clip_value is a tuple of two floats, gradients will be clipped when their value less than clip_value[0] or larger than clip_value[1]. :param metrics: metric tensor. :param sess: the current TensorFlow Session, if you want to used a pre-trained model, you should use the Session to load the pre-trained variables and pass it to estimator :param model_dir: location to save model checkpoint and summaries. :param backend: backend for estimator. Now it only can be “bigdl”. :return: an Estimator object.

static from_keras(keras_model, metrics=None, model_dir=None, backend='bigdl')

Create an Estimator from a tensorflow.keras model. The model must be compiled. :param keras_model: the tensorflow.keras model, which must be compiled. :param metrics: user specified metric. :param model_dir: location to save model checkpoint and summaries. :param backend: backend for estimator. Now it only can be “bigdl”. :return: an Estimator object.

get_train_summary(tag=None)

Get the scalar from model train summary Return list of summary data of [iteration_number, scalar_value, timestamp] # Arguments tag: The string variable represents the scalar wanted

get_validation_summary(tag=None)

Get the scalar from model validation summary Return list of summary data of [iteration_number, scalar_value, timestamp]

Note: The metric and tag may not be consistent Please look up following form to pass tag parameter Left side is your metric during compile Right side is the tag you should pass ‘Accuracy’ | ‘Top1Accuracy’ ‘BinaryAccuracy’ | ‘Top1Accuracy’ ‘CategoricalAccuracy’ | ‘Top1Accuracy’ ‘SparseCategoricalAccuracy’ | ‘Top1Accuracy’ ‘AUC’ | ‘AucScore’ ‘HitRatio’ | ‘HitRate@k’ (k is Top-k) ‘Loss’ | ‘Loss’ ‘MAE’ | ‘MAE’ ‘NDCG’ | ‘NDCG’ ‘TFValidationMethod’ | ‘${name + ” ” + valMethod.toString()}’ ‘Top5Accuracy’ | ‘Top5Accuracy’ ‘TreeNNAccuracy’ | ‘TreeNNAccuracy()’ ‘MeanAveragePrecision’ | ‘MAP@k’ (k is Top-k) (BigDL) ‘MeanAveragePrecision’ | ‘PascalMeanAveragePrecision’ (Zoo) ‘StatelessMetric’ | ‘${name}’ # Arguments tag: The string variable represents the scalar wanted

load_latest_orca_checkpoint(path)

Load latest Orca checkpoint under specified directory. :param path: directory containing Orca checkpoint files.

load_orca_checkpoint(path, version)

Load specified Orca checkpoint. :param path: checkpoint directory which contains model.* and optimMethod-TFParkTraining.* files. :param version: checkpoint version, which is the suffix of model.* file, i.e., for modle.4 file, the version is 4.

predict(data, **kwargs)

save_keras_model(path, overwrite=True)

Save tensorflow keras model in this estimator. :param path: keras model save path. :param overwrite: Whether to silently overwrite any existing file at the target location.

save_tf_checkpoint(path)

Save tensorflow checkpoint in this estimator. :param path: tensorflow checkpoint path.

set_tensorboard(log_dir, app_name)

Set summary information during the training process for visualization purposes. Saved summary can be viewed via TensorBoard. In order to take effect, it needs to be called before fit.

Training summary will be saved to ‘log_dir/app_name/train’ and validation summary (if any) will be saved to ‘log_dir/app_name/validation’.

# Arguments :param log_dir: The base directory path to store training and validation logs. :param app_name: The name of the application.

class zoo.orca.learn.tf.estimator.TFKerasWrapper(keras_model, metrics, model_dir)

Bases: zoo.orca.learn.tf.estimator.Estimator

evaluate(data, batch_size=4, feature_cols=None, labels_cols=None, hard_code_batch_size=False)

Evaluate model. :param data: evaluation data. It can be XShards, Spark DataFrame, tf.data.Dataset. If data is XShards, each element needs to be {‘x’: a feature numpy array or a tuple of feature numpy arrays, ‘y’: a label numpy array or a tuple of label numpy arrays} If data is tf.data.Dataset, each element is [feature tensor tuple, label tensor tuple] :param batch_size: batch size per thread. :param feature_cols: feature_cols: feature column names if train data is Spark DataFrame. :param labels_cols: label column names if train data is Spark DataFrame. :param hard_code_batch_size: whether to hard code batch size for evaluation. :return: evaluation result as a dictionary of {‘metric name’: metric value}

fit(data, epochs=1, batch_size=32, feature_cols=None, labels_cols=None, validation_data=None, hard_code_batch_size=False, session_config=None, checkpoint_trigger=None)

Train this keras model with train data. :param data: train data. It can be XShards, Spark DataFrame, tf.data.Dataset. If data is XShards, each element needs to be {‘x’: a feature numpy array or a tuple of feature numpy arrays, ‘y’: a label numpy array or a tuple of label numpy arrays} If data is tf.data.Dataset, each element is [feature tensor tuple, label tensor tuple] :param epochs: number of epochs to train. :param batch_size: total batch size for each iteration. :param feature_cols: feature column names if train data is Spark DataFrame. :param labels_cols: label column names if train data is Spark DataFrame. :param validation_data: validation data. Validation data type should be the same as train data. :param hard_code_batch_size: whether hard code batch size for training. Default is False. :param session_config: tensorflow session configuration for training. Should be object of tf.ConfigProto :param checkpoint_trigger: when to trigger checkpoint during training. Should be bigdl optimzer trigger, like EveryEpoch(), SeveralIteration(num_iterations),etc.

predict(data, batch_size=4, feature_cols=None, hard_code_batch_size=False)

Predict input data :param data: data to be predicted. It can be XShards, Spark DataFrame, or tf.data.Dataset. If data is XShard, each element needs to be {‘x’: a feature numpy array or a tuple of feature numpy arrays}. If data is tf.data.Dataset, each element is feature tensor tuple :param batch_size: batch size per thread :param feature_cols: list of feature column names if input data is Spark DataFrame. :param hard_code_batch_size: if require hard code batch size for prediction. The default value is False. :return: predicted result. If input data is XShards or tf.data.Dataset, the predict result is also a XShards, and the schema for each result is: {‘prediction’: predicted numpy array or list of predicted numpy arrays}. If input data is Spark DataFrame, the predict result is a DataFrame which includes original columns plus ‘prediction’ column. The ‘prediction’ column can be FloatType, VectorUDT or Array of VectorUDT depending on model outputs shape.

save_keras_model(path, overwrite=True)

Save tensorflow keras model in this estimator. :param path: keras model save path. :param overwrite: Whether to silently overwrite any existing file at the target location.

class zoo.orca.learn.tf.estimator.TFOptimizerWrapper(*, inputs, outputs, labels, loss, optimizer, clip_norm, clip_value, metrics, updates, sess, model_dir)

Bases: zoo.orca.learn.tf.estimator.Estimator

evaluate(data, batch_size=32, feature_cols=None, labels_cols=None, hard_code_batch_size=False)

Evaluate model. :param data: evaluation data. It can be XShards, Spark DataFrame, tf.data.Dataset. If data is XShards, each element needs to be {‘x’: a feature numpy array or a tuple of feature numpy arrays, ‘y’: a label numpy array or a tuple of label numpy arrays} If data is tf.data.Dataset, each element is a tuple of input tensors. :param batch_size: batch size per thread. :param feature_cols: feature_cols: feature column names if train data is Spark DataFrame. :param labels_cols: label column names if train data is Spark DataFrame. :param hard_code_batch_size: whether to hard code batch size for evaluation. :return: evaluation result as a dictionary of {‘metric name’: metric value}

fit(data, epochs=1, batch_size=32, feature_cols=None, labels_cols=None, validation_data=None, hard_code_batch_size=False, session_config=None, feed_dict=None, checkpoint_trigger=None)

Train this graph model with train data. :param data: train data. It can be XShards, Spark DataFrame, tf.data.Dataset. If data is XShards, each element needs to be {‘x’: a feature numpy array or a tuple of feature numpy arrays, ‘y’: a label numpy array or a tuple of label numpy arrays} If data is tf.data.Dataset, each element is a tuple of input tensors. :param epochs: number of epochs to train. :param batch_size: total batch size for each iteration. :param feature_cols: feature column names if train data is Spark DataFrame. :param labels_cols: label column names if train data is Spark DataFrame. :param validation_data: validation data. Validation data type should be the same as train data. :param hard_code_batch_size: whether hard code batch size for training. Default is False. :param session_config: tensorflow session configuration for training. Should be object of tf.ConfigProto :param feed_dict: a dictionary. The key is TensorFlow tensor, usually a placeholder, the value of the dictionary is a tuple of two elements. The first one of the tuple is the value to feed to the tensor in training phase and the second one is the value to feed to the tensor in validation phase. :param checkpoint_trigger: when to trigger checkpoint during training. Should be bigdl optimzer trigger, like EveryEpoch(), SeveralIteration(num_iterations),etc.

predict(data, batch_size=4, feature_cols=None, hard_code_batch_size=False)

Predict input data :param data: data to be predicted. It can be XShards, Spark DataFrame, or tf.data.Dataset. If data is XShards, each element needs to be {‘x’: a feature numpy array or a tuple of feature numpy arrays}. If data is tf.data.Dataset, each element is a tuple of input tensors. :param batch_size: batch size per thread :param feature_cols: list of feature column names if input data is Spark DataFrame. :param hard_code_batch_size: whether to hard code batch size for prediction. The default value is False. :return: predicted result. If input data is XShards or tf.data.Dataset, the predict result is a XShards, and the schema for each result is: {‘prediction’: predicted numpy array or list of predicted numpy arrays}. If input data is Spark DataFrame, the predict result is a DataFrame which includes original columns plus ‘prediction’ column. The ‘prediction’ column can be FloatType, VectorUDT or Array of VectorUDT depending on model outputs shape.

save_tf_checkpoint(path)

Save tensorflow checkpoint in this estimator. :param path: tensorflow checkpoint path.

zoo.orca.learn.tf.utils module

zoo.orca.learn.tf.utils.convert_predict_to_dataframe(df, prediction_rdd)

zoo.orca.learn.tf.utils.convert_predict_to_xshard(prediction_rdd)

zoo.orca.learn.tf.utils.find_latest_checkpoint(model_dir)

zoo.orca.learn.tf.utils.get_checkpoint_state(checkpoint_dir)

Get tf checkpoint state from checkpoint directory without using native tensorflow accessing remote method. :param checkpoint_dir: tensorflow checkpoint directory. Could be local, hdfs, s3 filesystems. :return: tf checkpoint protobuf

zoo.orca.learn.tf.utils.load_tf_checkpoint(sess, checkpoint_path, saver=None)

Load tensorflow checkpoint from checkpoint path without using native tensorflow accessing remote method. :param sess: tensorflow session to be loaded to. :param checkpoint_path: tensorflow checkpoint path. Could be local, hdfs, s3 filesystems. :param saver: tensorflow saver to load checkpoint

zoo.orca.learn.tf.utils.save_tf_checkpoint(sess, checkpoint_path, saver=None)

Save tf checkpoint without using native tensorflow remote access method. :param sess: tf session to be saved. :param checkpoint_path: checkpoint path. Could be local, hdfs, s3 filesystems. :param saver: tf saver to save checkpoint

zoo.orca.learn.tf.utils.to_dataset(data, batch_size, batch_per_thread, validation_data, feature_cols, labels_cols, hard_code_batch_size, sequential_order, shuffle)

zoo.orca.learn.tf.utils.xshards_to_tf_dataset(data_shard, batch_size=-1, batch_per_thread=-1, validation_data_shard=None, hard_code_batch_size=False, sequential_order=False, shuffle=True)

Module contents