Tensorflow 2.0 -- Everything you need to know

Tensorflow 2.0 -- Everything you need to know

The second version of Tensorflow is out now. Tensorflow is a general-purpose high-performance computing library open-sourced by Google in 2015. Its primary goal is to allow people to easily build high-performing Neural Networks and other machine learning models.

Over the years it has grown to be the most popular deep learning library. This is because of its extensive ecosystem, including easy deployment on many platforms with the help of Tensorflows SavedModel file format as well lot's of tutorials and resources available online.

Figure 1: Google Trends for Tensorflow, PyTorch and Keras (all are great)

Introduction to Tensorflow 2.0

With Tensorflow 2.0 the Tensorflow team made a huge step in making Tensorflow more accessible for people. Version 2.0 includes lots of new features including subclassing, a new way of creating model, as well as lots of quality of live updates.

Figure 2: Announcing Tensorflow 2.0 Video

Eager execution by default

Probably the most wanted update is eager execution as the main way for working with Tensorflow. For those of you who don't know with Tensorflow v1.x the abstract data structures need to be defined in something called a Graph. To then actually, execute the code a session must be used.

This system had a steep learning curve because it's so different from the normal way of Python programming. It's also really hard to debug such programs because when printing a graph node we aren't getting a specific value, rather we would only see a reference.

With version 2.0 Tensorflow moved away from the graph/session architecture and embraced eager execution.

Eager execution is an imperative programming environment that evaluates operations immediately, without building graphs. For more information check out the official eager execution guide.

Keras as the main API for model building

Tensorflow 1.x had multiple APIs for building networks. tf.slim, tf.layers, tf.contrib.layers, tf.keras could all be used to create Neural Networks. Most of the time it wasn't clear to beginners which API to use. Furthermore, if you trained a model with one API, it wasn't straight forward to use the code with another API.

In Tensorflow 2.0 Keras is the recommended high-level API. Keras is a high-level neural networks API, capable of running on top of Tensorflow, Theano, and CNTK. It enables fast experimentation through a high level, user-friendly, modular and extensible API.

Model Subclassing – The pythonic way of creating models

Tensorflow 2.0 also adds support for subclassing. Model subclassing is a user-friendly modular way of creating models or layers.

If you are already familiar with PyTorch you might already have heard of subclassing. In PyTorch, you need to overwrite the __init__ and forward methods. In Tensorflow 2.0 the prediction method isn't called forward, rather it's called call.

To create a neural network with a convolutional and two fully connected layers the below code can be used:

class MyModel(Model):
  def __init__(self):
    super(MyModel, self).__init__()
    self.conv1 = Conv2D(32, 3, activation='relu')
    self.flatten = Flatten()
    self.d1 = Dense(128, activation='relu')
    self.d2 = Dense(10, activation='softmax')

  def call(self, x):
    x = self.conv1(x)
    x = self.flatten(x)
    x = self.d1(x)
    return self.d2(x)

# Create an instance of the model
model = MyModel()

Creating your own layers

Tensorflow 2.0 also supports writing your custom layers. A layer encapsulates both a state (the layer's "weights") and a transformation from inputs to outputs (a "call", the layer's forward pass).

Writing your own layer is quite similar to the way you create a model with subclassing only that for a layer you won't inherit from Model. Rather you will inherit from layers.Layer.

For example, a linear layer can be created with the following code:

from tensorflow.keras import layers


class Linear(layers.Layer):

  def __init__(self, units=32, input_dim=32):
    super(Linear, self).__init__()
    w_init = tf.random_normal_initializer()
    self.w = tf.Variable(initial_value=w_init(shape=(input_dim, units),
                                              dtype='float32'),
                         trainable=True)
    b_init = tf.zeros_initializer()
    self.b = tf.Variable(initial_value=b_init(shape=(units,),
                                              dtype='float32'),
                         trainable=True)

  def call(self, inputs):
    return tf.matmul(inputs, self.w) + self.b

After creating the layer it can be used as a normal python method:

x = tf.ones((2, 2))
linear_layer = Linear(4, 2)
y = linear_layer(x)
print(y)
Output:
tf.Tensor(
[[ 0.04828779  0.07988431 -0.07137472  0.10912687]
 [ 0.04828779  0.07988431 -0.07137472  0.10912687]], shape=(2, 4), dtype=float32)

Train your model

In Tensorflow 2.0 we have two main ways of training our models. The fit method and the tf.GradientTape API.

When using Keras Sequential or Model API the fit method can be used without any modifications. For training a Subclassing model with the fit method the comput_output_shape method must be overridden.

If you want easier access to the gradients and loss as well as a clearer understanding of the training it might be worth to use tf.GradientTape instead of the fit method.

Using GradientTape, one can manually define each training step in the training procedure. The basic steps include:

  • Forward pass
  • Calculating loss
  • Backward pass
  • Updating the gradients

For image classification GradientTape could be used like in the following example:

@tf.function
def train_step(images, labels):
  with tf.GradientTape() as tape:
    predictions = model(images)
    loss = loss_object(labels, predictions)
  gradients = tape.gradient(loss, model.trainable_variables)
  optimizer.apply_gradients(zip(gradients, model.trainable_variables))

  train_loss(loss)
  train_accuracy(labels, predictions)

Converting from Tensorflow v1.x to v2.0

Upgrading all your code from Tensorflow v1.x to 2.0 manually would be tedious and error-prone. To streamline the changes, and to make the transition to TF 2.0 as seamless as possible, the TensorFlow team has created the tf_upgrade_v2 utility to help transition legacy code to the new API.

Typically the utility can be used like:

tf_upgrade_v2 \
  --intree my_project/ \
  --outtree my_project_v2/ \
  --reportfile report.txt

In most cases, the tf_upgrade_v2 utility uses the tf.compat.v1 module and doesn't make any further changes.

Tensorflow v1.x:

from __future__ import print_function

import tensorflow as tf

# Hello World
hello = tf.constant('Hello, TensorFlow!')

# Start tf session
sess = tf.Session()

# Run the op
print(sess.run(hello))
tf_upgrade_v2 --infile tf1_hello_world.py --outfile tf2_hello_world.py

Tensorflow 2.0:

from __future__ import print_function

import tensorflow as tf

# Hello World
hello = tf.constant('Hello, TensorFlow!')

# Start tf session
sess = tf.compat.v1.Session()

# Run the op
print(sess.run(hello))

For more information about how to update your code-bases check out the official migration guide.

Other changes

Besides the changes above Tensorflow 2.0 also significantly cleaned their API for easier usage and added lots of performance updates including multiple GPU support and their Distribution Strategy API, allowing us to distribute training with minimal code changes.

Conclusion

In this article, I went over the most important changes and additions in Tensorflows newest version v2. For more information about everything that changed check out the official Tensorflow page.

That’s all from this article. If you have any questions or just want to chat with me feel free to leave a comment below or contact me on social media. If you want to get continuous updates about my blog make sure to follow me on Medium and join my newsletter.