Stefano Cappellini

AI, Deep Learning, Machine Learning, Software Engineering

What are the differences between TensorFlow variable_scope and name_scope?

Written on , in: ,

TensorFlow scopes are a fundamental building block that helps you organize your code in a cleaner, clever and more understandable way. They also allow you to share existing variables, something really useful in some projects and architectures.

There are two types of scope you can choose from: name_scope and variable_scope. But what are the differences between them? Let’s see.

Note: This “post” assumes some previous knowledge of TensorFlow and scopes. You can read about them on the official doc.

Note 2: This post is associated with this Notebook

Table of contents

name_scope

Fact 1

It is used by all the operations and the variables created using the Variable constructor. It does not affect, however, the variables created using the get_variable function

def try_name_scope_one():
    tf.reset_default_graph()
    with tf.name_scope("first"):
        with tf.name_scope("second"):
            print(tf.constant(2, name="constant").name)
            print(tf.Variable(2, name="Variable").name)
            print(tf.multiply(tf.constant(2), tf.constant(3), name="multiply").name)
            print(tf.get_variable("get_variable", [1]).name)

try_name_scope_one()
#first/second/constant:0
#first/second/Variable:0
#first/second/multiply:0
#get_variable:0

Fact 2

It is not built with reuse in mind (normally, you want to share only variables). So, by default, if you reopen a previously closed name scope a brand new, unique, name scope will be created

def try_name_scope_two():
    tf.reset_default_graph()
    with tf.name_scope("first"):
        pass

    with tf.name_scope("first"):
        print(tf.constant(2, name="constant").name)
        print(tf.Variable(2, name="Variable").name)
        print(tf.multiply(tf.constant(2), tf.constant(3), name="multiply").name)

try_name_scope_two()
#first_1/constant:0
#first_1/Variable:0
#first_1/multiply:0

Fact 3

This is the default behaviour. You can change it.

If the name argument of the name_scope function is:

  • A string not ending with “/”, it will behave as expected. In particular:
    • The new name will be added as a suffix to the name_scope you are actually in
    • If the new scope isn’t unique, a suffix will be added to make it unique (as seen in fact 2)
  • A string ending with “/”, TensorFlow will treat it like a complete name scope. It will not add any prefix and it won’t try to make it unique. You are in control now.
def try_name_scope_three():
    tf.reset_default_graph()
    with tf.name_scope("first"):
        pass
    with tf.name_scope("first/"): #note the trailing slash
        print(tf.constant(2, name="constant").name)
        print(tf.Variable(2, name="Variable").name)
        print(tf.multiply(tf.constant(2), tf.constant(3), name="multiply").name)

try_name_scope_three()
#first/constant:0
#first/Variable:0
#first/multiply:0

  • An empty string or None, the name_scope will be reset to the top level (empty) name scope.

variable_scope

Fact 1

It is used by all the variables created using the get_variable function

def try_variable_scope_one():
    tf.reset_default_graph()
    with tf.variable_scope("first"):
        with tf.variable_scope("second"):
            print(tf.get_variable("get_variable", [1]).name)

try_variable_scope_one()
#first/second/get_variable:0

Fact 2

It is built with reuse in mind. So you can reopen a previously closed scope without any problem

def try_variable_scope_two():
    tf.reset_default_graph()
    with tf.variable_scope("first"):
        pass
    with tf.variable_scope("first"):
        print(tf.get_variable("get_variable", [1]).name)

try_variable_scope_two()
#first/second/get_variable:0

Fact 3

The variable_scope allows the variables sharing. How? Take a look at this other post.

How do they interact?

Fact 1

By default, when you open a variable_scope a name_scope with the same name will be created

def try_together_one():
    tf.reset_default_graph()
    with tf.variable_scope("first"):
        print(tf.constant(2, name="constant").name)
        print(tf.Variable(2, name="Variable").name)
        print(tf.multiply(tf.constant(2), tf.constant(3), name="multiply").name)
        print(tf.get_variable("get_variable", [1]).name)

try_together_one()
#first/constant:0
#first/Variable:0
#first/multiply:0
#first/get_variable:0

Fact 2

So, if you reopen a previously closed variable_scope, another name_scope with the same name will be created. As stated in name scope fact 2, a suffix will be added to this name scope to make it unique.

def try_together_two():
    tf.reset_default_graph()
    with tf.variable_scope("first"):
        pass
    with tf.variable_scope("first"):
        print(tf.constant(2, name="constant").name)
        print(tf.Variable(2, name="Variable").name)
        print(tf.multiply(tf.constant(2), tf.constant(3), name="multiply").name)
        print(tf.get_variable("get_variable", [1]).name)

try_together_two()
#first_1/constant:0
#first_1/Variable:0
#first_1/multiply:0
#first/get_variable:0

Fact 3

You can disable the automatic creation of a name_scope

def try_together_three():
    tf.reset_default_graph()
    with tf.variable_scope("first", auxiliary_name_scope = False):
        print(tf.constant(2, name="constant").name)
        print(tf.Variable(2, name="Variable").name)
        print(tf.multiply(tf.constant(2), tf.constant(3), name="multiply").name)
        print(tf.get_variable("get_variable", [1]).name)

try_together_three()
#constant:0
#Variable:0
#multiply:0
#first/get_variable:0

Fact 4

Once you know how the name_scope works, you can override the default behaviour the way you prefer

def try_together_four():
    tf.reset_default_graph()
    with tf.variable_scope("first"):
        with tf.name_scope("another/"):  #note the trailing slash
            print(tf.constant(2, name="constant").name)
            print(tf.Variable(2, name="Variable").name)
            print(tf.multiply(tf.constant(2), tf.constant(3), name="multiply").name)
            print(tf.get_variable("get_variable", [1]).name)

try_together_four()
#another/constant:0
#another/Variable:0
#another/multiply:0
#first/get_variable:0

Fact 5

With that in mind, you can now reopen a previously closed variable_scope and obtain the same name_scope as before

def try_together_five():
    tf.reset_default_graph()
    with tf.variable_scope("first") as scope:
        pass
    with tf.variable_scope(scope, auxiliary_name_scope = False):
         with tf.name_scope(scope.original_name_scope):
            print(tf.constant(2, name="constant").name)
            print(tf.Variable(2, name="Variable").name)
            print(tf.multiply(tf.constant(2), tf.constant(3), name="multiply").name)
            print(tf.get_variable("get_variable", [1]).name)

try_together_five()
#first/constant:0
#first/Variable:0
#first/multiply:0
#first/get_variable:0

An alternative way is the following:

def try_together_six():
    tf.reset_default_graph()
    with tf.variable_scope("first"):
        pass
    with tf.variable_scope("first", auxiliary_name_scope = False):
         with tf.name_scope("first/"):
            print(tf.constant(2, name="constant").name)
            print(tf.Variable(2, name="Variable").name)
            print(tf.multiply(tf.constant(2), tf.constant(3), name="multiply").name)
            print(tf.get_variable("get_variable", [1]).name)

try_together_six()
#first/constant:0
#first/Variable:0
#first/multiply:0
#first/get_variable:0

comments powered by Disqus