Upgrade from r11 to r12 prodeuces "Variables not defined" when using any optimizer but GradientDescentOptimizer

[germanRos-TRI]

After a recent upgrade to the latest version of tensorflow in github, several things stop working. I found out that all the optimizers, such as Adam or Adagrad are now producing an error related to variable scope that I have not managed to solve yet. However, GradientDescentOptimizer works fine.

It may be related to the issue: #5652

The error looks like this:

File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 651, in _get_single_variable
    "VarScope?" % name)
ValueError: Variable filter/Adadelta/ does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?

It works fine with tensorflow r11

Operating System: Ubuntu 16 and Ubuntu 14
Installed version of CUDA and cuDNN: cuda 8.0, cuda 5.1
cuda.txt
The commit hash 6dc8dea
Build time: Wed Nov 2 17:54:14 2016 (1478109254)
Build timestamp: 1478109254
Build timestamp as int: 1478109254

Find below a minimal version that causes the error:

import tensorflow as tf
import pdb

def main():

    ## !!! change this to test the different behaviors !!!
    #optimizer = tf.train.GradientDescentOptimizer(1e-3)                 # This one is working
    optimizer = tf.train.AdamOptimizer(1e-3, beta1=0.9, beta2=0.999999) # This one is not working
    #optimizer = tf.train.AdagradOptimizer(1e-3)                         # This one is not working
    #optimizer = tf.train.AdadeltaOptimizer(1e-3)                        # This one is not working
	
    list_grads = []
    for i in xrange(2):
        with tf.device('/gpu:%d' % i):
            with tf.name_scope('%d' % i) as scope:
                W = tf.get_variable(name="filter", initializer=tf.random_uniform_initializer(dtype=tf.float32), shape=[5, 1])
                X = tf.get_variable(name="data", initializer=tf.random_uniform_initializer(dtype=tf.float32), shape=[5, 1])
                Y_ = tf.get_variable(name="out", initializer=tf.random_uniform_initializer(dtype=tf.float32), shape=[5, 1])
                Y = W+X
                loss =tf.reduce_mean(Y-Y_)
                grad = optimizer.compute_gradients(loss)
                list_grads.append(grad)

                tf.get_variable_scope().reuse_variables()	
    
    grads = list_grads[0] + list_grads[1]
    #pdb.set_trace()

    op_train = optimizer.apply_gradients(grads)

    init_global = tf.global_variables_initializer()
    init_local =  tf.local_variables_initializer()

    sess = tf.Session()
    sess.run([init_global, init_local])

    _, sol = sess.run([op_train, loss])
    print(str(sol))

if (__name__ == '__main__'):
	main()

[germanRos-TRI]

Comment and uncomment the different optimizers to see the behavior. As explained, the only work working is GradientDescentOptimizer. This behavior does not occur in version r11 and it does not happen either if the averaging is not performed. Any clue?

[sherrym]

In particular, this commit causes the problem: 0fc86dd.

[germanRos-TRI]

Please, tell me if I can help.

[sherrym]

You can revert the changes to slot_creater.py or fix the changes and send a PR. Thanks.

[lukaszkaiser]

Sorry sherry -- the current behaviour is correct. Your code is leaking reuse -- it just wasn't checked before. It could cause all other troubles, and I think we should correct the leaky reuse cases, not revert the slot change. I'll write more on the test cases, closing this.

[germanRos-TRI]

Then, just to be clear. How do we get the desired results? Does the reuse need to be done in a different way? This has been directly taken from the cifar10 multi-gpu example.

Thanks

[lukaszkaiser]

To clarify, we just need to put a scope around the model-construction part.

with tf.variable_scope(tf.get_variable_scope()) as scope:
  for i in xrange(2):
    ... code as before until ...reuse_varables() ....

grads = list_grads[0] + list_grads[1]
... rest of code as before ...

Hope that helps!

[germanRos-TRI]

Thanks lukaszkaiser. It works perfectly fine now!

[wookayin]

@lukaszkaiser Hello, I found that your workaround to put a variable_scope which wraps the outermost num_gpus loop, but I am still confused why it does eliminate the error.

with tf.variable_scope(tf.get_variable_scope()) as vscope:
  for i in xrange(FLAGS.num_gpus):
    with tf.device('/gpu:%d' % i):
      with tf.name_scope('%s_%d' % (cifar10.TOWER_NAME, i)) as scope:
        loss = tower_loss(scope)
        tf.get_variable_scope().reuse_variables()     # HERE

Is it just because that the tf.get_variable_scope() (which is identical to vscope) is explicitly created than the implicit default? Then, what do these two VariableScope objects differ in?

What do you mean by "leaky reuse"? Could you please clarify me?
/cc @cesc-park

[lukaszkaiser]

Sure, let me try to clarify.

When you do tf.get_variable_scope().reuse_variables() you set the current scope to reuse variables. If you call the optimizer in such scope, it's trying to reuse slot variables, which it cannot find, so it throws an error. If you put a scope around, the tf.get_variable_scope().reuse_variables() only affects that scope, so when you exit it, you're back in the non-reusing mode, the one you want.

Hope that helps, let me know if I should clarify more.

[wookayin]

Ah, great. Your explanation is clear and helpful. Thanks!

To sum, a thing to remember is that where the (Adam-like) optimizer acts, i.e. opt.apply_gradients(...) (which is where the error is thrown from) should lie in the scope with reuse=False in order to properly create the slot variables.

[yanxp]

@wagonhelm hello,when use Adam,how do you solve it?please tell me the details,thank you

[zhouyang209117]

Find below a minimal version that causes the error:

import tensorflow as tf
import numpy as np
class SimpleModel:
    def __init__(self):
        self.loss = self.calc_loss()
        self.train = self.train_model(self.loss)
    def calc_loss(self):
        W = tf.get_variable("w", [1])
        b = tf.Variable(tf.zeros([1]))
        y = W * x_data + b
        return tf.reduce_mean(tf.square(y - y_data))
    def train_model(self, loss):
        return tf.train.AdamOptimizer(0.5).minimize(loss)
        # return tf.train.GradientDescentOptimizer(0.5)
x_data = np.random.rand(100).astype(np.float32)
y_data = x_data * 0.1 + 0.3
s1 = SimpleModel()
tf.get_variable_scope().reuse_variables()
s2 = SimpleModel()

The error looks like this:

 File "D:\MyProgram\Install\Anaconda3\envs\tensorflow121\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 682, in _get_single_variable
    "VarScope?" % name)
ValueError: Variable embeddings/Adam_2/ does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?

tensorflow version

1.2.1

[Huayra007]

@lukaszkaiser Hi, I've confronted with this problem when using AdamOptimizer, I've tried your suggestion but it still doesn't work. Could you please help me change the code?
Besides, I'm not very familiar with TF, wish you can help me point out anything not appropriate in this code. Thank you!

import tensorflow as tf
import numpy as np
import datetime
import matplotlib.pyplot as plt
import os
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data/')
os.environ['CUDA_VISIBLE_DEVICES']='4'

sample_image = mnist.train.next_batch(1)[0]
print (sample_image.shape)

sample_image = sample_image.reshape([28,28])
plt.imshow(sample_image,cmap='Greys')

def discriminator(images,reuse=False,):
    if(reuse):
        tf.get_variable_scope().reuse_variables()

    with tf.variable_scope('D_conv1'):
        d_w1 = tf.get_variable('d_w1',[5,5,1,32],initializer=tf.truncated_normal_initializer(stddev=0.02))
        d_b1 = tf.get_variable('d_b1',[32],initializer=tf.constant_initializer(0))
        d1 = tf.nn.conv2d(input=images,filter=d_w1,strides=[1,1,1,1],padding='SAME')+d_b1
        d1 = tf.nn.relu(d1)
        d1 = tf.nn.avg_pool(d1,ksize=[1,2,2,1],strides=[1,2,2,1],padding='SAME')

    with tf.variable_scope('D_conv2'):
        d_w2 = tf.get_variable('d_w2',[5,5,32,64],initializer=tf.truncated_normal_initializer(stddev=0.02))
        d_b2 = tf.get_variable('d_b2',[64],initializer=tf.constant_initializer(0))
        d2 = tf.nn.conv2d(input=d1,filter=d_w2,strides=[1,1,1,1],padding='SAME')+d_b2
        d2 = tf.nn.relu(d2)
        d2 = tf.nn.avg_pool(d2,ksize=[1,2,2,1],strides=[1,2,2,1],padding='SAME')

    with tf.variable_scope('D_fcn3'):
        d_w3 = tf.get_variable('d_w3',[7*7*64,1024],initializer=tf.truncated_normal_initializer(stddev=0.02))
        d_b3 = tf.get_variable('d_b3',[1024],initializer=tf.constant_initializer(0))
        d3 = tf.matmul(tf.reshape(d2,[-1,7*7*64]),d_w3)+d_b3
        d3 = tf.nn.relu(d3)

    with tf.variable_scope('D_fc4'):
        d_w4 = tf.get_variable('d_w4',[1024,1],initializer=tf.truncated_normal_initializer(stddev=0.02))
        d_b4 = tf.get_variable('d_b4',[1],initializer=tf.constant_initializer(0))
        d4 = tf.matmul(d3,d_w4)+d_b4
        d4 = tf.nn.sigmoid(d4,name='d4')

    return d4

def generator(z,batch_size,z_dim):
    g_w1 = tf.get_variable('g_w1',[z_dim,56*56],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g_b1 = tf.get_variable('g_b1',[56*56],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g1 = tf.matmul(z,g_w1)+g_b1
    g1 = tf.reshape(g1,[-1,56,56,1])
    g1 = tf.contrib.layers.batch_norm(g1,epsilon=1e-5,scope='bn1')
    g1 = tf.nn.relu(g1)

    g_w2 = tf.get_variable('g_w2',[3,3,1,z_dim/2],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g_b2 = tf.get_variable('g_b2',[z_dim/2],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g2 = tf.nn.conv2d(g1,filter=g_w2,strides=[1,1,1,1],padding='SAME')+g_b2
    g2 = tf.contrib.layers.batch_norm(g2,epsilon=1e-5,scope='bn2')
    g2 = tf.nn.relu(g2)
    g2 = tf.image.resize_images(g2,[56,56])

    g_w3 = tf.get_variable('g_w3',[3,3,z_dim/2,z_dim/4],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g_b3 = tf.get_variable('g_b3',[z_dim/4],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g3 = tf.nn.conv2d(g2,filter=g_w3,strides=[1,1,1,1],padding='SAME')+g_b3
    g3 = tf.contrib.layers.batch_norm(g3,epsilon=1e-5,scope='bn3')
    g3 = tf.nn.relu(g3)
    g3 = tf.image.resize_images(g3,[56,56])

    g_w4 = tf.get_variable('g_w4',[1,1,z_dim/4,1],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g_b4 = tf.get_variable('g_b4',[1],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g4 = tf.nn.conv2d(g3,filter=g_w4,strides=[1,2,2,1],padding='SAME')+g_b4
    g4 = tf.nn.sigmoid(g4)

    return g4

tf.reset_default_graph()
batch_size =100
z_dimension = 100

z_placeholder = tf.placeholder(tf.float32,[None,z_dimension],name='z_placeholder')
x_placeholder = tf.placeholder(tf.float32,[None,28,28,1],name='x_placeholder')

with tf.variable_scope(tf.get_variable_scope()):
    Gz = generator(z_placeholder,batch_size,z_dimension)
    Dx = discriminator(x_placeholder)
    Dg = discriminator(Gz,reuse=True)

d_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=tf.ones_like(Dx),logits=Dx))
d_loss_fake = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=tf.zeros_like(Dg),logits=Dg))
g_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=tf.ones_like(Dg),logits=Dg))

tvars = tf.trainable_variables()
d_vars = [var for var in tvars if 'd_' in var.name]
g_vars = [var for var in tvars if 'g_' in var.name]

print ([v.name for v in d_vars])
print ([v.name for v in g_vars])
'''
d_trainer_real = tf.train.GradientDescentOptimizer(0.0003).minimize(d_loss_real,var_list=d_vars)
d_trainer_fake = tf.train.GradientDescentOptimizer(0.003).minimize(d_loss_fake,var_list=d_vars)
g_trainer = tf.train.GradientDescentOptimizer(0.001).minimize(g_loss,var_list=g_vars)
'''
with tf.variable_scope(tf.get_variable_scope()):
    d_trainer_real = tf.train.AdamOptimizer(0.0003).minimize(d_loss_real, var_list=d_vars)
    tf.get_variable_scope().reuse_variables()
    d_trainer_fake = tf.train.AdamOptimizer(0.0003).minimize(d_loss_fake, var_list=d_vars)
    g_trainer = tf.train.AdamOptimizer(0.0001).minimize(g_loss, var_list=g_vars)

tf.summary.scalar('Discriminator_loss_real',d_loss_real)
tf.summary.scalar('Discriminator_loss_fake',d_loss_fake)
tf.summary.scalar('Generator_loss',g_loss)

images_for_tensorboard = generator(z_placeholder,batch_size,z_dimension)
tf.summary.image('Generated_images',images_for_tensorboard,5)
merged = tf.summary.merge_all()
logdir = 'Tensorboard/' + datetime.datetime.now().strftime('%Y%m%d-%H%M%S') + '/'
writer = tf.summary.FileWriter(logdir,sess.graph)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

#Pre-train discriminator
for i in range(3000):
    z_batch = np.random.normal(0, 1, size=[batch_size,z_dimension])
    real_image_batch = mnist.train.next_batch(batch_size)[0].reshape([batch_size,28,28,1])
    _,__,dLossReal,dLossFake = sess.run([d_trainer_real,d_trainer_fake,d_loss_real,d_loss_fake],
                                       feed_dict={x_placeholder:real_image_batch,z_placeholder:z_batch})
    if(i%100==0):
        print ('dLossReal: ',dLossReal,'dLossFake: ',dLossFake)

#Train discriminator and generator together
for i in range(100000):
    real_image_batch = mnist.train.next_batch(batch_size)[0].reshape([batch_size,28,28,1])
    z_batch = np.random.normal(0,1,size=[batch_size,z_dimension])

    _,__,dLossReal,dLossFake = sess.run([d_trainer_real,d_trainer_fake,d_loss_real,d_loss_fake],
                                        feed_dict={x_placeholder:real_image_batch,z_placeholder:z_batch})
    z_batch = np.random.normal(0,1,size=[batch_size,z_dimension])
    _ = sess.run(g_trainer,feed_dict={z_placeholder:z_batch})

    if i%10 ==0:
        z_batch = np.random.normal(0,1,size=[batch_size,z_dimension])
        summary = sess.run(merged,feed_dict={x_placeholder:real_image_batch,z_placeholder:z_batch})
        writer.add_summary(summary,i)

    if i%100==0:
        print ('Iteration:',i,'at',datetime.datetime.now())
        z_batch = np.random.normal(0,1,size=[batch_size,z_dimension])
        generated_image = generator(z_placeholder,1,z_dimension)
        images = sess.run(generated_image,feed_dict={z_placeholder:z_batch})
        plt.imshow(images[0].reshape([28,28]),cmap='Greys')
        plt.savefig('Generated_images/'+str(i)+'.jpg')

        img = images[0].reshape([1,28,28,1])
        result = discriminator(x_placeholder)
        estimate = sess.run(result,feed_dict={x_placeholder:img})
        print ('Estimate:',estimate)

[ivanjacobs]

@Huayra007
if you remove the

images_for_tensorboard = generator(z_placeholder,batch_size,z_dimension)
tf.summary.image('Generated_images',images_for_tensorboard,5)

should be able to run it. You are calling 2 time your generator. So or you remove the snippet or you add reuse to your generator code as such:

def generator(z,batch_size,z_dim,reuse=False):
    if (reuse):
        tf.get_variable_scope().reuse_variables()

    g_w1 = tf.get_variable('g_w1',[z_dim,56*56],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g_b1 = tf.get_variable('g_b1',[56*56],dtype=tf.float32,initializ

and when you call it for the tensorboard as such:

with tf.variable_scope(tf.get_variable_scope()) as scope:
    images_for_tensorboard = generator(z_placeholder, batch_size, z_dimensions,reuse=True)
    tf.summary.image('Generated_images', images_for_tensorboard, 5)

I hope this helps. Good luck with your GANs ;-)

[haojitianya]

i am a student,i am not very familiar with tensorflow, i just follow @lukaszkaiser
and use with ' tf.variable_scope(tf.get_variable_scope(),reuse=tf.AUTO_REUSE) as scope:'
and delete the 'tf.get_variable_scope().reuse_variables()' my code is work .
i am runing the code of ROLO.

[farzinh]

I'm new in TF, I tried to use:
with tf.variable_scope(tf.get_variable_scope()) as scope:
but it didn't work, and I changed it to default, can you help me to change it in a right way?
I attached the python code :
exprgan.py.gz
this 4 method has tf.get_variable_scope().reuse_variables():
encoder, generator, discriminator_z and discriminator_img

[zaiedsarra]

@lukaszkaiser Hello I need your help :( , in the code i have the same error :ValueError: Variable G_fc/w does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=tf.AUTO_REUSE in VarScope?
But i didn't understand haw i can change the code (it works with GradientDescent)

this is the function

def generator(self, z, y, gender=None, reuse_variables=False, enable_tile_label=True, tile_ratio=1.0):
    if reuse_variables:
        tf.get_variable_scope().reuse_variables()
    num_layers = int(np.log2(self.size_image)) - int(self.size_kernel / 2)
    if enable_tile_label:
        duplicate = int(self.num_z_channels * tile_ratio / self.y_dim)
    else:
        duplicate = 1
    z = concat_label(z, y, duplicate=duplicate)
    if enable_tile_label:
        duplicate = int(self.num_z_channels * tile_ratio / 2)
    else:
        duplicate = 1

    size_mini_map = int(self.size_image / 2 ** num_layers)

    name = 'G_fc'
    current = fc(
        input_vector=z,
        num_output_length=self.num_gen_channels * size_mini_map * size_mini_map,
        name=name,
         # reuse = reuse_variables
    )

    current = tf.reshape(current, [-1, size_mini_map, size_mini_map, self.num_gen_channels])
    current = tf.nn.relu(current)
    current = concat_label(current, y)

    for i in range(num_layers):
        name = 'G_deconv' + str(i)
        current = tf.image.resize_nearest_neighbor(current, [size_mini_map * 2 ** (i + 1), size_mini_map * 2 ** (i + 1)])
        current = custom_conv2d(input_map=current, num_output_channels=int(self.num_gen_channels / 2 ** (i + 1)), name=name)
        current = tf.nn.relu(current)
        current = concat_label(current, y)

    name = 'G_deconv' + str(i + 1)
    current = tf.image.resize_nearest_neighbor(current, [self.size_image, self.size_image])
    current = custom_conv2d(input_map=current, num_output_channels=int(self.num_gen_channels / 2 ** (i + 2)), name=name)
    current = tf.nn.relu(current)
    current = concat_label(current, y)

    name = 'G_deconv' + str(i + 2)
    current = custom_conv2d(input_map=current, num_output_channels=self.num_input_channels, name=name)

    return tf.nn.tanh(current)

---

And this is the G_fc function

def fc(input_vector, num_output_length, name='fc'):
with tf.variable_scope(name):
stddev = np.sqrt(1.0 / (np.sqrt(input_vector.get_shape()[-1].value * num_output_length)))
w = tf.get_variable(
name='w',
shape=[input_vector.get_shape()[1], num_output_length],
dtype=tf.float32,
initializer=tf.random_normal_initializer(stddev=stddev)
)
b = tf.get_variable(
name='b',
shape=[num_output_length],
dtype=tf.float32,
initializer=tf.constant_initializer(0.0)
)
return tf.matmul(input_vector, w) + b

[SystemErrorWang]

i got the same error in multi-gpu training script, even when i include all the model define inside a with tf.variable_scope(tf.get_variable_scope(), reuse=tf.AUTO_REUSE) as scope: and manually added tf.get_variable_scope().reuse_variables() in each gpu. would like to know what happened, any information or suggestion is appreciated

[Traeyee]

i got the same error in multi-gpu training script, even when i include all the model define inside a with tf.variable_scope(tf.get_variable_scope(), reuse=tf.AUTO_REUSE) as scope: and manually added tf.get_variable_scope().reuse_variables() in each gpu. would like to know what happened, any information or suggestion is appreciated

Just do as @wookayin suggested

[SystemErrorWang]

@Traeyee thanks, fixed the problem after many attempts of changing tf.name_scope and tf.variable_scope.