Inception V3，发生异常：从检查点还原NotFoundError失败

Question

我正在尝试验证我培训的模型。奇怪的是，一个月前我用同样的管道成功地做到了这一点。但现在我似乎无法理解为什么会出现以下错误

我运行的紧跟在错误之前的代码：

    with tf.Graph().as_default():
        # Get images and labels from the dataset.
        images, labels, all_filenames, filename_queue = image_processing.inputs(dataset)

    # Number of classes in the Dataset label set plus 1.
    # Label 0 is reserved for an (unused) background class
    num_classes = dataset.num_classes() + 1
    print("there are %d classes!" % dataset.num_classes())

    # Build a Graph that computes the logits predictions from the
    # inference model.
    logits, _, end_points, net2048, sel_end_points = inception.inference(images, num_classes)

    # Calculate predictions.
    #max_percent =  tf.argmax(logits,1)
    #max_percent = tf.reduce_max(logits, reduction_indices=[1]) / tf.add_n(logits)
    max_percent = end_points['predictions']
    # max_percent = len(end_points)
    #for kk in range(len(labels)):
    #   #max_percent.append(end_points['predictions'][kk][labels[kk]])
    #   max_percent.append(labels[kk])
    if FLAGS.mode == '0_softmax':
      top_1_op = tf.nn.in_top_k(logits, labels, 1)
      top_5_op = tf.nn.in_top_k(logits, labels, 5)
    elif FLAGS.mode == '1_sigmoid':
      top_1_op = None
      top_5_op = None
    # Restore the moving average version of the learned variables for eval.

    variable_averages = tf.train.ExponentialMovingAverage(
        inception.MOVING_AVERAGE_DECAY)
    variables_to_restore = variable_averages.variables_to_restore()

    saver = tf.train.Saver(variables_to_restore)

    # Build the summary operation based on the TF collection of Summaries.
    summary_op = tf.summary.merge_all()

    graph_def = tf.get_default_graph().as_graph_def()
    summary_writer = tf.summary.FileWriter(FLAGS.eval_dir, graph_def=graph_def)

    while True:
      precision_at_1, current_score = _eval_once(saver, summary_writer, top_1_op, top_5_op, summary_op, max_percent, all_filenames, filename_queue, net2048, sel_end_points, logits, labels)

tf.global_variables_initializer()
  with tf.Session() as sess:
    ckpt = tf.train.get_checkpoint_state(FLAGS.checkpoint_dir)
    if ckpt and ckpt.model_checkpoint_path:
      if os.path.isabs(ckpt.model_checkpoint_path):
        # Restores from checkpoint with absolute path.

        saver.restore(sess, ckpt.model_checkpoint_path) <---- error encountered here

我认为我的检查点文件可能有问题，但当我加载一个月前使用的检查点时，我知道它没有损坏，它会给我同样的错误

`variables_to_restore = variable_averages.variables_to_restore()`

似乎给了我一堆不在我的检查点中的变量，我不知道为什么。好像密码在我背后变了

以下是要还原的变量中的变量 aux_logits/FC/weights/ExponentialMovingAveragemixed_8x8x2048b/branch3x3dbl/Conv_2/biases/ExponentialMovingAveragemixed_35x35x288a/branch1x1/Conv/biases/ExponentialMovingAveragemixed_8x8x2048b/branch1x1/Conv/biases/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv_4/biases/ExponentialMovingAveragemixed_35x35x288b/branch5x5/Conv_1/biases/ExponentialMovingAveragemixed_35x35x288b/branch3x3dbl/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv/biases/ExponentialMovingAverageaux_logits/proj/biases/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768b/branch7x7/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv_3/biases/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv_4/biases/ExponentialMovingAveragemixed_17x17x768c/branch7x7/Conv_2/biases/ExponentialMovingAveragemixed_8x8x2048b/branch1x1/Conv/weights/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv_4/weights/ExponentialMovingAverageaux_logits/proj/weights/ExponentialMovingAveragemixed_35x35x256a/branch3x3dbl/Conv_2/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv_4/biases/ExponentialMovingAveragemixed_8x8x2048a/branch3x3dbl/Conv/biases/ExponentialMovingAveragemixed_35x35x256a/branch3x3dbl/Conv_2/biases/ExponentialMovingAveragemixed_35x35x256a/branch_pool/Conv/biases/ExponentialMovingAveragemixed_17x17x768c/branch1x1/Conv/biases/ExponentialMovingAveragemixed_17x17x1280a/branch3x3/Conv_1/weights/ExponentialMovingAveragemixed_8x8x2048a/branch_pool/Conv/weights/ExponentialMovingAveragemixed_17x17x768a/branch3x3dbl/Conv_2/weights/ExponentialMovingAveragemixed_35x35x256a/branch3x3dbl/Conv/biases/ExponentialMovingAveragemixed_8x8x2048a/branch3x3dbl/Conv_1/biases/ExponentialMovingAveragemixed_8x8x2048b/branch3x3dbl/Conv_2/weights/ExponentialMovingAverageconv0/biases/ExponentialMovingAveragemixed_17x17x768a/branch3x3dbl/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7/Conv/biases/ExponentialMovingAveragemixed_35x35x256a/branch5x5/Conv/biases/ExponentialMovingAveragemixed_35x35x288b/branch3x3dbl/Conv_2/weights/ExponentialMovingAveragemixed_17x17x768c/branch1x1/Conv/weights/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv/weights/ExponentialMovingAveragemixed_17x17x768d/branch7x7/Conv/weights/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv_3/weights/ExponentialMovingAveragemixed_17x17x768e/branch7x7/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv_4/weights/ExponentialMovingAveragemixed_17x17x1280a/branch7x7x3/Conv_1/weights/ExponentialMovingAveragemixed_35x35x288a/branch5x5/Conv/biases/ExponentialMovingAveragemixed_8x8x2048b/branch3x3dbl/Conv_3/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv_4/biases/ExponentialMovingAverageconv0/weights/ExponentialMovingAveragemixed_35x35x256a/branch3x3dbl/Conv/weights/ExponentialMovingAveragemixed_35x35x288a/branch5x5/Conv/weights/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv_2/weights/ExponentialMovingAveragemixed_17x17x768c/branch7x7/Conv_2/weights/ExponentialMovingAverageaux_logits/FC/biases/ExponentialMovingAveragemixed_35x35x288a/branch3x3dbl/Conv/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7/Conv_2/biases/ExponentialMovingAveragemixed_8x8x2048b/branch3x3/Conv/biases/ExponentialMovingAveragemixed_35x35x288a/branch3x3dbl/Conv_2/biases/ExponentialMovingAverageconv1/biases/ExponentialMovingAveragemixed_17x17x768e/branch_pool/Conv/biases/ExponentialMovingAveragemixed_35x35x288b/branch1x1/Conv/biases/ExponentialMovingAveragemixed_8x8x2048b/branch3x3/Conv/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv_4/weights/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv/biases/ExponentialMovingAveragemixed_8x8x2048a/branch_pool/Conv/biases/ExponentialMovingAveragemixed_35x35x256a/branch_pool/Conv/weights/ExponentialMovingAveragemixed_17x17x768d/branch_pool/Conv/biases/ExponentialMovingAveragemixed_8x8x2048b/branch3x3/Conv_2/biases/ExponentialMovingAveragemixed_35x35x256a/branch3x3dbl/Conv_1/biases/ExponentialMovingAveragemixed_8x8x2048b/branch3x3dbl/Conv_3/weights/ExponentialMovingAverageaux_logits/Conv/weights/ExponentialMovingAveragemixed_8x8x2048a/branch3x3dbl/Conv/weights/ExponentialMovingAveragemixed_35x35x256a/branch1x1/Conv/biases/ExponentialMovingAveragemixed_8x8x2048a/branch3x3/Conv_2/biases/ExponentialMovingAveragemixed_35x35x256a/branch1x1/Conv/weights/ExponentialMovingAveragemixed_35x35x288a/branch_pool/Conv/biases/ExponentialMovingAverageaux_logits/Conv/biases/ExponentialMovingAveragemixed_35x35x288b/branch1x1/Conv/weights/ExponentialMovingAveragemixed_17x17x1280a/branch7x7x3/Conv_1/biases/ExponentialMovingAveragemixed_8x8x2048a/branch3x3dbl/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv/biases/ExponentialMovingAveragemixed_17x17x768e/branch7x7/Conv_2/weights/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv_2/weights/ExponentialMovingAveragemixed_8x8x2048a/branch1x1/Conv/biases/ExponentialMovingAveragemixed_17x17x768c/branch7x7/Conv/biases/ExponentialMovingAveragemixed_17x17x768e/branch_pool/Conv/weights/ExponentialMovingAveragemixed_17x17x768c/branch7x7/Conv/weights/ExponentialMovingAveragemixed_8x8x2048b/branch_pool/Conv/biases/ExponentialMovingAveragelogits/logits/weights/ExponentialMovingAverageconv2/biases/ExponentialMovingAveragemixed_35x35x288b/branch3x3dbl/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv_1/weights/ExponentialMovingAveragemixed_8x8x2048a/branch3x3/Conv_2/weights/ExponentialMovingAveragemixed_35x35x288b/branch5x5/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768d/branch_pool/Conv/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv_4/weights/ExponentialMovingAveragelogits/logits/biases/ExponentialMovingAveragemixed_35x35x256a/branch5x5/Conv/weights/ExponentialMovingAveragemixed_35x35x288b/branch3x3dbl/Conv_1/weights/ExponentialMovingAveragemixed_8x8x2048a/branch1x1/Conv/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7/Conv_2/weights/ExponentialMovingAveragemixed_35x35x288a/branch_pool/Conv/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv/weights/ExponentialMovingAveragemixed_35x35x256a/branch3x3dbl/Conv_1/weights/ExponentialMovingAveragemixed_35x35x288a/branch3x3dbl/Conv_2/weights/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768d/branch1x1/Conv/biases/ExponentialMovingAveragemixed_8x8x2048a/branch3x3dbl/Conv_1/weights/ExponentialMovingAverageconv2/weights/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv_3/biases/ExponentialMovingAveragemixed_35x35x288b/branch5x5/Conv/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv_2/weights/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv_3/weights/ExponentialMovingAveragemixed_35x35x288b/branch5x5/Conv/weights/ExponentialMovingAveragemixed_8x8x2048a/branch3x3dbl/Conv_2/weights/ExponentialMovingAveragemixed_8x8x2048a/branch3x3dbl/Conv_3/biases/ExponentialMovingAveragemixed_17x17x768b/branch7x7/Conv/biases/ExponentialMovingAveragemixed_17x17x1280a/branch3x3/Conv_1/biases/ExponentialMovingAverageconv3/weights/ExponentialMovingAveragemixed_17x17x768c/branch7x7/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv_3/biases/ExponentialMovingAveragemixed_17x17x768c/branch7x7/Conv_1/biases/ExponentialMovingAveragemixed_17x17x1280a/branch7x7x3/Conv/weights/ExponentialMovingAveragemixed_17x17x768a/branch3x3/Conv/biases/ExponentialMovingAveragemixed_17x17x1280a/branch7x7x3/Conv/biases/ExponentialMovingAveragemixed_17x17x1280a/branch7x7x3/Conv_3/biases/ExponentialMovingAveragemixed_8x8x2048a/branch3x3/Conv/biases/ExponentialMovingAveragemixed_17x17x768e/branch7x7/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv_1/biases/ExponentialMovingAveragemixed_8x8x2048a/branch3x3/Conv/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv_3/weights/ExponentialMovingAveragemixed_17x17x768a/branch3x3/Conv/weights/ExponentialMovingAveragemixed_8x8x2048a/branch3x3dbl/Conv_3/weights/ExponentialMovingAveragemixed_8x8x2048b/branch3x3dbl/Conv/biases/ExponentialMovingAveragemixed_8x8x2048a/branch3x3/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768c/branch_pool/Conv/biases/ExponentialMovingAverageconv3/biases/ExponentialMovingAveragemixed_17x17x768a/branch3x3dbl/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768b/branch1x1/Conv/biases/ExponentialMovingAveragemixed_8x8x2048b/branch3x3dbl/Conv/weights/ExponentialMovingAveragemixed_17x17x1280a/branch7x7x3/Conv_3/weights/ExponentialMovingAveragemixed_35x35x288b/branch3x3dbl/Conv/biases/ExponentialMovingAveragemixed_35x35x288a/branch3x3dbl/Conv_1/biases/ExponentialMovingAveragemixed_35x35x288a/branch1x1/Conv/weights/ExponentialMovingAverageconv1/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768e/branch1x1/Conv/biases/ExponentialMovingAveragemixed_35x35x288b/branch_pool/Conv/biases/ExponentialMovingAverageconv4/biases/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv_3/biases/ExponentialMovingAveragemixed_35x35x288a/branch5x5/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768a/branch3x3dbl/Conv/biases/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768c/branch_pool/Conv/weights/ExponentialMovingAveragemixed_17x17x768d/branch7x7/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768a/branch3x3dbl/Conv/weights/ExponentialMovingAveragemixed_17x17x768e/branch1x1/Conv/weights/ExponentialMovingAveragemixed_35x35x288b/branch3x3dbl/Conv/weights/ExponentialMovingAveragemixed_8x8x2048b/branch3x3dbl/Conv_1/biases/ExponentialMovingAveragemixed_8x8x2048b/branch_pool/Conv/weights/ExponentialMovingAveragemixed_17x17x1280a/branch7x7x3/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768b/branch_pool/Conv/biases/ExponentialMovingAveragemixed_35x35x288b/branch_pool/Conv/weights/ExponentialMovingAveragemixed_17x17x1280a/branch3x3/Conv/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7/Conv_1/biases/ExponentialMovingAveragemixed_35x35x256a/branch5x5/Conv_1/weights/ExponentialMovingAveragemixed_8x8x2048b/branch3x3/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768b/branch1x1/Conv/weights/ExponentialMovingAveragemixed_17x17x768d/branch7x7dbl/Conv/weights/ExponentialMovingAveragemixed_17x17x768e/branch7x7/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv/weights/ExponentialMovingAveragemixed_35x35x288a/branch5x5/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768e/branch7x7dbl/Conv/biases/ExponentialMovingAveragemixed_17x17x768e/branch7x7/Conv/biases/ExponentialMovingAveragemixed_17x17x1280a/branch3x3/Conv/weights/ExponentialMovingAveragemixed_35x35x288a/branch3x3dbl/Conv/weights/ExponentialMovingAveragemixed_17x17x768b/branch_pool/Conv/weights/ExponentialMovingAveragemixed_17x17x768e/branch7x7/Conv/weights/ExponentialMovingAveragemixed_17x17x1280a/branch7x7x3/Conv_2/weights/ExponentialMovingAveragemixed_8x8x2048a/branch3x3/Conv_1/weights/ExponentialMovingAveragemixed_8x8x2048b/branch3x3/Conv_1/weights/ExponentialMovingAverageconv4/weights/ExponentialMovingAveragemixed_35x35x288a/branch3x3dbl/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768a/branch3x3dbl/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv_2/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7dbl/Conv_1/weights/ExponentialMovingAveragemixed_17x17x768c/branch7x7dbl/Conv_3/weights/ExponentialMovingAveragemixed_17x17x768d/branch7x7/Conv_2/weights/ExponentialMovingAveragemixed_8x8x2048b/branch3x3dbl/Conv_1/weights/ExponentialMovingAveragemixed_35x35x256a/branch5x5/Conv_1/biases/ExponentialMovingAveragemixed_17x17x768d/branch7x7/Conv_2/biases/ExponentialMovingAveragemixed_17x17x768d/branch1x1/Conv/weights/ExponentialMovingAveragemixed_8x8x2048b/branch3x3/Conv_2/weights/ExponentialMovingAveragemixed_17x17x768b/branch7x7/Conv/weights/ExponentialMovingAverage

这是一个错误：

    Exception has occurred: NotFoundError Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: Key aux_logits/Conv/biases/ExponentialMovingAverage not found in checkpoint [[node save/RestoreV2 (defined at /02_testing/xClasses/inception/nc_inception_eval.py:404) ]] 

    Original stack trace for 'save/RestoreV2': 
    File "/.vscode-server/extensions/ms-python.python-2019.10.44104/pythonFiles/ptvsd_launcher.py", line 43, in <module> main(ptvsdArgs) 
    File "/.vscode-server/extensions/ms-python.python-2019.10.44104/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 432, in main run() 
    File "/.vscode-server/extensions/ms-python.python-2019.10.44104/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 316, in run_file runpy.run_path(target, run_name='__main__') 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) 
    File "/02_testing/xClasses/nc_imagenet_eval.py", line 271, in <module> tf.app.run() 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/absl/app.py", line 300, in run _run_main(main, args) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main sys.exit(main(argv)) 
    File "/02_testing/xClasses/nc_imagenet_eval.py", line 65, in main precision_at_1, current_score = nc_inception_eval.evaluate(dataset) 
    File "/02_testing/xClasses/inception/nc_inception_eval.py", line 404, in evaluate saver = tf.train.Saver(variables_to_restore) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 831, in __init__ self.build() 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 843, in build self._build(self._filename, build_save=True, build_restore=True) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 881, in _build build_restore=build_restore) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 514, in _build_internal restore_sequentially, reshape) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 334, in _AddRestoreOps restore_sequentially) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 581, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1696, in restore_v2 name=name) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op op_def=op_def) 
    File "/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__ self._traceback = tf_stack.extract_stack() File "/home/ubuntu/02_testing/xClasses/inception/nc_inception_eval.py", line 74, in _eval_once saver.restore(sess, ckpt.model_checkpoint_path) 
    File "/home/ubuntu/02_testing/xClasses/inception/nc_inception_eval.py", line 413, in evaluate precision_at_1, current_score = _eval_once(saver, summary_writer, top_1_op, top_5_op, summary_op, max_percent, all_filenames, filename_queue, net2048, sel_end_points, logits, labels) 
    File "/home/ubuntu/02_testing/xClasses/nc_imagenet_eval.py", line 65, in main precision_at_1, current_score = nc_inception_eval.evaluate(dataset) 
    File "/home/ubuntu/02_testing/xClasses/nc_imagenet_eval.py", line 271, in <module> tf.app.run()