INSTRUCTIONS.md

Dependencies:

• TensorFlow ≥ 1.6 with GPU support
• Tensorpack ≥ 0.9.8
• OpenCV ≥ 3
• horovod ≥ 0.15 with NCCL support
  • horovod has many installation options to optimize its multi-machine/multi-GPU performance. You might want to follow them.
• ImageNet data in its standard directory structure.
• TensorFlow zmq_ops (needed only for training with real data)

Model Zoo:

Model (click for details)	error rate (%)	error rate / attack success rate (%)
	clean images	10-step PGD	100-step PGD	1000-step PGD
ResNet152 Baseline --arch ResNet -d 152 ⬇️	37.7	47.5/5.5	58.3/31.0	61.0/36.1
ResNet152 Denoise --arch ResNetDenoise -d 152 ⬇️	34.7	44.3/4.9	54.5/26.6	57.2/32.7
ResNeXt101 DenoiseAll --arch ResNeXtDenoiseAll -d 101 ⬇️	31.6	44.0/4.9	55.6/31.5	59.6/38.1

Click the first column to download the model and obtain the flags to be used with the script.

Note:

1. As mentioned in the paper, the threat model is:

   1. Targeted attack, with one target label associated with each image. The target label is independently generated by uniformly sampling the incorrect labels.
   2. Maximum perturbation per pixel is 16.

   We do not consider untargeted attacks, nor do we let the attacker control the target labels, because we think such tasks are not realistic on the ImageNet-1k categories.

2. For each (attacker, model) pair, we provide both the error rate of our model, and the attack success rate of the attacker, on ImageNet validation set. A targeted attack is considered successful if the image is classified to the target label.

   For attackers, if you develop a new targeted attack method against our models, please compare its attack success rate with PGD. Error rate / accuracy is not a reasonable metric, because then the method can cheat by becoming close to untargeted attacks.

   For defenders, if you develop a new robust model, please compare its accuracy with our models. Attack success rate is not a reasonable metric, because then the model can cheat by making random predictions.

3. ResNeXt101 DenoiseAll is the submission that won the champion of black-box defense track in Competition on Adversarial Attacks and Defenses 2018. This model was trained with slightly different training settings therefore its results are not directly comparable with other models.

Evaluate White-Box Robustness:

To evaluate on one GPU, run this command:

python main.py --eval --load /path/to/model_checkpoint --data /path/to/imagenet \
  --attack-iter [INTEGER] --attack-epsilon 16.0 [--architecture-flags]

To reproduce our evaluation results, take "architecture flags" from the first column in the model zoo, and set the attack iteration. Iteration can be set to 0 to evaluate its clean image error rate. Note that the evaluation result may have a ±0.3 fluctuation due to the randomly-chosen target attack label and attack initialization.

Using a K-step attacker makes the evaluation K-times slower. To speed up evaluation, run it under MPI with multi-GPU or multiple machines, e.g.:

mpirun -np 8 python main.py --eval --load /path/to/model_checkpoint --data /path/to/imagenet \
  --attack-iter [INTEGER] --attack-epsilon 16.0 [--architecture-flags]

Evaluating the Res152 Denoise model against 100-step PGD attackers takes about 1 hour with 16 V100s.

Evaluate Black-Box Robustness:

We provide a command line option to produce predictions for an image directory, e.g.:

python main.py --eval-directory /path/to/image/directory --prediction-file predictions.txt \
	--load X101-DenseDenoise.npz -d 101 --arch ResNeXtDenoiseAll --batch 20

This will produce a file "predictions.txt" which contains the filename and predicted label for each image found in the directory. You can use this to evaluate its black-box robustness.

Our CAAD2018 submission is equivalent to the above command.

Train:

Our code can be used for both standard ImageNet training (with --attack-iter 0) and adversarial training. Adversarial training takes a long time and we recommend doing it only when you have a lot of GPUs.

To train, first start one data serving process on each machine:

$ ./third_party/serve-data.py --data /path/to/imagenet/ --batch 32

Then, launch a distributed job with MPI. You may need to consult your cluster administrator for the MPI command line arguments you should use. On a cluster with InfiniBand, it may look like this:

 mpirun -np 16 -H host1:8,host2:8 --output-filename train.log \
    -bind-to none -map-by slot -mca pml ob1 \
    -x NCCL_IB_CUDA_SUPPORT=1 -x NCCL_IB_DISABLE=0 -x NCCL_DEBUG=INFO \
    -x PATH -x PYTHONPATH -x LD_LIBRARY_PATH \
    python main.py --data /path/to/imagenet \
        --batch 32 --attack-iter [INTEGER] --attack-epsilon 16.0 [--architecture-flags]

If your cluster is managed by slurm , we provide some sample slurm job scripts for your reference.

The training code will also perform distributed evaluation of white-box robustness.

Training Speed:

With 30 attack iterations during training, the Res152 Baseline model takes about 52 hours to finish training on 128 V100s.

Under the same setting, the Res152 Denoise model takes about 90 hours on 128 V100s. Note that the model actually does not add much computation to the baseline, but it lacks efficient GPU implementation for the softmax version of non-local operation. The dot-product version, on the other hand, is much faster.

If you use CUDA≥9.2, TF≥1.12 on Volta GPUs, the flag --use-fp16xla will enable XLA-optimized FP16 PGD attack, which reduces training time about 2x, with a drop of about 3% robustness.