Ph.D. in Computer Science

University of Exeter

Student: Han Wu
Supervisors: Prof. Johan Wahlström and Prof. Sareh Rowlands
Expected completion date: 24/May/2024

① Adversarial Driving: Attacking End-to-End Autonomous Driving System.

② Adversarial Detection: Attacking Object Detection in Real Time.

③ A Man-in-the-Middle Attack against Object Detection System.

④ Distributed Black-box Attack against Image Classification.

⑤ Coming soon.

⑥ Adversarial Tracking: Real-time Adversarial Attacks against Object Tracking.

⑦ Adversarial Patch: Physical Patch in Carla Simulator.

⑧ A WHite-box Adversarial Toolbox (WHAT).

⑨ A Black-box Adversarial Toolbox (BAT).

⑩ Coming soon.

Week 0

2022/08/15 - 2022/09/18

Three Demo Videos for ICRA 2023.

• Adversarial Driving: https://driving.wuhanstudio.uk/

• Adversarial Detection: https://detection.wuhanstudio.uk/

• Man-in-the-Middle Attack: https://minm.wuhanstudio.uk/

Three Draft Papers for ICRA 2023:

• Adversarial Driving: https://arxiv.org/abs/2103.09151

• Adversarial Detection: https://arxiv.org/abs/2209.01962

• Man-in-the-Middle Attack: https://arxiv.org/abs/2208.07174

The ICRA 2023 paper submission deadline is 15 September 2022.

One Draft Paper for MLSys 2023:

• Distributed Black-box Attack against Image Classification Cloud Services.

The MLSys 2023 paper submission deadline is 28 October 2022.

Week 1

2022/09/26 - 2022/10/02

Whether black-box attacks are real threats or just research stories?

MLSys 2023:

• Paper Submission: Friday, October 28, 2023 4pm ET
• Page Length: Up to 10 pages long, not including references (10 + n)

Survey

Black-box attacks rely on queries but attacking real-world image classification models in cloud services could cost 20,000 to 50,000 queries for a single attack, which means attacking a single image could cost $480 - $1200 and 5-14 hours (Ilyas et al.) and the attack is not guaranteed to succeed (the success rate is not 100%).

The following evaluation metrics are important for black-box attacks:

• Success Rate: Initial research focused on improving the success rate.
• Number of Queries: Recent research interests shifted to reducing the number of queries.
• Time Cost: I notice that reducing the number of queries is not the only way to accelerate black-box attacks (also possible via distributed queries).

In a survey paper (Bhambri et al.), black-box attacks are classified into:

• Gradient-based Methods
• Local Search
• Combinatorics
• Transferability

Problems

For black-box attacks against cloud services:

• The more queries we sent simultaneously, the faster the attack is.
• Do we need to start from scratch every time we attack the same model?
• Experiments: we shouldn't assume access to pre-processing methods.

Plan

Week 1 - Week 5 (Total: 5 weeks)

Week 1:

• Local Search

  • SimBA (2019)
  • Square Attack (2020)

Week 2:

• Gradient Estimation

  • Limited Query (2018)
  • Bandits (2019)

• The Draft for MLSys 2023. (Introduction)

Week 3: The Draft for MLSys 2023. (Methodology)

Week 4: The Draft for MLSys 2023. (Experiment)

Week 5: Revision & Submission.

Week 2

2022/10/03 - 2022/10/09

Acceleration Ratio

The distributed square attack can reduce the attack time from ~15h to ~4h.

Source Code: https://github.com/wuhanstudio/adversarial-classification

Common Mistakes

While implementing distributed black-box attacks, I noticed that some prior research made several common mistakes in their code. Their methods outperformed state-of-the-art partly because these mistakes gave them access to extra information that should not be available for black-box attacks. For example:

• They apply the perturbation after image resizing, assuming they know the input image size of a black-box model.
• The bandit attack does not clip the image while estimating gradients, assuming they can send invalid images (pixel value > 255 or < 0) to the black-box model.

Besides these mistakes, some methods are less effective while attacking real-world image classification APIs:

• They assume they can perturb a pixel value from 105 to 105.45 (float), while real-world input images are 8-bit integers, thus some perturbations are mitigated after data type conversion: int(105.45) == 105.

• Cloud APIs accept JPEG or PNG images (lossy compression) as input, while prior research assumes they can add perturbations to the raw pixels. After JPEG encoding, some adversarial perturbation gets lost.

As a result, some prior research compared their methods with others under on unfair settings. To prevent making these mistakes, I designed my own image classification cloud service for further research, so that we have a fair comparison.

Why they made these mistakes

They tested their attacks against local models on their computers (they have access to the file model.h5), and rely on themselves to restrain access to model information. Intentionally or unintentionally, they exploit extra information to improve their methods. For example:

• A Pytorch / Tensorflow model makes predictions using the function model.predict(X), and the function only accepts input images X as arrays. We can't stack images of different shapes to be an array, thus they resize the images to be the same size so that the function won't give errors, which is a mistake.
• The function model.predict(X) won't give errors even if the image X contains negative values. Thus they are unaware that their methods generate invalid images.
• The function model.predict(X) accepts float numbers, thus they did not convert their input images to be integers.
• The function model.predict(X) accepts raw images, thus they did not encode their inputs.

Plan

Week 1:

• Local Search

  [●] SimBA (2019)

  [●] Square Attack (2020)

Week 2:

• Gradient Estimation

  [  ] Limited Query (2018) (NES Gradient Estimation)

  [●] Bandits (2019) (The Data and Time priors)

• Cloud Service APIs

  [●] Cloud Vision (Google)

  [●] Imagga

  [●] Deep API (ours)

• The Draft for MLSys 2023. (Introduction)

Week 3

2022/10/10 - 2022/10/16

Most experiments were completed.

1. Pre-processing

Experiment Settings:

• Attacking 1000 images in the ImageNet dataset.

• $L_{\inf}$ = 0.05

• Max number of queries for each image = 1,000

Model Accuracy (1000 samples):

Before Pre-processing (Local):

After Pre-processing (Cloud):

2. Distributed Queries

Distributed queries (8 workers):

3. Distributed Attacks

Experiment Settings:

• Attacking 100 images in the ImageNet dataset.

• $L_{\inf}$ = 0.05

• max number of queries for each image = 1,000

3.1 Non-Distributed:

3.2 Horizontal Distribution:

3.3 Vertical Distribution:

Plan

• Week 4: Draft

• Week 5: Revision and Submission

Week 4

2022/10/17 - 2022/10/23

Completed all experiments except vertical distribution (need one more day).

The experiments used up $300 free tier provided by Microsoft Azure.

Plan

• Week 5: Revision and Submission

Week 5

2022/10/24 - 2022/10/30

One Draft Paper for MLSys 2023:

• Distributed Black-box Attack against Image Classification Cloud Service: PDF

One Demo Video for MLSys 2023.

• Adversarial Classification: https://distributed.wuhanstudio.uk/

Plan

• Week 6: Reinforcement Learning

Week 6

2022/10/31 - 2022/11/06

Reinforcement Learning

Textbook (Sutton & Barto):

• Reinforcement Learning

Online Courses:

• https://www.coursera.org/specializations/reinforcement-learning
• https://www.udemy.com/course/beginner-master-rl-1

Week 7

2022/11/07 - 2022/11/13

1. Reinforcement Learning

Trade-off between exploration and exploitation.

1.1 Tabular Solution

The state and action spaces are small enough for the approximate value functions to be represented as arrays or tables.

• Finite Markov Decision Process
• Dynamic Programming
• Monte Carlo Methods

TD learning combines some of the features of both Monte Carlo and Dynamic Programming (DP) methods.

• Temporal-Difference Learning
• n-step Bootsrapping

1.2 Approximate Solution Methods

Problems with arbitrary large state spaces.

• Prediction Task: Evaluate a given policy by estimating the value of taking actions following the policy.

• Control Task: Find the optimal policy that gets most rewards.

• On-policy: Estimate the value of a policy while using it for control.

• Off-policy: The policy used to generate behaviour, called the behaviour policy, may be unrelated to the policy that is evaluated and improved, called the estimation policy.

• On-policy TD Prediction
  • TD(0)
• On-policy TD Control
  • SARSA

• Off-policy TD Control
  • Q-Learning
  • Dyna-Q and Dynq-Q+
  • Expected SARSA

• Policy Gradient Methods
  • REINFORCE
  • Actor-Critic
  • Advantage Actor-Critic (A2C)

2. Webots Simulator (ROS2)

Multi-agent Reinforcement Learning.

Week 8

2022/11/14 - 2022/11/20

Multi-Agent Reinforcement Learning (MARL)

Trade-off between exploration and exploitation.

Paper:

• Multi-agent reinforcement learning: An overview
• Recent Progresses in Multi-Agent RL Theory

Books:

• RL Theory: Reinforcement Learning (Sutton & Barto, 2nd Edition)
• RL Practice: Reinforcement Learning: Industrial Applications of Intelligent Agents

Environments:

• Carla: Multi-Agent Connected Autonomous Driving (MACAD) Gym environments

Week 9

2022/11/21 - 2022/11/27

Travel

• Exeter --> London --> Hong Kong

Week 10

2022/11/28 - 2022/12/04

Travel

• Hong Kong (self-isolation)

Week 11

2022/12/05 - 2022/12/11

Travel

• Shen Zhen (self-isolation)

Week 12

2022/12/12 - 2022/12/18

Travel

• Visiting Student at SUSTech

Week 13

2022/12/19 - 2022/12/25

Travel

• Visiting Student at SUSTech

Week 14

2022/12/26 - 2023/01/01

Travel

• Guangzhou --> Singapore --> London --> Exeter

Week 0

2023/01/02 - 2023/01/08

Research Plan

• Milestone 1: Reinforcement Learning: Is reinforcement-learning-based end-to-end driving model secure?
• Milestone 2: Adversarial Tracking: Real-time adversarial attacks against Object Tracking.
• Milestone 3: Adversarial Patch: Physical Patch in Carla Simulator.

Week 1

2023/01/09 - 2023/01/15

Multi-Agent Connected Autonomous Driving (MACAD)

• A Review: https://arxiv.org/abs/2203.07676
• An RL env: https://github.com/praveen-palanisamy/macad-gym

OpenAI Gym environment

• Environments:

  • Hete Ncom | Inde | PO Intrx MA | TLS 1B2C1P TWN3-v0

  • Homo Ncom | Inde | PO Intrx MA | SS 3C TWN3-v0

• Observation Space: Images (168 x 168 x 3).

• Action Space: 9 discrete actions.

RL Methods

Baseline

Discrete Action Space:

• Deep Q (2013)
• A2C / A3C (2016)

Continuous Action Space:

• DDPG (2015)
• TRPO & PPO (2017)

Advanced

Continuous Action Space:

• SAC (2018)
• TD3 (2019)

Other Platforms

• https://github.com/oxwhirl/smac
• https://github.com/facebookresearch/nocturne

Week 2

2023/01/16 - 2023/01/22

MLSys 2023 (Author Feedback)

There will be another discussion before making the final decision (17th Feb.).

Future Plan

IROS (Mar. 2023)

• Adversarial Driving
  • End-to-End Imitation Learning.
  • End-to-End Reinforcement Learning.
• Adversarial Detection

IEEE Journal & Conference

• IEEE Intelligent Vehicle Symposium
• IEEE Intelligent Transportation Systems Conference.

ICCV (Jun. 2023) or BMVC (Jul. 2023)

• Man-in-the-Middle Attack (WHAT)
• Distributed Black-box Attack (BAT)

CVPR (Nov. 2023) or IJCAI (Jan. 2024)

• Adversarial Tracking (Carla)
• Adversarial Patch (Carla)

Week 3

2023/01/23 - 2023/01/29

IEEE Intelligent Vehicle Symposium

IEEE Conference

• Paper Submission Deadline: February 01, 2023.
• Page Limit: At most 8 pages.

Re-structured two manuscripts:

• Adversarial Driving

  • Added more references.
  • Highligted the difference between online and offline attacks.
  • Added experiments on the FPS of the attack (CPU and GPU).

• Adversarial Detection

  • Added more references.
  • Added experiments on the FPS of the attack (GPU).

Week 4

2023/01/30 - 2023/02/05

IEEE Intelligent Vehicle Symposium

Notification of Acceptance March 30, 2023.

• Adversarial Driving
• Adversarial Detection

Reinforcement Learning

[●] Deep SARSA
[●] Deep Q Network

[  ] REINFORCE
[  ] A2C / A3C

[  ] DDPG
[  ] TRPO & PPO

[  ] SAC
[  ] TD3

Week 5

2023/02/06 - 2023/02/12

UWE Bristol

Adversarial Attacks

• Adversarial Driving
• Adversarial Detection

• [22/May] Third-Year Report
• Imitation Learning vs Reinforcement Learning

• A Man-in-the-Middle Attack (WHAT)
• Distributed Black-box Attack (BAT)
  
  
• Adversarial Tracking (Carla)
• Physical Patch (Carla)

Adversarial RL

• Carla Leaderboard
  
  
• Evaluating the Robustness of Deep Reinforcement Learning for Autonomous and Adversarial Policies in a Multi-agent Urban Driving Environment
• Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies

Reinforcement Learning

[●] Deep SARSA
[●] Deep Q Network

[●] REINFORCE
[●] A2C / A3C

[  ] DDPG
[  ] TRPO & PPO

[  ] SAC
[  ] TD3

Week 6

2023/02/13 - 2023/02/19

OpenEuler Meetup

Recorded video will be available on YouTube and Bilibili this Thursday.

MSc Project

• Real-time End-to-End Driving

  • Wei Yu
  • Jinming Wang

• Real-time Vehicle Tracking

  • Bhavana
  • Jagadeesh

Deep Q Network

• Double DQN
• Dueling DQN
• Prioritized Experience Replay
• Noisy DQN
• N-step DQN
• Distributional DQN

Reinforcement Learning

[●] Deep SARSA
[●] Deep Q Network

[●] REINFORCE
[●] A2C / A3C

[  ] DDPG
[  ] TRPO & PPO

[  ] SAC
[  ] TD3

Week 7

2023/02/20 - 2023/02/26

Paper Submission

[IEEE IV] Decision: 30 March

• Adversarial Driving
• Adversarial Detection

[IEEE ITS] Submission: 13 May

• Adversarial Driving
• Adversarial Detection

[BMVC] Submission: 12 May

• Man-in-the-Middle Attack
• Distributed Black Box Attack

[Third-year Report] Submission: 24 May

[ICLR / CVPR] Submission: Sep 2024

• Adversarial Tracking
• Physical Patch

Research Plan

• Jan. Reinforcement Learning
• Feb. Reinforcement Learning
  
  
• Mar. Object Tracking
• Apr. Object Tracking
  
  
• May. WHAT & BAT (Resubmission)

HPC ( JADE 2 Cluster )

Hartree Centre Login

https://um.hartree.stfc.ac.uk/hartree/login.jsp

The Slurm Scheduler

https://docs.jade.ac.uk/en/latest/jade/scheduler/

Allocate a temporary node with 8 GPUS:

srun --gres=gpu:8 --pty bash

Allocate a small partition with 1 GPU:

srun --gres=gpu:1 -p small --pty bash

# 20 CPU Cores / 40 Threads
# 32 GB VRAM / 512 GB RAM

Submit and monitor jobs:

sbatch
sacct
squeue
scancel

Deep Q Network

• Double DQN
• Dueling DQN
• Prioritized Experience Replay
• Noisy DQN
• N-step DQN
• Distributional DQN

Reinforcement Learning

[●] Deep SARSA
[●] Deep Q Network

[●] REINFORCE
[●] A2C / A3C

[●] DDPG
[●] TRPO & PPO

[  ] SAC
[  ] TD3

Week 8

2023/02/27 - 2023/03/05

Research Plan

• Jan. Reinforcement Learning
• Feb. Reinforcement Learning
  
  
• Mar. Object Tracking
• Apr. Object Tracking
  
  
• May. WHAT & BAT (Resubmission)

Reinforcement Learning

Four Courses on Coursera:

Four Courses on Udemy:

Two Books:

• RL Theory: Reinforcement Learning (Sutton & Barto, 2nd Edition)
• RL Practice: Reinforcement Learning: Industrial Applications of Intelligent Agents

Deep Q Network

• Double DQN
• Dueling DQN
• Prioritized Experience Replay
• Noisy DQN
• N-step DQN
• Distributional DQN

Reinforcement Learning

[●] Deep SARSA
[●] Deep Q Network

[●] REINFORCE
[●] A2C / A3C

[●] GAE / NAF / HER

[●] DDPG
[●] TRPO & PPO

[●] SAC
[●] TD3

Week 9

2023/03/06 - 2023/03/12

Research Plan

• Jan. Reinforcement Learning
• Feb. Reinforcement Learning
  
  
• Mar. Object Tracking
  
  
• Apr. Man-in-the-Middle - WHAT (Resubmission)
• May. Distributed Black-Box - BAT (Resubmission)

Adversarial Tracking

[●] Object Tracking (Computer Vision)
[   ] Vehicle Tracking (Autonomous Driving)
[   ] Adversarial Tracking
[   ] Adversarial Patch

Week 10

2023/03/13 - 2023/03/19

Research Plan

• Jan. Reinforcement Learning
• Feb. Reinforcement Learning
  
  
• Mar. Object Tracking
  
  
• Apr. Man-in-the-Middle - WHAT (Resubmission)
• May. Distributed Black-Box - BAT (Resubmission)

Research Papers

Papers Accepted (x3):

• Interpretable Machine Learning for COVID-19, IEEE Trans on AI.
• Adversarial Driving: Attacking End-to-End Autonomous Driving System, IEEE Intelligent Vehicle.
• Adversarial Detection: Attacking Object Detection in Real Time, IEEE Intelligent Vehicle.

Papers to be submitted (x2):

• A Man-in-the-Middle Attack against Object Detection System.
• Distributed Black-box Attack against Image Classification Cloud Services.

Papers to be written (x2):

• Adversarial Tracking: Real-time Adversarial Attacks against Object Tracking.
• Adversarial Patch: Physical Patch in Carla Simulator.

Undetermined (x2):

• Reinforcement Learning.
• Interpretation and Defence.

Adversarial Tracking

[●] Object Tracking (Computer Vision)
[●] Vehicle Tracking (Autonomous Driving)
[   ] Adversarial Tracking
[   ] Adversarial Patch

Week 10

2023/03/20 - 2023/03/26

Research Plan

• Jan. Reinforcement Learning
• Feb. Reinforcement Learning
  
  
• Mar. Object Tracking
  
  
• Apr. Man-in-the-Middle - WHAT (Resubmission)
• May. Distributed Black-Box - BAT (Resubmission)

Adversarial Tracking

[●] Object Tracking (Computer Vision)
[●] Vehicle Tracking (Autonomous Driving)
[●] Adversarial Tracking (Research Proposal)
[   ] Adversarial Patch