Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

Refactoring proxy lifetime to only shutdown when proxy is out-of-date. #839

Merged
120 commits merged into from
May 20, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
120 commits
Select commit Hold shift + click to select a range
c2584fc
Refactoring proxy lifetime to only shutdown when proxy is out-of-date.
nharper285 Apr 29, 2021
bdc93ab
Running isort and black for formatting. Fixing typo.
nharper285 Apr 29, 2021
b360278
Editing based of optional timestamp for proxy.
nharper285 Apr 29, 2021
f6fa247
isort and black.
nharper285 Apr 29, 2021
b9b2572
Potential fix for time issues.
nharper285 May 3, 2021
e238542
Forgot other .now() changes.
nharper285 May 3, 2021
8452b81
Running isort and black.
nharper285 May 3, 2021
509eb2e
Correcting datetime to datetime.aware
nharper285 May 3, 2021
4ce51e5
Changing to 30 min lifetime to test.
nharper285 May 3, 2021
8389d35
Isort and black.
nharper285 May 3, 2021
17de463
Editing comments.
nharper285 May 4, 2021
aa590f3
Creating new 'created_timestamp' field to track create time.
nharper285 May 4, 2021
2effb04
Changing how created_timestamp is initialized.
nharper285 May 4, 2021
abc00df
Working to make proxy rollover more fluid.
nharper285 May 4, 2021
da279a0
Typo.
nharper285 May 4, 2021
fe8af47
Fixing imports.
nharper285 May 4, 2021
c1ab4bb
Removing trailing white space.
nharper285 May 4, 2021
9913bc1
Resolving type error.
nharper285 May 4, 2021
c635bd7
Type issues (cont)
nharper285 May 4, 2021
349cd80
Fixing change in proxy.py is_alive()
nharper285 May 5, 2021
41ee0f9
Updating proxy with new identifying field. Updated save config with n…
nharper285 May 6, 2021
fb573a8
Proxy searches on outdated field and new id.
nharper285 May 6, 2021
b011111
Fixing search() call.
nharper285 May 6, 2021
a17026a
Specifying number of search results.
nharper285 May 6, 2021
62ad0ff
isort and black
nharper285 May 6, 2021
c5f98cd
Fixin list entry.
nharper285 May 6, 2021
8f8b75d
Pulling first entry from proxy list.
nharper285 May 6, 2021
07a6c26
isort and black.
nharper285 May 6, 2021
5e0d999
Index issue.
nharper285 May 7, 2021
5ed4b4b
Typo.
nharper285 May 7, 2021
d7a602f
Typo.
nharper285 May 7, 2021
7eb4742
Fixing extension save.
nharper285 May 7, 2021
c452b11
Adding proxy_id to event fields.
nharper285 May 7, 2021
67b1ec1
Fixing generate docs.
nharper285 May 7, 2021
8c7a127
Resolving event issues.
nharper285 May 7, 2021
4181690
Fixing webhook events.
nharper285 May 7, 2021
5789793
Generate Docs.
nharper285 May 8, 2021
f06957f
Merge branch 'main' into user/noharper/proxy-refactor
bmc-msft May 10, 2021
f80ab84
CRLF to LF for webhook_events.py.
nharper285 May 10, 2021
f1b118a
Merge branch 'user/noharper/proxy-refactor' of https://github.com/nha…
nharper285 May 10, 2021
93e80b6
Merge branch 'main' into user/noharper/proxy-refactor
bmc-msft May 10, 2021
a148a95
Rerunning.
nharper285 May 10, 2021
ecc9278
Merge branch 'main' into user/noharper/proxy-refactor
bmc-msft May 11, 2021
6ccb769
Native to ascii.
nharper285 May 11, 2021
12283ee
Merge branch 'user/noharper/proxy-refactor' of https://github.com/nha…
nharper285 May 11, 2021
3e93a0b
Type.
nharper285 May 11, 2021
c368ec9
Uploading webhook_events.md
nharper285 May 11, 2021
988cd45
Reworking webhook_events.md.
nharper285 May 11, 2021
7ac914b
Updating generate_docs to output webhook_events.md file with ascii/LF…
nharper285 May 11, 2021
1b2f4f8
isort and black for generate-docs.
nharper285 May 11, 2021
d2c3682
Allowing path to be command line argument.
nharper285 May 11, 2021
33f625a
Fixing file type.
nharper285 May 11, 2021
81a311c
Static variable for file directory.
nharper285 May 11, 2021
4d68273
Isort and black for generate_docs
nharper285 May 11, 2021
89cf455
Removing IO statements.
nharper285 May 11, 2021
6b898d4
Replacing Webhook events
nharper285 May 11, 2021
b155178
Fixing webhood_events.md and generate-docs.
nharper285 May 11, 2021
dd67da1
isort and black
nharper285 May 11, 2021
f114f5e
Removing old imports.
nharper285 May 11, 2021
4f01c83
Formatting.
nharper285 May 11, 2021
96bf458
Trying old version of script.
nharper285 May 12, 2021
5fb226c
Fixing generate docs and shell script.
nharper285 May 12, 2021
778c1c4
Trying to debug.
nharper285 May 12, 2021
cb2af93
Isort and black.
nharper285 May 12, 2021
c9ebf8d
Edits to agent and ProxyForward.
nharper285 May 12, 2021
e18098e
Reformatting.
nharper285 May 12, 2021
2679ebc
Reformatting generate-docs.py
nharper285 May 12, 2021
16a64b8
Fixing proxy_id references.
nharper285 May 12, 2021
0dc5f78
Fixing imports.
nharper285 May 12, 2021
ad146c7
Working on proxy init.py
nharper285 May 12, 2021
05c07ba
Fixing init.py in proxy.
nharper285 May 12, 2021
2f51d3b
Trying to remove proxy_id from ProxyForward.
nharper285 May 12, 2021
42f17a3
Removing proxy_id reference from ProxyForward.
nharper285 May 12, 2021
17a6feb
Removing proxy_id reference.
nharper285 May 12, 2021
794d682
Adding proxy_id to heartbeat.
nharper285 May 12, 2021
b11fc96
Adding proxy_id field to heartbeat.
nharper285 May 12, 2021
7e1056a
Adding is_used() chat to get_or_create before stopping VM.
nharper285 May 13, 2021
9c609ee
black reformatting.
nharper285 May 13, 2021
804bf1b
Fixing proxy.save calls.
nharper285 May 13, 2021
17b2f6a
Rerunning due to agent error.
nharper285 May 13, 2021
7ecc3ee
Merge branch 'main' into user/noharper/proxy-refactor
bmc-msft May 13, 2021
9ba96c6
New file for running lint tools against api-service.
nharper285 May 13, 2021
22d9b3b
Rerunning.
nharper285 May 13, 2021
0b1738a
Fixing timer_daily to check if existing proxies for region. Working t…
nharper285 May 13, 2021
e5a1734
Fixing region reference.
nharper285 May 14, 2021
9a9415d
Trying to trigger merge.
nharper285 May 14, 2021
da2069c
Merging changes to onefuzztypes and generate-docs with main.
nharper285 May 14, 2021
bbd8a36
Pulling merge in from upstream/main to fix merge conflicts.
nharper285 May 14, 2021
1e4e4a0
Reformatting generate_docs.
nharper285 May 14, 2021
bfee9b2
Making proxy_id optional.
nharper285 May 14, 2021
6349946
Merge remote-tracking branch 'upstream/main' into user/noharper/proxy…
nharper285 May 14, 2021
9bbf3df
Adding save for Proxy Forwards.
nharper285 May 14, 2021
28a8ab1
Fixing call to is_used
nharper285 May 17, 2021
7e37972
Updating timer_daily with separate conditions.
nharper285 May 17, 2021
ce80603
Merging.
nharper285 May 17, 2021
bab9d3e
Removing import.
nharper285 May 17, 2021
84281da
Changing minutes to 5 minutes for testing.
nharper285 May 17, 2021
1feebb5
Final Draft: Changing Proxy lifetime back to 7 days.
nharper285 May 17, 2021
ac9bf25
Fixing line endings.
nharper285 May 18, 2021
07d5b2c
Changes based of initial review.
nharper285 May 18, 2021
e5b1f88
Removing outdated line.
nharper285 May 18, 2021
c116d86
Updating webhook_events.md
nharper285 May 18, 2021
02c1ae7
Refactoring timer_daily proxy check.
nharper285 May 18, 2021
9b6efb9
Changing lifespan back to minutes for testing.
nharper285 May 18, 2021
a72308f
Fixing lambda.
nharper285 May 18, 2021
12cc824
Apply suggestions from code review
bmc-msft May 19, 2021
c21b241
Apply suggestions from code review
bmc-msft May 19, 2021
5f5bdcc
Apply suggestions from code review
bmc-msft May 19, 2021
2df8272
Apply suggestions from code review
bmc-msft May 19, 2021
94c703a
Apply suggestions from code review
bmc-msft May 19, 2021
773c1c8
Apply suggestions from code review
bmc-msft May 19, 2021
eaa7a07
Apply suggestions from code review
bmc-msft May 19, 2021
e25df0f
Apply suggestions from code review
bmc-msft May 19, 2021
e8f196a
Update src/api-service/__app__/onefuzzlib/proxy.py
bmc-msft May 19, 2021
2000b1e
Merge branch 'main' into user/noharper/proxy-refactor
bmc-msft May 19, 2021
271f125
Update src/api-service/__app__/onefuzzlib/proxy.py
bmc-msft May 19, 2021
81aca12
Apply suggestions from code review
bmc-msft May 19, 2021
1fee542
Changing 'minutes' back to 'days.'
nharper285 May 19, 2021
7913c3f
Apply suggestions from code review
bmc-msft May 19, 2021
db447fb
Update src/api-service/__app__/onefuzzlib/proxy.py
bmc-msft May 19, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions docs/webhook_events.md
Original file line number Diff line number Diff line change
Expand Up @@ -1255,6 +1255,7 @@ Each event will be submitted via HTTP POST to the user provided URL.

```json
{
"proxy_id": "00000000-0000-0000-0000-000000000000",
"region": "eastus"
}
```
Expand All @@ -1264,6 +1265,11 @@ Each event will be submitted via HTTP POST to the user provided URL.
```json
{
"properties": {
"proxy_id": {
"format": "uuid",
"title": "Proxy Id",
"type": "string"
},
"region": {
"title": "Region",
"type": "string"
Expand All @@ -1283,6 +1289,7 @@ Each event will be submitted via HTTP POST to the user provided URL.

```json
{
"proxy_id": "00000000-0000-0000-0000-000000000000",
"region": "eastus"
}
```
Expand All @@ -1292,6 +1299,11 @@ Each event will be submitted via HTTP POST to the user provided URL.
```json
{
"properties": {
"proxy_id": {
"format": "uuid",
"title": "Proxy Id",
"type": "string"
},
"region": {
"title": "Region",
"type": "string"
Expand All @@ -1317,6 +1329,7 @@ Each event will be submitted via HTTP POST to the user provided URL.
"example error message"
]
},
"proxy_id": "00000000-0000-0000-0000-000000000000",
"region": "eastus"
}
```
Expand Down Expand Up @@ -1379,6 +1392,11 @@ Each event will be submitted via HTTP POST to the user provided URL.
"error": {
"$ref": "#/definitions/Error"
},
"proxy_id": {
"format": "uuid",
"title": "Proxy Id",
"type": "string"
},
"region": {
"title": "Region",
"type": "string"
Expand Down Expand Up @@ -4889,6 +4907,11 @@ Each event will be submitted via HTTP POST to the user provided URL.
},
"EventProxyCreated": {
"properties": {
"proxy_id": {
"format": "uuid",
"title": "Proxy Id",
"type": "string"
},
"region": {
"title": "Region",
"type": "string"
Expand All @@ -4902,6 +4925,11 @@ Each event will be submitted via HTTP POST to the user provided URL.
},
"EventProxyDeleted": {
"properties": {
"proxy_id": {
"format": "uuid",
"title": "Proxy Id",
"type": "string"
},
"region": {
"title": "Region",
"type": "string"
Expand All @@ -4918,6 +4946,11 @@ Each event will be submitted via HTTP POST to the user provided URL.
"error": {
"$ref": "#/definitions/Error"
},
"proxy_id": {
"format": "uuid",
"title": "Proxy Id",
"type": "string"
},
"region": {
"title": "Region",
"type": "string"
Expand Down
4 changes: 2 additions & 2 deletions src/api-service/__app__/onefuzzlib/extension.py
Original file line number Diff line number Diff line change
Expand Up @@ -341,11 +341,11 @@ def repro_extensions(
return extensions


def proxy_manager_extensions(region: Region) -> List[Extension]:
def proxy_manager_extensions(region: Region, proxy_id: UUID) -> List[Extension]:
urls = [
get_file_sas_url(
Container("proxy-configs"),
"%s/config.json" % region,
"%s/%s/config.json" % (region, proxy_id),
StorageType.config,
read=True,
),
Expand Down
75 changes: 52 additions & 23 deletions src/api-service/__app__/onefuzzlib/proxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import logging
import os
from typing import List, Optional, Tuple
from uuid import UUID, uuid4

from azure.mgmt.compute.models import VirtualMachine
from onefuzztypes.enums import ErrorCode, VmState
Expand Down Expand Up @@ -37,27 +38,33 @@
PROXY_SKU = "Standard_B2s"
PROXY_IMAGE = "Canonical:UbuntuServer:18.04-LTS:latest"
PROXY_LOG_PREFIX = "scaleset-proxy: "
PROXY_LIFESPAN = datetime.timedelta(days=7)


# This isn't intended to ever be shared to the client, hence not being in
# onefuzztypes
class Proxy(ORMMixin):
timestamp: Optional[datetime.datetime] = Field(alias="Timestamp")
created_timestamp: datetime.datetime = Field(
default_factory=datetime.datetime.utcnow
)
proxy_id: UUID = Field(default_factory=uuid4)
region: Region
state: VmState = Field(default=VmState.init)
auth: Authentication = Field(default_factory=build_auth)
ip: Optional[str]
error: Optional[Error]
version: str = Field(default=__version__)
heartbeat: Optional[ProxyHeartbeat]
outdated: bool = Field(default=False)

@classmethod
def key_fields(cls) -> Tuple[str, Optional[str]]:
return ("region", None)
return ("region", "proxy_id")

def get_vm(self) -> VM:
vm = VM(
name="proxy-%s" % self.region,
name="proxy-%s-%s" % (self.region, self.proxy_id),
region=self.region,
sku=PROXY_SKU,
image=PROXY_IMAGE,
Expand Down Expand Up @@ -104,7 +111,9 @@ def set_failed(self, error: Error) -> None:
return

logging.error(PROXY_LOG_PREFIX + "vm failed: %s - %s", self.region, error)
send_event(EventProxyFailed(region=self.region, error=error))
send_event(
EventProxyFailed(region=self.region, proxy_id=self.proxy_id, error=error)
)
self.error = error
self.state = VmState.stopping
self.save()
Expand All @@ -131,7 +140,7 @@ def extensions_launch(self) -> None:
return
self.ip = ip

extensions = proxy_manager_extensions(self.region)
extensions = proxy_manager_extensions(self.region, self.proxy_id)
result = vm.add_extensions(extensions)
if isinstance(result, Error):
self.set_failed(result)
Expand All @@ -154,6 +163,29 @@ def stopped(self) -> None:
logging.info(PROXY_LOG_PREFIX + "removing proxy: %s", self.region)
self.delete()

def is_outdated(self) -> bool:
nharper285 marked this conversation as resolved.
Show resolved Hide resolved
if self.version != __version__:
logging.info(
PROXY_LOG_PREFIX + "mismatch version: proxy:%s service:%s state:%s",
self.version,
__version__,
self.state,
)
return True
if self.created_timestamp is not None:
proxy_timestamp = self.created_timestamp
if proxy_timestamp < (
datetime.datetime.now(tz=datetime.timezone.utc) - PROXY_LIFESPAN
):
logging.info(
PROXY_LOG_PREFIX
+ "proxy older than 7 days:proxy-created:%s state:%s",
self.created_timestamp,
self.state,
)
return True
return False

def is_used(self) -> bool:
if len(self.get_forwards()) == 0:
logging.info(PROXY_LOG_PREFIX + "no forwards: %s", self.region)
Expand Down Expand Up @@ -194,7 +226,9 @@ def is_alive(self) -> bool:

def get_forwards(self) -> List[Forward]:
forwards: List[Forward] = []
for entry in ProxyForward.search_forward(region=self.region):
for entry in ProxyForward.search_forward(
region=self.region, proxy_id=self.proxy_id
):
if entry.endtime < datetime.datetime.now(tz=datetime.timezone.utc):
entry.delete()
else:
Expand All @@ -212,7 +246,7 @@ def save_proxy_config(self) -> None:
proxy_config = ProxyConfig(
url=get_file_sas_url(
Container("proxy-configs"),
"%s/config.json" % self.region,
"%s/%s/config.json" % (self.region, self.proxy_id),
StorageType.config,
read=True,
),
Expand All @@ -223,14 +257,15 @@ def save_proxy_config(self) -> None:
),
forwards=forwards,
region=self.region,
proxy_id=self.proxy_id,
instance_telemetry_key=os.environ.get("APPINSIGHTS_INSTRUMENTATIONKEY"),
microsoft_telemetry_key=os.environ.get("ONEFUZZ_TELEMETRY"),
instance_id=get_instance_id(),
)

save_blob(
Container("proxy-configs"),
"%s/config.json" % self.region,
"%s/%s/config.json" % (self.region, self.proxy_id),
proxy_config.json(),
StorageType.config,
)
Expand All @@ -244,28 +279,22 @@ def search_states(cls, *, states: Optional[List[VmState]] = None) -> List["Proxy

@classmethod
def get_or_create(cls, region: Region) -> Optional["Proxy"]:
proxy = Proxy.get(region)
if proxy is not None:
if proxy.version != __version__:
logging.info(
PROXY_LOG_PREFIX + "mismatch version: proxy:%s service:%s state:%s",
proxy.version,
__version__,
proxy.state,
)
if proxy.state != VmState.stopping:
# If the proxy is out-of-date, delete and re-create it
proxy.state = VmState.stopping
proxy.save()
return None
proxy_list = Proxy.search(query={"region": [region], "outdated": [False]})
for proxy in proxy_list:
if proxy.is_outdated():
proxy.outdated = True
proxy.save()
continue
if proxy.state not in VmState.available():
continue
return proxy

logging.info(PROXY_LOG_PREFIX + "creating proxy: region:%s", region)
proxy = Proxy(region=region)
proxy.save()
send_event(EventProxyCreated(region=region))
send_event(EventProxyCreated(region=region, proxy_id=proxy.proxy_id))
return proxy

def delete(self) -> None:
super().delete()
send_event(EventProxyDeleted(region=self.region))
send_event(EventProxyDeleted(region=self.region, proxy_id=self.proxy_id))
11 changes: 10 additions & 1 deletion src/api-service/__app__/onefuzzlib/proxy_forward.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ class ProxyForward(ORMMixin):
port: int
scaleset_id: UUID
machine_id: UUID
proxy_id: Optional[UUID]
dst_ip: str
dst_port: int
endtime: datetime.datetime = Field(default_factory=datetime.datetime.utcnow)
Expand Down Expand Up @@ -93,11 +94,15 @@ def remove_forward(
cls,
scaleset_id: UUID,
*,
proxy_id: Optional[UUID] = None,
machine_id: Optional[UUID] = None,
dst_port: Optional[int] = None,
) -> List[Region]:
entries = cls.search_forward(
scaleset_id=scaleset_id, machine_id=machine_id, dst_port=dst_port
scaleset_id=scaleset_id,
machine_id=machine_id,
proxy_id=proxy_id,
dst_port=dst_port,
)
regions = set()
for entry in entries:
Expand All @@ -112,6 +117,7 @@ def search_forward(
scaleset_id: Optional[UUID] = None,
region: Optional[Region] = None,
machine_id: Optional[UUID] = None,
proxy_id: Optional[UUID] = None,
dst_port: Optional[int] = None,
) -> List["ProxyForward"]:

Expand All @@ -125,6 +131,9 @@ def search_forward(
if machine_id is not None:
query["machine_id"] = [machine_id]

if proxy_id is not None:
query["proxy_id"] = [proxy_id]

if dst_port is not None:
query["dst_port"] = [dst_port]

Expand Down
2 changes: 0 additions & 2 deletions src/api-service/__app__/onefuzzlib/tasks/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@
from ..azure.storage import StorageType
from ..events import send_event
from ..orm import MappingIntStrAny, ORMMixin, QueryFilter
from ..proxy_forward import ProxyForward
from ..workers.nodes import Node, NodeTasks
from ..workers.pools import Pool
from ..workers.scalesets import Scaleset
Expand Down Expand Up @@ -125,7 +124,6 @@ def init(self) -> None:

def stopping(self) -> None:
logging.info("stopping task: %s:%s", self.job_id, self.task_id)
ProxyForward.remove_forward(self.task_id)
Node.stop_task(self.task_id)
if not NodeTasks.get_nodes_by_task_id(self.task_id):
self.stopped()
Expand Down
2 changes: 2 additions & 0 deletions src/api-service/__app__/proxy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ def post(req: func.HttpRequest) -> func.HttpResponse:

proxy = Proxy.get_or_create(scaleset.region)
if proxy:
forward.proxy_id = proxy.proxy_id
forward.save()
proxy.save_proxy_config()
return ok(get_result(forward, proxy))

Expand Down
2 changes: 1 addition & 1 deletion src/api-service/__app__/queue_proxy_update/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ def main(msg: func.QueueMessage, dashboard: func.Out[str]) -> None:
logging.info(PROXY_LOG_PREFIX + "heartbeat: %s", body)
raw = json.loads(body)
heartbeat = ProxyHeartbeat.parse_obj(raw)
proxy = Proxy.get(heartbeat.region)
proxy = Proxy.get(heartbeat.region, heartbeat.proxy_id)
if proxy is None:
logging.warning(
PROXY_LOG_PREFIX + "received heartbeat for missing proxy: %s", body
Expand Down
Loading