Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V7 beta #12

Merged
merged 2 commits into from
Sep 19, 2023
Merged

V7 beta #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
88cec3c
update README
otkrickey Sep 19, 2023
5895b03
試験的な機能を削除
otkrickey Sep 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 1 addition & 145 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,145 +1 @@
# letus-scraper

```bash
/home/otkrickey/dev/letus-scraper/.venv/bin/python /home/otkrickey/dev/letus-scraper/main.py -l info -e 1
/home/otkrickey/dev/letus-scraper/.venv/bin/python /home/otkrickey/dev/letus-scraper/main.py -s checkAccount -d 601235188571176961 -l info -e 1098772108579962950:1099020068198809610
```

## TODO

### Task on MongoDB

- default

- Discord: (user_id, guild_id, category_id, channel_id, message_id)

- Account

- Register Account
- TUS: (student_id, encrypted_password)
- Update Account Info
- TUS: (student_id, encrypted_password)
- Unregister Account
- TUS: (student_id, encrypted_password)
- Check Account Info
- TUS: (student_id)

- Page
- Register Page
- LETUS: (page_id)
- TUS: (student_id) # Optional: If not provided, use the student_id from the account
- Fetch Page
- LETUS: (page_id)
- TUS: (student_id) # Optional: If not provided, use the student_id from the account
- Unregister Page
- LETUS: (page_id)
- TUS: (student_id) # Optional: If not provided, use the student_id from the account

### Flow

```mermaid
graph TD

subgraph ServiceManager
SM{ServiceManager}

S1[CheckAccount]
S2[RegisterAccount]
S3[CheckContent]

SM --> S1
SM --> S3
SM --> S2
end

subgraph Middleware
M1[fetchAccount]
M2[loginAccount]
M3[registerAccount]
M4[uploadAccount]
M5[fetchPage]
M6[fetchContent]
end

subgraph Manager
subgraph AccountManager

AM1[check_account_info]
AM2[fetch_account_info]
AM3[register_account]
AM4[update_account]

AM1 --> AM2
AM3 --> AM4
end
end

subgraph Model
subgraph LetusAccount
LA1[discord_id]
subgraph LetusPage
LP1[discord_id]
subgraph LetusContent
LC1[discord_id]
LC2[DB_args]
end
end
end

subgraph LetusSession
LS1[login]
LS2[register]
LS3[__login_letus]
LS4[__login_microsoft]
LS5[load_cookie]

LS1 --> LS2 --> LS3 --> LS4 --> LS5
end
end

S1 --> M1
S2 --> M3
S3 --> M5

M1 --> AM1
M2 --> LS1
M3 --> LS2
M4 --> AM3
M5 --> M6

LS5 --> M4

AM2 --> M2
LetusPage --> LetusContent
```

```json
[
{
"task": "account:status",
"discord_id": "601235188571176961"
},
{
"task": "account:login",
"multi_id": "601235188571176961",
},
{
"task": "account:register",
"student_id": "7322023",
"discord_id": "601235188571176961",
"Letus": {
"encrypted_password": "password",
"email": "email"
},
"Discord": {
"username": "otkrickey",
"discriminator": "otkrickey"
}
}
]





```
# letusc: `v7.0.0`
3 changes: 1 addition & 2 deletions letusc/models/account.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
from dataclasses import dataclass, field

from letusc.logger import L

from ..db import DBManager
from ..logger import L
from .base import BaseDatabase, BaseModel, attrs, from_api_attrs, to_api_attrs, types
from .cookie import Cookie
from .discord import DiscordUser, DiscordUserAny, DiscordUserBase
Expand Down
12 changes: 1 addition & 11 deletions letusc/models/page.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
from ..db import DBManager
from ..logger import L
from ..session import SessionManager
from ..util import env_bool
from .base import (
BaseDatabase,
BaseModel,
Expand Down Expand Up @@ -150,13 +149,6 @@ async def get(self, cookie: Cookie) -> bs4.BeautifulSoup:
html = await response.text()
return BeautifulSoup(html, "html.parser")

async def get_local(self) -> bs4.BeautifulSoup:
_l = self._l.gm("get_local")
_l.info(f"Requesting page: {self.url}")
with open(f"{self.page_id}.example.html", "r") as f:
html = f.read()
return BeautifulSoup(html, "html.parser")


@dataclass
class Page(PageBase):
Expand Down Expand Up @@ -233,9 +225,7 @@ async def parse(cls, code: PageCode, cookie: Cookie) -> "NewPage":
page = NewCoursePage(code.code)
case _:
raise ValueError(_l.c("UnknownPageType"))
soup = (
await page.get_local() if env_bool("USE_LOCAL") else await page.get(cookie)
)
soup = await page.get(cookie)
await page._parse(soup)
return page

Expand Down