Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

38 scrape tripadvisor airline reviews #41

Merged
merged 71 commits into from
Oct 5, 2023
Merged

38 scrape tripadvisor airline reviews #41

merged 71 commits into from
Oct 5, 2023

Conversation

algo7
Copy link
Owner

@algo7 algo7 commented Oct 5, 2023

No description provided.

@algo7
ad airline.js
055f49b
@algo7
add logging level and container_provisioner env var
06999cc
@algo7
add dataSourceAirline var to app.js
fe5543f
@algo7
change data source path in airline scraper init func
34446a3
@algo7
add extractAllReviewPageUrls func template to airline.js
8332352
@algo7
rename input vars forairline page url extraction
3300d18
@algo7
change rev per page
9ad1152
@algo7
fix denominator value for page calculation
1fb4156
@algo7
change review page url forming logic
a7bdbad
@algo7
change url var name
961cb01
@algo7
add scrape function template to airline.js
e00a86c
@algo7
comment title block selection logic
4442acf
@algo7
change comment rating extraction logic
139ea6c
@algo7
remove commented block
4bab453
@algo7
import monthStringToNumber, commentRatingStringToNumber to airlinejs
bd0769d
@algo7
change monthStringToNumber function to account for recent review date
ff77a89
@algo7
add logic to handle diff review date formatting
ff8aef0
@algo7
handle expandble reviews
03876c2
@algo7
refactor clicking action on expandable review
3f959a0
@algo7
fix typo
2bf7b11
@algo7
format return data from the scrape function
05591f2
@algo7
fix var name, comment desc, add start func to airline.js
883271e
@algo7
change func desc for airlineScraperInit
4350ce9
@algo7
add AIRLINE as a scrape mode to the init func
c492d55
@algo7
let user be able to start airline scraper by passing args via env var
6cebf58
@algo7
fix airline scraper rawData env var
b0d42d9
@algo7
fix args pass to resto scraper
8f3cfdc
@algo7
fix scraper name
2f4512b
@algo7
change browser arg type in func desc
e25cbc7
@algo7
remove lang arg being passed to airline scraper
8f0f732
@algo7
add form selection parsing logic
19ba0be
@algo7
add regex pattern for restos and airlines. compile regex and pkg level
8b8a9b5
@algo7
refactor ValidateTripAdvisorURL to account for diff scrape options
d0cca94
@algo7
add backend logic to validate scrape target
0ffea0f
@algo7
refactor get scrape target name func
fba9213
@algo7
add ContainerConfigGenerator
1d0ac31
@algo7
refactor CreateContainer func
6f953fc
@algo7
refactor Scrape function
e870fef
@algo7
refactor ListContainers
e6e4a86
@algo7
refactor
b4a6f54
@algo7
refactor tasks.html
74587ff
@algo7
refac restoScraperInit to take in env var
7d98785
@algo7
formatting
038d400
@algo7
fix ValidateTripAdvisorURL switch statement
188d8b5
@algo7
refactor ListContainers function
94c78b7
@algo7
refactor list container func and remove running tasks struct
dad1b68
@algo7
refactor getRunningTasks func
2a10e29
@algo7
remove debug logging
ab8ddb3
@algo7
refactor logging logic for resto scraper
06e061a
@algo7
add exit logic to hotel and resto scraper init func
6c0c7a2
@algo7
add logging
c480dbe
@algo7
remove CountRunningContainer func
eb84183
@algo7
fix resto logging logic
2bd6a9e
@algo7
change invalid url msg
d994f6c
@algo7
change main.html url bar text
c75af07
@algo7
add IP ban warning
5fa4b18
@algo7 algo7 self-assigned this Oct 5, 2023
@algo7 algo7 linked an issue Oct 5, 2023 that may be closed by this pull request
Scrape Tripadvisor airline reviews #38
Closed
@algo7 algo7 merged commit 9aebdd0 into main Oct 5, 2023
2 checks passed
@algo7 algo7 deleted the 38-scrape-tripadvisor-airline-reviews branch October 5, 2023 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@algo7 algo7

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Scrape Tripadvisor airline reviews
1 participant
@algo7