Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/api scraper #176

Merged
merged 70 commits into from
Jan 23, 2024
Merged

Feature/api scraper #176

merged 70 commits into from
Jan 23, 2024

Commits on Jan 22, 2024

  1. create the new scraper project structure

    @algo7
    algo7 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    6da1e10 View commit details
    Browse the repository at this point in the history

  2. rename old scraper folder to scraperjs

    @algo7
    algo7 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    6cf304f View commit details
    Browse the repository at this point in the history

  3. add Request struct to the TripAdvisor package

    @algo7
    algo7 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    9ec0ef7 View commit details
    Browse the repository at this point in the history

  4. Add Response struct to represent TripAdvisor response body

    @algo7
    algo7 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    848cfe9 View commit details
    Browse the repository at this point in the history

  5. Basic call for tripadvisor api

    @algo7
    algo7 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    6ccc15c View commit details
    Browse the repository at this point in the history

  6. Update X-Requested-By header value in tripadvisor.go

    @algo7
    algo7 committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    36906ee View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. Refactor TripAdvisor scraper code

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    bce55e3 View commit details
    Browse the repository at this point in the history

  2. add CreateReviewsCSV function

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    ce28981 View commit details
    Browse the repository at this point in the history

  3. Add CSV writing functionality and refactor code

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    e14234f View commit details
    Browse the repository at this point in the history

  4. merge files module into tripadvisor module

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    221828a View commit details
    Browse the repository at this point in the history

  5. Refactor file handling in TripAdvisor package

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    0ef803c View commit details
    Browse the repository at this point in the history

  6. Update TripAdvisor query function signature

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    994e491 View commit details
    Browse the repository at this point in the history

  7. Add pre-registered query IDs and update struct comments

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    a099298 View commit details
    Browse the repository at this point in the history

  8. Refactor Query function to accept queryID parameter

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    9105607 View commit details
    Browse the repository at this point in the history

  9. Refactor Query function to MakeRequest function in tripadvisor.go

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    697e813 View commit details
    Browse the repository at this point in the history

  10. Add error handling for response status code

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    10bcf3e View commit details
    Browse the repository at this point in the history

  11. Handle rate limiting in MakeRequest function

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    4ef86be View commit details
    Browse the repository at this point in the history

  12. Refactor Response struct and add FetchReviewCount function

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    12b7770 View commit details
    Browse the repository at this point in the history

  13. Update PrefsCacheKey to use locationID dynamically

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    d58f65b View commit details
    Browse the repository at this point in the history

  14. Refactor FetchReviewCount to support different query types

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    5535a59 View commit details
    Browse the repository at this point in the history

  15. comment out unused code

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    7e64c03 View commit details
    Browse the repository at this point in the history

  16. comments

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    39d7d15 View commit details
    Browse the repository at this point in the history

  17. Add ReviewLimit constant and CalculateIterations function

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    7db8c7c View commit details
    Browse the repository at this point in the history

  18. add offsetcalculation function

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    dface03 View commit details
    Browse the repository at this point in the history

  19. remove the csv related functions. implement delay for scraping

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    159fbe5 View commit details
    Browse the repository at this point in the history

  20. Refactor Review struct in models.go

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    252ac04 View commit details
    Browse the repository at this point in the history

  21. Refactor TripAdvisor scraper to improve code readability and performance

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    07ad261 View commit details
    Browse the repository at this point in the history

  22. Refactor main function to use environment variables for query type an… 

    …d location ID
    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    39f4baf View commit details
    Browse the repository at this point in the history

  23. Add initialization function to check environment variables

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    0282352 View commit details
    Browse the repository at this point in the history

  24. comments

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    6f40e72 View commit details
    Browse the repository at this point in the history

  25. formatting

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    5c05560 View commit details
    Browse the repository at this point in the history

  26. comments

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    dfe2df5 View commit details
    Browse the repository at this point in the history

  27. move endpoint url to models.go

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    eaa6b67 View commit details
    Browse the repository at this point in the history

  28. add proxy package

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    c50f854 View commit details
    Browse the repository at this point in the history

  29. Add GetHTTPClientWithProxy function to proxy package

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    d9c086c View commit details
    Browse the repository at this point in the history

  30. Add proxy support and handle no reviews found

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    2266edf View commit details
    Browse the repository at this point in the history

  31. Fix log message format

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    f461618 View commit details
    Browse the repository at this point in the history

  32. change proxy implementation.

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    047ab73 View commit details
    Browse the repository at this point in the history

  33. Add proxy and IP logging

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    9324b50 View commit details
    Browse the repository at this point in the history

  34. Add proxy server operational check

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    b26b189 View commit details
    Browse the repository at this point in the history

  35. log ip addr only iof using proxy

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    6e49c69 View commit details
    Browse the repository at this point in the history

  36. Add error handling for location ID input

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    2d1bcc2 View commit details
    Browse the repository at this point in the history

  37. set timeout for HTTP client. use writeAll instead of write to write t… 

    …o the csv
    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    24ba15b View commit details
    Browse the repository at this point in the history

  38. Refactor CSV writing logic

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    020b96e View commit details
    Browse the repository at this point in the history

  39. enable ip check again

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    f263edb View commit details
    Browse the repository at this point in the history

  40. Refactor GetHTTPClientWithProxy to use proxy URL instead of custom di… 

    …aler
    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    af86a84 View commit details
    Browse the repository at this point in the history

  41. Update proxy configuration in scraper package

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    088545e View commit details
    Browse the repository at this point in the history

  42. comments

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    9088ef8 View commit details
    Browse the repository at this point in the history

  43. formatting

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    8725b39 View commit details
    Browse the repository at this point in the history

  44. formatting

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    e295529 View commit details
    Browse the repository at this point in the history

  45. comments

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    7ced905 View commit details
    Browse the repository at this point in the history

  46. remove the defer call from csv. as we are using writeAll

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    bc209c0 View commit details
    Browse the repository at this point in the history

  47. fix misleading comments

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    b6d37c5 View commit details
    Browse the repository at this point in the history

  48. comments

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    7134d27 View commit details
    Browse the repository at this point in the history

  49. Add URL validation and parsing functions

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    621577c View commit details
    Browse the repository at this point in the history

  50. Refactor ValidateURL function to GetURLType

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    4b13d3a View commit details
    Browse the repository at this point in the history

  51. Refactor location ID parsing in TripAdvisor scraper

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    44925df View commit details
    Browse the repository at this point in the history

  52. Refactor scraper main function to take in location url instead of loc… 

    …ation id and query type
    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    db93ccf View commit details
    Browse the repository at this point in the history

  53. Refactor ContainerConfigGenerator function to simplify and improve re… 

    …adability
    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    57cc91d View commit details
    Browse the repository at this point in the history

  54. Refactor code to remove unused function

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    80d0ba7 View commit details
    Browse the repository at this point in the history

  55. Refactor code to get location name from URL

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    a0c3b79 View commit details
    Browse the repository at this point in the history

  56. Refactor URL validation in utils.go

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    560ac64 View commit details
    Browse the repository at this point in the history

  57. Refactor container provisioner code

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    cb8cbfb View commit details
    Browse the repository at this point in the history

  58. Add logging statements to the scraper

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    9c17c77 View commit details
    Browse the repository at this point in the history

  59. add dockerfile for the scraper

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    6d1b380 View commit details
    Browse the repository at this point in the history

  60. add scraper makefile

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    dcfd919 View commit details
    Browse the repository at this point in the history

  61. update dockerfile platform. update scraper ci

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    4fa03b3 View commit details
    Browse the repository at this point in the history

  62. add readme and deprecation warning to the old scraper js

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    9fd8b83 View commit details
    Browse the repository at this point in the history

  63. update readme

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    51092be View commit details
    Browse the repository at this point in the history

  64. update scraper readme

    @algo7
    algo7 committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    3b8243b View commit details
    Browse the repository at this point in the history