Automate Article Collection and README Updates from Blog Source

[sikehish]

This pull request introduces a script for fetching articles from a specified URL(https://blog.dagworks.io/archive), processing the data, and updating the README file with the fetched articles. It addresses issue #1046.

Changes

• Added a script to fetch articles using Selenium and BeautifulSoup.
• Implemented a function to get a cutoff date from user input.
• Included functionality to update a README file with newly fetched articles.
• Added error handling for date input.
• Included a requirements file for dependencies.

How I tested this

• Verified that the script correctly fetches articles from the specified URL and logs the output.
• Tested the cutoff date input to ensure it correctly formats and validates the date.
• Confirmed that the README file updates with the new articles correctly and maintains the existing content.

Checklist

• PR has an informative and human-readable title (this will be pulled into the release notes)
• Changes are limited to a single goal (no scope creep)
• Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future TODOs are captured in comments
• Project documentation has been updated if adding/changing functionality.

[ellipsis-dev]

❌ Changes requested. Reviewed everything up to 34f80c6 in 18 seconds

More details

• Looked at 166 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 6 drafted comments based on config settings.

1. scripts/update_blogs_in_learning_resources.py:20

• Draft comment:
  Consider using an HTTP library like requests instead of Selenium for fetching articles if JavaScript execution is not required. This can improve performance and reduce resource usage.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The script uses Selenium to scrape a webpage, which can be resource-intensive and slow. It would be more efficient to use an HTTP library like requests if the page content can be accessed without JavaScript execution.

2. scripts/update_blogs_in_learning_resources.py:110

• Draft comment:
  Consider creating a backup of the README file before modifying it to prevent data loss in case of an error.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The script directly modifies the README file without creating a backup. It's a good practice to create a backup before making changes to important files.

3. scripts/update_blogs_in_learning_resources.py:45

• Draft comment:
  Avoid using hardcoded blog links if they can be fetched dynamically. This ensures the script always has the most up-to-date information.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The script uses hardcoded blog links which might not be necessary if the links can be fetched dynamically. This could lead to outdated information if the hardcoded links change.

4. scripts/update_blogs_in_learning_resources.py:20

• Draft comment:
  The fetch_articles function is handling both fetching and processing articles. Consider separating the fetching logic from the processing logic to adhere to the Single Responsibility Principle.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable:
  The comment is suggesting a refactor to improve adherence to the Single Responsibility Principle, which is a good practice. The function indeed handles two distinct tasks: fetching data and processing it. Separating these tasks could make the code cleaner and easier to maintain. This aligns with the rules for suggesting code quality refactors.
  The function is not overly complex, and the current implementation might be sufficient for its purpose. The suggested refactor could be seen as unnecessary if the function's current form is clear and maintainable.
  While the function is not overly complex, adhering to the Single Responsibility Principle is generally beneficial for future maintenance and scalability. The suggestion is reasonable and aligns with best practices.
  The comment is valid as it suggests a reasonable refactor to adhere to the Single Responsibility Principle. It should be kept as it provides actionable advice for improving code quality.

5. scripts/update_blogs_in_learning_resources.py:77

• Draft comment:
  Ensure function names follow a consistent naming pattern. For example, use get_cutoff_date consistently with other functions like fetch_articles and update_readme.
• Reason this comment was not posted:
  Comment did not seem useful.

6. scripts/update_blogs_in_learning_resources.py:1

• Draft comment:
  Consider adding this script to the Sphinx documentation under docs/ to provide users with guidance on its purpose and usage.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The script should be documented in the Sphinx documentation under docs/ to help users understand its purpose and usage.

Workflow ID: wflow_xRPaLBAeGbNhfbbM

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

[ellipsis-dev]

Using recursion for input validation can lead to a stack overflow. Consider using a loop to repeatedly prompt for input until a valid date is entered.

[skrawcz]

Awesome, looking forward to trying this tomorrow / Monday!

Otherwise one request -- could you include a "print to standard out" option please?

[sikehish]

Awesome, looking forward to trying this tomorrow / Monday!

Otherwise one request -- could you include a "print to standard out" option please?

Hi @skrawcz. I've added that option. I've also added a docstring in the python script file so that it would be easier for the users to get started with it.

[skrawcz]

@sikehish you need to:

1. squash these changes down to a single commit (this will make rebasing simpler)
2. rebase.

Then I can merge this. Otherwise tested locally and it works, thank you!

[sikehish]

Hi @skrawcz
I tried squashing and rebasing all commits...Is this alright?
I've created a backup branch where I've cherry picked all the commits relevant to this PR(the 6-7 commits relevant to the issue). I can create another PR from the other branch and close this if this looks messed up.

[skrawcz]

Thanks @sikehish !