Skip to content

Commit

Permalink
prepare README for the upcoming release
Browse files Browse the repository at this point in the history
  • Loading branch information
shouya committed Mar 8, 2024
1 parent 19e4e58 commit fd8e811
Showing 1 changed file with 32 additions and 42 deletions.
74 changes: 32 additions & 42 deletions README.org
Original file line number Diff line number Diff line change
@@ -1,24 +1,28 @@
* RSS Funnel

*This project is in early stage of development. Please expect frequent breaking changes in configuration syntax and features.*
/This project is in the early stage of development. Please expect bugs and frequent breaking changes./

The RSS Funnel is a modular RSS processing pipeline. It is designed to be used to modify existing RSS source in various interesting ways such as:
RSS Funnel is a modular RSS processing pipeline system. It can manipulate RSS/Atom sources in various ways, such as:

- Fetch full content
- Generate a RSS feed from an HTML page
- Remove unwanted elements from the article (using a CSS selector)
- Keep or remove articles matching keywords or patterns
- Highlight keywords in articles
- Redact or replace text in the article (using a regular expression)
- Split a single RSS article into multiple articles
- Merge multiple feeds into a single feed
- Run arbitrary JS code to transform the feed or articles ([[https://github.com/shouya/rss-funnel/wiki/JavaScript-API][API]])
- Fetching full content
- Generating an RSS feed from an HTML page
- Removing unwanted elements and text
- Redacting or replacing text with regular expressions
- Keeping or removing articles matching keywords or patterns
- Highlighting keywords
- Splitting a single RSS article into multiple articles
- Merging multiple feeds into a single feed
- Running JavaScript code to transform the feed or articles
- Converting between feed formats
- [[https://github.com/shouya/rss-funnel/wiki/Filter-config][And more!]]

The tool comes with a web UI for inspecting the feed and the result of each filter, all within a single statically-built binary with no external dependencies.

[[https://rss-funnel-demo.fly.dev/][Try out the live demo!]]

** Installation

You can use the docker image ([[https://github.com/shouya/rss-funnel/pkgs/container/rss-funnel][latest version]]) in your =docker-compose.yaml=:
You can use the Docker image ([[https://github.com/shouya/rss-funnel/pkgs/container/rss-funnel][latest version]]) in your =docker-compose.yaml=:

#+begin_src yaml
version: "3.8"
Expand All @@ -29,27 +33,31 @@ services:
- 4080:4080
volumes:
- ./funnel.yaml:/funnel.yaml
command: /rss-funnel -c /funnel.yaml server -b 0.0.0.0:4080
environment:
RSS_FUNNEL_CONFIG: /funnel.yaml
RSS_FUNNEL_BIND: 0.0.0.0
#+end_src

The web UI will be available at =http://localhost:4080=.

Alternatively, you can build it directly from source:

#+begin_src bash
git clone https://github.com/shouya/rss-funnel.git
cd rss-funnel

# first build the front-end assets
# First, build the front-end assets
cd inspector && npm i && npm run build && cd ..

# then build the binary
# Then, build the binary
cargo build --release
#+end_src

Or if you prefer not to build from source, you can download the pre-built artifacts from [[https://github.com/shouya/rss-funnel/releases][release page]].
If you prefer not to build from source, you can download the pre-built artifacts from the [[https://github.com/shouya/rss-funnel/releases][release page]].

** Usage

To use =rss-funnel=, you need to supply a configuration file in YAML. Here is an example configuration.
To use =rss-funnel=, you need to supply a configuration file in YAML format. Here's an example configuration:

#+begin_src yaml
endpoints:
Expand All @@ -69,40 +77,22 @@ endpoints:
append_mode: true
#+end_src

Save above file to =/path/to/funnel.yaml= and run the following command:
Save the above file as =/path/to/funnel.yaml= and run the following command:

#+begin_src
rss-funnel -c /path/to/funnel.yaml server
#+end_src

You can optionally specify the bind address and port (default =127.0.0.1:4080=). Detailed usage can be found in =--help= output.

The endpoints like =http://127.0.0.1:4080/tokio-blog.xml= should be serving the filtered feeds.

** Endpoint

The most important part of the configuration is the definition of endpoints. Each endpoint correspond to a RSS feed ready for consumption.
You can optionally specify the bind address and port (default =127.0.0.1:4080=). Detailed usage can be found in the =--help= output.

Properties:
The endpoints like =http://127.0.0.1:4080/tokio-blog.xml= should serve the filtered feeds.

- =path= (required): The path of the endpoint. The path should start with =/=.
- =note= (optional): A note for the endpoint. Only used for display purpose.
- =source= (optional): The source url of the RSS feed.
+ If not specified, you must specify =?source=<url>= query in the request. This allows for usages like applying same filters for different feeds.
+ If the source points to a HTML page, =rss-funnel= will try to generate a RSS feed from the page with a single article. You can then use =split= filter to split the single article into multiple articles. See [[https://github.com/shouya/rss-funnel/wiki/Cookbook#hacker-news-top-links][Cookbook: Hacker News Top Links]] for an example.
- =filters= (required): A list of filters to apply to the feed.
+ The feed from the =source= goes through the filters in the order specified. You can think of each filter as corresponding to a transformation on the =Feed=.
+ Each filter is specified as an YAML object with the singleton key being the name of the filter and the value being the configuration of the filter.
- For example, in the filter definition: =- keep_element: .p_mainnew=
+ the filter's name is =keep_element=
+ the configuration is the string value =.p_mainnew=. Depending on the filter, the configuration can have different types.
+ The =Feed= object from the last filter is returned as the response.
- =client= (optional): The configuration for the HTTP client used to fetch the source like the user_agent. See [[https://github.com/shouya/rss-funnel/wiki/Client-config][Client config]] for detail.
** Config Syntax and Filters

** Filters
You can check out the [[https://github.com/shouya/rss-funnel/wiki/Config-syntax][Config syntax]] documentation for information about each component in the configuration file.

See [[https://github.com/shouya/rss-funnel/wiki/Filters][Filters]] for the documentations for all available filters.
See the [[https://github.com/shouya/rss-funnel/wiki/Filters][Filters]] documentation for all available filters.

** Cookbook

See [[https://github.com/shouya/rss-funnel/wiki/Cookbook][Cookbook]] for some examples of using =rss-funnel=.
Check out the [[https://github.com/shouya/rss-funnel/wiki/Cookbook][Cookbook]] for some examples of using =rss-funnel=.

0 comments on commit fd8e811

Please sign in to comment.