Skip to content

Commit

Permalink
chore(website): fix url getter function name
Browse files Browse the repository at this point in the history
  • Loading branch information
j-mendez committed Apr 28, 2024
1 parent 6113181 commit 21e4013
Show file tree
Hide file tree
Showing 6 changed files with 25 additions and 25 deletions.
6 changes: 3 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion spider/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "spider"
version = "1.93.11"
version = "1.93.13"
authors = [
"j-mendez <jeff@a11ywatch.com>"
]
Expand Down
28 changes: 14 additions & 14 deletions spider/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ This is a basic async example crawling a web page, add spider to your `Cargo.tom

```toml
[dependencies]
spider = "1.93.11"
spider = "1.93.13"
```

And then the code:
Expand Down Expand Up @@ -93,7 +93,7 @@ We have the following optional feature flags.

```toml
[dependencies]
spider = { version = "1.93.11", features = ["regex", "ua_generator"] }
spider = { version = "1.93.13", features = ["regex", "ua_generator"] }
```

1. `ua_generator`: Enables auto generating a random real User-Agent.
Expand Down Expand Up @@ -138,7 +138,7 @@ Move processing to a worker, drastically increases performance even if worker is

```toml
[dependencies]
spider = { version = "1.93.11", features = ["decentralized"] }
spider = { version = "1.93.13", features = ["decentralized"] }
```

```sh
Expand Down Expand Up @@ -169,7 +169,7 @@ Use the subscribe method to get a broadcast channel.

```toml
[dependencies]
spider = { version = "1.93.11", features = ["sync"] }
spider = { version = "1.93.13", features = ["sync"] }
```

```rust,no_run
Expand Down Expand Up @@ -199,7 +199,7 @@ Allow regex for blacklisting routes

```toml
[dependencies]
spider = { version = "1.93.11", features = ["regex"] }
spider = { version = "1.93.13", features = ["regex"] }
```

```rust,no_run
Expand All @@ -226,7 +226,7 @@ If you are performing large workloads you may need to control the crawler by ena

```toml
[dependencies]
spider = { version = "1.93.11", features = ["control"] }
spider = { version = "1.93.13", features = ["control"] }
```

```rust
Expand Down Expand Up @@ -296,7 +296,7 @@ Use cron jobs to run crawls continuously at anytime.

```toml
[dependencies]
spider = { version = "1.93.11", features = ["sync", "cron"] }
spider = { version = "1.93.13", features = ["sync", "cron"] }
```

```rust,no_run
Expand Down Expand Up @@ -335,7 +335,7 @@ the feature flag [`chrome_intercept`] to possibly speed up request using Network

```toml
[dependencies]
spider = { version = "1.93.11", features = ["chrome", "chrome_intercept"] }
spider = { version = "1.93.13", features = ["chrome", "chrome_intercept"] }
```

You can use `website.crawl_concurrent_raw` to perform a crawl without chromium when needed. Use the feature flag `chrome_headed` to enable headful browser usage if needed to debug.
Expand Down Expand Up @@ -365,7 +365,7 @@ Enabling HTTP cache can be done with the feature flag [`cache`] or [`cache_mem`]

```toml
[dependencies]
spider = { version = "1.93.11", features = ["cache"] }
spider = { version = "1.93.13", features = ["cache"] }
```

You need to set `website.cache` to true to enable as well.
Expand Down Expand Up @@ -396,7 +396,7 @@ Intelligently run crawls using HTTP and JavaScript Rendering when needed. The be

```toml
[dependencies]
spider = { version = "1.93.11", features = ["smart"] }
spider = { version = "1.93.13", features = ["smart"] }
```

```rust,no_run
Expand All @@ -422,7 +422,7 @@ Use OpenAI to generate dynamic scripts to drive the browser done with the featur

```toml
[dependencies]
spider = { version = "1.93.11", features = ["openai"] }
spider = { version = "1.93.13", features = ["openai"] }
```

```rust
Expand All @@ -448,7 +448,7 @@ Set a depth limit to prevent forwarding.

```toml
[dependencies]
spider = { version = "1.93.11", features = ["budget"] }
spider = { version = "1.93.13", features = ["budget"] }
```

```rust,no_run
Expand Down Expand Up @@ -511,7 +511,7 @@ async fn main() -> Result<(), Error> {
{
Ok(mut website) => {
let handle = tokio::spawn(async move {
println!("Starting Crawl - {:?}", website.get_domain().inner());
println!("Starting Crawl - {:?}", website.get_url().inner());

let start = Instant::now();
website.crawl().await;
Expand All @@ -525,7 +525,7 @@ async fn main() -> Result<(), Error> {

println!(
"{:?} - Time elapsed in website.crawl() is: {:?} for total pages: {:?}",
website.get_domain().inner(),
website.get_url().inner(),
duration,
links.len()
);
Expand Down
6 changes: 3 additions & 3 deletions spider/src/website.rs
Original file line number Diff line number Diff line change
Expand Up @@ -558,12 +558,12 @@ impl Website {
}

/// Domain parsed url getter.
pub fn get_domain_parsed(&self) -> &Option<Box<Url>> {
pub fn get_url_parsed(&self) -> &Option<Box<Url>> {
&self.domain_parsed
}

/// Domain name getter.
pub fn get_domain(&self) -> &CaseInsensitiveString {
pub fn get_url(&self) -> &CaseInsensitiveString {
&self.url
}

Expand Down Expand Up @@ -4563,7 +4563,7 @@ impl Job for Website {
async fn handle(&mut self) {
log::info!(
"CRON: {} - cron job running {}",
self.get_domain().as_ref(),
self.get_url().as_ref(),
self.now()
);
if self.configuration.cron_type == CronType::Crawl {
Expand Down
4 changes: 2 additions & 2 deletions spider_cli/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "spider_cli"
version = "1.93.11"
version = "1.93.13"
authors = [
"madeindjs <contact@rousseau-alexandre.fr>",
"j-mendez <jeff@a11ywatch.com>",
Expand Down Expand Up @@ -29,7 +29,7 @@ quote = "1.0.18"
failure_derive = "0.1.8"

[dependencies.spider]
version = "1.93.11"
version = "1.93.13"
path = "../spider"

[[bin]]
Expand Down
4 changes: 2 additions & 2 deletions spider_worker/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "spider_worker"
version = "1.93.11"
version = "1.93.13"
authors = [
"madeindjs <contact@rousseau-alexandre.fr>",
"j-mendez <jeff@a11ywatch.com>",
Expand All @@ -25,7 +25,7 @@ lazy_static = "1.4.0"
env_logger = "0.11.3"

[dependencies.spider]
version = "1.93.11"
version = "1.93.13"
path = "../spider"
features = ["serde", "flexbuffers"]

Expand Down

0 comments on commit 21e4013

Please sign in to comment.