From 426daab980e646a9ee24634cb95960c1765e3049 Mon Sep 17 00:00:00 2001 From: Arnan de Gans Date: Mon, 15 Jan 2024 12:31:22 -0600 Subject: [PATCH] Version 1.2.1 1.2.1 - January 15, 2024 - [new] Merge identical downloads (determined by info hash) from different torrent sites that provide hashes - [new] Option to cache to flat files instead of APCu, files stored in /cache/ folder - [new] Blank index.php files in all subfolders to shield from prying eyes - [tweak] Improved version check - [fix] Stray periods in some Limetorrent categories - [fix] Inconsistent size indication for torrent results --- assets/css/index.php | 4 ++ assets/images/index.php | 4 ++ assets/index.php | 4 ++ cache/index.php | 4 ++ config.default.php | 29 ++++++---- engines/image/index.php | 4 ++ engines/image/yahoo.php | 1 + engines/index.php | 4 ++ engines/search-image.php | 20 +++---- engines/search-torrent.php | 41 +++++++++++-- engines/search.php | 23 ++++---- engines/search/google.php | 2 +- engines/search/index.php | 4 ++ engines/special/index.php | 4 ++ engines/torrent/1337x.php | 13 +++-- engines/torrent/eztv.php | 15 +++-- engines/torrent/index.php | 4 ++ engines/torrent/lime.php | 17 +++--- engines/torrent/nyaa.php | 19 +++--- engines/torrent/thepiratebay.php | 16 ++++-- engines/torrent/yts.php | 15 ++--- functions/index.php | 4 ++ functions/search_engine.php | 30 +++++++--- functions/tools.php | 99 +++++++++++++++++++++++--------- readme.md | 26 ++++++--- 25 files changed, 282 insertions(+), 124 deletions(-) create mode 100644 assets/css/index.php create mode 100644 assets/images/index.php create mode 100644 assets/index.php create mode 100644 cache/index.php create mode 100644 engines/image/index.php create mode 100644 engines/index.php create mode 100644 engines/search/index.php create mode 100644 engines/special/index.php create mode 100644 engines/torrent/index.php create mode 100644 functions/index.php diff --git a/assets/css/index.php b/assets/css/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/assets/css/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/assets/images/index.php b/assets/images/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/assets/images/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/assets/index.php b/assets/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/assets/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/cache/index.php b/cache/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/cache/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/config.default.php b/config.default.php index 0eafcc2..d435cdc 100644 --- a/config.default.php +++ b/config.default.php @@ -23,13 +23,23 @@ Disclaimer: This is not meant to 'hack proof' or truly secure the setup. Just a simple token to keep surface level prying eyes out. CACHE: - If you have ACPu enabled in your server it is highly recommended to enable caching as it'll speed up repeat searches by a lot. - "on" (Recommended) for active sites, requires APCu - "off" Disables cache, useful for testing or if your server lacks APCu support + It is highly recommended to enable caching as it'll speed up repeat searches by a lot. + "on" (Recommended) + "off" Disables cache CACHE_TIME: - Minutes the result should be cached in ACPu. - Accepts a numeric value between 1 and 30. + Minutes the result should be cached. Accepts a numeric value between 1 and 720. + APCu stores in memory, using a longer time takes up more of it. It is recommended to not exceed 30 minutes for it. + The file cache is only limited by your hosting storage space and can safely be much much longer if you want. + To not show outdated results the 'limit' is 720 minutes, which equals 12 hours. + Ignored if above 'cache' option is set to off. + +CACHE_TYPE: + Choose how to cache results. The cache is NOT unique per user but shared between all users. Different users searching for the exact same thing get the same results. + Default caching method is APCu. Alternatively, you can store the results in text files in the /cache/ folder. + Ignored if above 'cache' option is set to off. + "apcu" (Recommended) faster, utilize memory. + "file" Store results in text files. LANGUAGE: DuckDuckGo, Google and Ecosia are language agnostic. But they DO profile you for your locale. @@ -82,12 +92,12 @@ "off" Disable this special search USER AGENTS: - Add more or less user agents to the list. Keep at least one. + Add more or less user agents to the list. Keep at least one! On every search Goosle picks one at random to identify as. Keep them generic to prevent profiling, but also so that the request comes off as a generic boring browser and not as a server/crawler. Safari, Firefox and Internet Explorer/Edge should be safe to use. - Chrome may attract attention because of the lack of Chrome information (tracking) aside from the user agent. The search engine will know something is wrong. + Chrome may attract attention because of the lack of Chrome information (tracking) aside from the user agent. The search engine may know something is 'weird'. Opera/Edge/Brave and many others use Chrome under the hood and are not a good pick for that reason. Mobile agents may work, but some services like Wikipedia are a bit picky when it comes to answering API calls. Mobile users generally do not use APIs, so they may block your search. @@ -120,6 +130,7 @@ "hash" => "j9fg-i2du-er6m", "hash_auth" => "off", // Default: off "cache" => "off", // Default: off + "cache_type" => "apcu", // Default: apcu "cache_time" => 30, // Default: 30 (Minutes) "duckduckgo_language" => "uk-en", // Default: uk-en (United Kingdom) @@ -158,11 +169,7 @@ "udp://tracker.opentrackr.org:1337/announce", "udp://exodus.desync.com:6969/announce", "udp://tracker.torrent.eu.org:451/announce", - "udp://tracker.yify-torrents.com/announce", - "udp://coppersurfer.tk:6969/announce", "udp://opentracker.i2p.rocks:6969/announce", - "udp://tracker.internetwarriors.net:1337/announce", - "udp://tracker.zer0day.to:1337/announce", ) ); ?> diff --git a/engines/image/index.php b/engines/image/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/engines/image/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/engines/image/yahoo.php b/engines/image/yahoo.php index 453dec5..1fb07a6 100644 --- a/engines/image/yahoo.php +++ b/engines/image/yahoo.php @@ -101,6 +101,7 @@ public function parse_results($response) { $results['search'][] = array ("id" => $id, "source" => "Yahoo! Images", "image" => $image, "alt" => $alt, "url" => $url, "width" => $dimensions_w, "height" => $dimensions_h, "filesize" => $filesize, "direct_link" => $link, "engine_rank" => $rank); $rank -= 1; } + unset($response, $xpath, $scrape, $rank); // Add error if there are no search results if(empty($results['search'])) { diff --git a/engines/index.php b/engines/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/engines/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/engines/search-image.php b/engines/search-image.php index 3721671..30d3c4b 100644 --- a/engines/search-image.php +++ b/engines/search-image.php @@ -29,18 +29,15 @@ public function parse_results($response) { $engine_result = $request->get_results(); if(!empty($engine_result)) { - if(array_key_exists('search', $engine_result)) { - - if(array_key_exists('did_you_mean', $engine_result)) { - $results['did_you_mean'] = $engine_result['did_you_mean']; - } - - if(array_key_exists('search_specific', $engine_result)) { - $results['search_specific'][] = $engine_result['search_specific']; - } - - $query_terms = explode(" ", preg_replace("/[^a-z0-9 ]+/", "", strtolower($request->query))); + if(array_key_exists('did_you_mean', $engine_result)) { + $results['did_you_mean'] = $engine_result['did_you_mean']; + } + + if(array_key_exists('search_specific', $engine_result)) { + $results['search_specific'][] = $engine_result['search_specific']; + } + if(array_key_exists('search', $engine_result)) { // Merge duplicates and apply relevance scoring foreach($engine_result['search'] as $result) { if(array_key_exists('search', $results)) { @@ -55,6 +52,7 @@ public function parse_results($response) { $results['search'][$found_key]['goosle_rank'] += $result['engine_rank']; } else { // First find, rank and add to results + $query_terms = explode(" ", preg_replace("/[^a-z0-9 ]+/", "", strtolower($request->query))); $match_rank = match_count($result['alt'], $query_terms); $result['goosle_rank'] = $result['engine_rank'] + $match_rank; diff --git a/engines/search-torrent.php b/engines/search-torrent.php index f2b76b1..103091d 100644 --- a/engines/search-torrent.php +++ b/engines/search-torrent.php @@ -38,7 +38,36 @@ public function parse_results($response) { $engine_result = $request->get_results(); if(!empty($engine_result)) { - $results_temp = array_merge($results_temp, $engine_result); + // No merging of results +// $results_temp = array_merge($results_temp, $engine_result); + + // Merge duplicates and apply relevance scoring + foreach($engine_result as $result) { + if(count($results_temp) > 1 && !is_null($result['hash'])) { + $result_urls = array_column($results_temp, "hash", "id"); + $found_key = array_search($result['hash'], $result_urls); + } else { + $found_key = false; + } + + if($found_key !== false) { + // Duplicate result from another source + // If seeders and/or leechers mismatch, assume they're different users + if($results_temp[$found_key]['seeders'] != $result['seeders']) $results_temp[$found_key]['combo_seeders'] += $result['seeders']; + if($results_temp[$found_key]['leechers'] != $result['leechers']) $results_temp[$found_key]['combo_leechers'] += $result['leechers']; + + $results_temp[$found_key]['combo_source'][] = $result['source']; + } else { + // First find, rank and add to results + $result['combo_seeders'] = $result['seeders']; + $result['combo_leechers'] = $result['leechers']; + $result['combo_source'][] = $result['source']; + + $results_temp[$result['id']] = $result; + } + + unset($result, $result_urls, $found_key, $social_media_multiplier, $goosle_rank, $match_rank); + } } } else { $request_result = curl_getinfo($request->ch); @@ -54,7 +83,7 @@ public function parse_results($response) { if(count($results_temp) > 0) { // Sort by highest seeders - $seeders = array_column($results_temp, "seeders"); + $seeders = array_column($results_temp, "combo_seeders"); array_multisort($seeders, SORT_DESC, $results_temp); // Cap results to 50 @@ -102,18 +131,18 @@ public static function print_results($results, $opts) { foreach($results['search'] as $result) { // Extra data $meta = array(); - if($opts->show_search_source == "on") $meta[] = "Source: ".$result['source']; if(array_key_exists('quality', $result)) $meta[] = "Quality: ".$result['quality']; if(array_key_exists('year', $result)) $meta[] = "Year: ".$result['year']; if(array_key_exists('category', $result)) $meta[] = "Category: ".$result['category']; if(array_key_exists('runtime', $result)) $meta[] = "Runtime: ".date('H:i', mktime(0, $result['runtime'])); - if(array_key_exists('date_added', $result)) $meta[] = "Added: ".date('M d, Y', $result['date_added']); - if(array_key_exists('url', $result)) $meta[] = "Torrent page"; + if(array_key_exists('date_added', $result)) $meta[] = "Added on: ".date('M d, Y', $result['date_added']); + if(array_key_exists('url', $result)) $url = " - torrent page"; // Put result together echo "
  • "; echo "

    ".stripslashes($result['name'])."

    "; - echo "
    Seeds: ".$result['seeders']." - Peers: ".$result['leechers']." - Size: ".$result['size']."
    ".implode(" - ", $meta)."
    "; + echo "
    Seeds: ".$result['combo_seeders']." - Peers: ".$result['combo_leechers']." - Size: ".$result['size']."
    ".implode(" - ", $meta)."
    "; + if($opts->show_search_source == "on") echo "
    Found on: ".replace_last_comma(implode(", ", $result['combo_source'])).$url."
    "; echo "
    • "; unset($result, $meta); diff --git a/engines/search.php b/engines/search.php index 0ca8d8a..27f2cc0 100644 --- a/engines/search.php +++ b/engines/search.php @@ -38,18 +38,15 @@ public function parse_results($response) { $engine_result = $request->get_results(); if(!empty($engine_result)) { - if(array_key_exists('search', $engine_result)) { - - if(array_key_exists('did_you_mean', $engine_result)) { - $results['did_you_mean'] = $engine_result['did_you_mean']; - } - - if(array_key_exists('search_specific', $engine_result)) { - $results['search_specific'][] = $engine_result['search_specific']; - } - - $query_terms = explode(" ", preg_replace("/[^a-z0-9 ]+/", "", strtolower($request->query))); + if(array_key_exists('did_you_mean', $engine_result)) { + $results['did_you_mean'] = $engine_result['did_you_mean']; + } + + if(array_key_exists('search_specific', $engine_result)) { + $results['search_specific'][] = $engine_result['search_specific']; + } + if(array_key_exists('search', $engine_result)) { // Merge duplicates and apply relevance scoring foreach($engine_result['search'] as $result) { if(array_key_exists('search', $results)) { @@ -68,6 +65,7 @@ public function parse_results($response) { $results['search'][$found_key]['combo_source'][] = $result['source']; } else { // First find, rank and add to results + $query_terms = explode(" ", preg_replace("/[^a-z0-9 ]+/", "", strtolower($request->query))); $match_rank = match_count($result['title'], $query_terms); $match_rank += match_count($result['description'], $query_terms); // $match_rank += match_count($result['url'], $query_terms); @@ -111,7 +109,8 @@ public function parse_results($response) { array_multisort($keys, SORT_DESC, $results['search']); // Count results per source - $results['sources'] = array_count_values(array_column($results['search'], 'source')); + $sources = array_count_values(array_column($results['search'], 'source')); + if(count($sources) > 0) $results['sources'] = $sources; unset($keys); } else { diff --git a/engines/search/google.php b/engines/search/google.php index 0d20696..2432424 100644 --- a/engines/search/google.php +++ b/engines/search/google.php @@ -78,7 +78,7 @@ public function parse_results($response) { $title = htmlspecialchars(trim($title->textContent)); $id = uniqid(rand(0, 9999)); - $results['search'][] = array ("id" => $id, "source" => "Google", "title" => $title, "url" => $url, "description" => $description, "engine_rank" => $rank); + $results['search'][] = array("id" => $id, "source" => "Google", "title" => $title, "url" => $url, "description" => $description, "engine_rank" => $rank); $rank -= 1; } unset($response, $xpath, $scrape, $rank); diff --git a/engines/search/index.php b/engines/search/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/engines/search/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/engines/special/index.php b/engines/special/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/engines/special/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/engines/torrent/1337x.php b/engines/torrent/1337x.php index c23c6e6..c63e892 100644 --- a/engines/torrent/1337x.php +++ b/engines/torrent/1337x.php @@ -111,8 +111,9 @@ public function parse_results($response) { foreach($xpath->query("//table/tbody/tr") as $result) { $name = sanitize($xpath->evaluate(".//td[@class='coll-1 name']/a", $result)[1]->textContent); $url = "https://1337x.to".sanitize($xpath->evaluate(".//td[@class='coll-1 name']/a/@href", $result)[1]->textContent); - $seeders = sanitize($xpath->evaluate(".//td[@class='coll-2 seeds']", $result)[0]->textContent); - $leechers = sanitize($xpath->evaluate(".//td[@class='coll-3 leeches']", $result)[0]->textContent); + $magnet = "./engines/torrent/magnetize_1337x.php?url=".$url; + $seeders = sanitize_numeric(sanitize($xpath->evaluate(".//td[@class='coll-2 seeds']", $result)[0]->textContent)); + $leechers = sanitize_numeric(sanitize($xpath->evaluate(".//td[@class='coll-3 leeches']", $result)[0]->textContent)); $size_unformatted = explode(" ", sanitize($xpath->evaluate(".//td[contains(@class, 'coll-4 size')]", $result)[0]->textContent)); $size = $size_unformatted[0] . " " . preg_replace("/[0-9]+/", "", $size_unformatted[1]); @@ -127,21 +128,23 @@ public function parse_results($response) { if(in_array($category, $this->opts->leetx_categories_blocked)) continue; // Filter by Season (S01) or Season and Episode (S01E01) + // Where [0][0] = Season and [0][1] = Episode if(preg_match_all("/(S[0-9]{1,3})|(E[0-9]{1,3})/i", $this->query, $query_episode) && preg_match_all("/(S[0-9]{1,3})|(E[0-9]{1,3})/i", $name, $match_episode)) { if($query_episode[0][0] != $match_episode[0][0] || (array_key_exists(1, $query_episode[0]) && array_key_exists(1, $match_episode[0]) && $query_episode[0][1] != $match_episode[0][1])) { continue; } } + + $id = uniqid(rand(0, 9999)); $results[] = array ( // Required - "source" => "1337x.to", "name" => $name, "magnet" => "./engines/torrent/magnetize_1337x.php?url=".$url, "seeders" => $seeders, "leechers" => $leechers, "size" => $size, + "id" => $id, "source" => "1337x.to", "name" => $name, "magnet" => $magnet, "hash" => null, "seeders" => $seeders, "leechers" => $leechers, "size" => $size, // Extra "category" => $categories[$category], "url" => $url ); - - unset($name, $seeders, $leechers, $size_unformatted, $size, $category, $url); } + unset($response, $xpath); return $results; } diff --git a/engines/torrent/eztv.php b/engines/torrent/eztv.php index a1a5396..90ab2ce 100644 --- a/engines/torrent/eztv.php +++ b/engines/torrent/eztv.php @@ -36,11 +36,12 @@ public function parse_results($response) { if($json_response['torrents_count'] == 0) return $results; // Use API result - foreach ($json_response['torrents'] as $episode) { + foreach($json_response['torrents'] as $episode) { $name = sanitize($episode['title']); $magnet = sanitize($episode['magnet_url']); - $seeders = sanitize($episode['seeds']); - $leechers = sanitize($episode['peers']); + $hash = sanitize($episode['hash']); + $seeders = sanitize_numeric(sanitize($episode['seeds'])); + $leechers = sanitize_numeric(sanitize($episode['peers'])); $size = sanitize($episode['size_bytes']); // Ignore results with 0 seeders? @@ -51,6 +52,7 @@ public function parse_results($response) { $date_added = sanitize($episode['date_released_unix']); // Filter by Season (S01) or Season and Episode (S01E01) + // Where [0][0] = Season and [0][1] = Episode $season = sanitize($episode['season']); $episode = sanitize($episode['episode']); @@ -60,15 +62,16 @@ public function parse_results($response) { } } + $id = uniqid(rand(0, 9999)); + $results[] = array ( // Required - "source" => "EZTV", "name" => $name, "magnet" => $magnet, "seeders" => $seeders, "leechers" => $leechers, "size" => human_filesize($size), + "id" => $id, "source" => "EZTV", "name" => $name, "magnet" => $magnet, "hash" => $hash, "seeders" => $seeders, "leechers" => $leechers, "size" => human_filesize($size), // Extra "quality" => $quality, "date_added" => $date_added ); - - unset($name, $magnet, $seeders, $leechers, $size, $quality, $category, $date_added, $season, $episode); } + unset($json_response); return $results; } diff --git a/engines/torrent/index.php b/engines/torrent/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/engines/torrent/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/engines/torrent/lime.php b/engines/torrent/lime.php index a9b7222..34cf832 100644 --- a/engines/torrent/lime.php +++ b/engines/torrent/lime.php @@ -29,35 +29,38 @@ public function parse_results($response) { $name = sanitize($xpath->evaluate(".//td[@class='tdleft']//a[2]", $result)[0]->textContent); $hash = sanitize($xpath->evaluate(".//td[@class='tdleft']//a[1]/@href", $result)[0]->textContent); $hash = explode("/", substr($hash, 0, strpos($hash, ".torrent?"))); - $magnet = "magnet:?xt=urn:btih:".$hash[array_key_last($hash)]."&dn=".$name."&tr=".implode("&tr=", $this->opts->torrent_trackers); - $seeders = sanitize($xpath->evaluate(".//td[@class='tdseed']", $result)[0]->textContent); - $leechers = sanitize($xpath->evaluate(".//td[@class='tdleech']", $result)[0]->textContent); + $hash = $hash[array_key_last($hash)]; + $magnet = "magnet:?xt=urn:btih:".$hash."&dn=".urlencode($name)."&tr=".implode("&tr=", $this->opts->torrent_trackers); + $seeders = sanitize_numeric(sanitize($xpath->evaluate(".//td[@class='tdseed']", $result)[0]->textContent)); + $leechers = sanitize_numeric(sanitize($xpath->evaluate(".//td[@class='tdleech']", $result)[0]->textContent)); $size = sanitize($xpath->evaluate(".//td[@class='tdnormal'][2]", $result)[0]->textContent); // Ignore results with 0 seeders? if($this->opts->show_zero_seeders == "off" AND $seeders == 0) continue; // Get extra data - $category = explode(" ", sanitize($xpath->evaluate(".//td[@class='tdnormal'][1]", $result)[0]->textContent)); + $category = explode(" ", trim(sanitize($xpath->evaluate(".//td[@class='tdnormal'][1]", $result)[0]->textContent), ".,")); $category = $category[array_key_last($category)]; $url = "https://www.limetorrents.lol".sanitize($xpath->evaluate(".//td[@class='tdleft']//a[2]/@href", $result)[0]->textContent); // Filter by Season (S01) or Season and Episode (S01E01) + // Where [0][0] = Season and [0][1] = Episode if(preg_match_all("/(S[0-9]{1,3})|(E[0-9]{1,3})/i", $this->query, $query_episode) && preg_match_all("/(S[0-9]{1,3})|(E[0-9]{1,3})/i", $name, $match_episode)) { if($query_episode[0][0] != $match_episode[0][0] || (array_key_exists(1, $query_episode[0]) && array_key_exists(1, $match_episode[0]) && $query_episode[0][1] != $match_episode[0][1])) { continue; } } + + $id = uniqid(rand(0, 9999)); $results[] = array ( // Required - "source" => "limetorrents.lol", "name" => $name, "magnet" => $magnet, "seeders" => $seeders, "leechers" => $leechers, "size" => $size, + "id" => $id, "source" => "limetorrents.lol", "name" => $name, "magnet" => $magnet, "hash" => $hash, "seeders" => $seeders, "leechers" => $leechers, "size" => $size, // Extra "category" => $category, "url" => $url ); - - unset($name, $seeders, $leechers, $size, $hash, $magnet, $category, $url); } + unset($response, $xpath); return $results; } diff --git a/engines/torrent/nyaa.php b/engines/torrent/nyaa.php index 2286703..ae2ee5d 100644 --- a/engines/torrent/nyaa.php +++ b/engines/torrent/nyaa.php @@ -32,10 +32,13 @@ public function parse_results($response) { $name = sanitize($xpath->evaluate(".//td[@colspan='2']//a[not(contains(@class, 'comments'))]/@title", $result)[0]->textContent); $magnet = sanitize($xpath->evaluate(".//a[2]/@href", $meta[0])[0]->textContent); - $seeders = sanitize($meta[3]->textContent); - $leechers = sanitize($meta[4]->textContent); - $size = sanitize($meta[1]->textContent); - + $hash = parse_url($magnet, PHP_URL_QUERY); + parse_str($hash, $hash_parameters); + $hash = str_replace("urn:btih:", "", $hash_parameters['xt']); + $seeders = sanitize_numeric(sanitize($meta[3]->textContent)); + $leechers = sanitize_numeric(sanitize($meta[4]->textContent)); + $size = str_replace("GiB", "GB", str_replace("MiB", "MB", sanitize($meta[1]->textContent))); + // Ignore results with 0 seeders? if($this->opts->show_zero_seeders == "off" AND $seeders == 0) continue; @@ -48,21 +51,23 @@ public function parse_results($response) { $date_added = mktime(0, 0, 0, intval($date_added[1]), intval($date_added[2]), intval($date_added[0])); // Filter by Season (S01) or Season and Episode (S01E01) + // Where [0][0] = Season and [0][1] = Episode if(preg_match_all("/(S[0-9]{1,3})|(E[0-9]{1,3})/i", $this->query, $query_episode) && preg_match_all("/(S[0-9]{1,3})|(E[0-9]{1,3})/i", $name, $match_episode)) { if($query_episode[0][0] != $match_episode[0][0] || (array_key_exists(1, $query_episode[0]) && array_key_exists(1, $match_episode[0]) && $query_episode[0][1] != $match_episode[0][1])) { continue; } } + $id = uniqid(rand(0, 9999)); + $results[] = array ( // Required - "source" => "nyaa.si", "name" => $name, "magnet" => $magnet, "seeders" => $seeders, "leechers" => $leechers, "size" => $size, + "id" => $id, "source" => "nyaa.si", "name" => $name, "magnet" => $magnet, "hash" => $hash, "seeders" => $seeders, "leechers" => $leechers, "size" => $size, // Extra "category" => $category, "url" => $url, "date_added" => $date_added ); - - unset($name, $magnet, $seeders, $leechers, $size, $category, $url, $date_added, $meta); } + unset($response, $xpath); return $results; } diff --git a/engines/torrent/thepiratebay.php b/engines/torrent/thepiratebay.php index f071992..dc0c99c 100644 --- a/engines/torrent/thepiratebay.php +++ b/engines/torrent/thepiratebay.php @@ -93,10 +93,12 @@ public function parse_results($response) { // Nothing found if($response['name'] == "No results returned") break; + $name = sanitize($response['name']); - $magnet = "magnet:?xt=urn:btih:".sanitize($response['info_hash'])."&dn=".$name."&tr=".implode("&tr=", $this->opts->torrent_trackers); - $seeders = sanitize($response['seeders']); - $leechers = sanitize($response['leechers']); + $hash = sanitize($response['info_hash']); + $magnet = "magnet:?xt=urn:btih:".$hash."&dn=".urlencode($name)."&tr=".implode("&tr=", $this->opts->torrent_trackers); + $seeders = sanitize_numeric(sanitize($response['seeders'])); + $leechers = sanitize_numeric(sanitize($response['leechers'])); $size = sanitize($response['size']); // Ignore results with 0 seeders? @@ -111,21 +113,23 @@ public function parse_results($response) { if(in_array($category, $this->opts->piratebay_categories_blocked)) continue; // Filter by Season (S01) or Season and Episode (S01E01) + // Where [0][0] = Season and [0][1] = Episode if(preg_match_all("/(S[0-9]{1,3})|(E[0-9]{1,3})/i", $this->query, $query_episode) && preg_match_all("/(S[0-9]{1,3})|(E[0-9]{1,3})/i", $name, $match_episode)) { if($query_episode[0][0] != $match_episode[0][0] || (array_key_exists(1, $query_episode[0]) && array_key_exists(1, $match_episode[0]) && $query_episode[0][1] != $match_episode[0][1])) { continue; } } + $id = uniqid(rand(0, 9999)); + $results[] = array( // Required - "source" => "thepiratebay.org", "name" => $name, "magnet" => $magnet, "seeders" => $seeders, "leechers" => $leechers, "size" => human_filesize($size), + "id" => $id, "source" => "thepiratebay.org", "name" => $name, "magnet" => $magnet, "hash" => $hash, "seeders" => $seeders, "leechers" => $leechers, "size" => human_filesize($size), // Extra "category" => $categories[$category], "url" => $url, "date_added" => $date_added, ); - - unset($name, $magnet, $seeders, $leechers, $size, $category, $url, $date_added); } + unset($json_response); return $results; } diff --git a/engines/torrent/yts.php b/engines/torrent/yts.php index ae4a761..26c4227 100644 --- a/engines/torrent/yts.php +++ b/engines/torrent/yts.php @@ -51,9 +51,10 @@ public function parse_results($response) { $date_added = sanitize($movie['date_uploaded_unix']); foreach ($movie['torrents'] as $torrent) { - $magnet = "magnet:?xt=urn:btih:".sanitize($torrent['hash'])."&dn=".urlencode($name)."&tr=".implode("&tr=", $this->opts->torrent_trackers); - $seeders = sanitize($torrent['seeds']); - $leechers = sanitize($torrent['peers']); + $hash = sanitize($torrent['hash']); + $magnet = "magnet:?xt=urn:btih:".$hash."&dn=".urlencode($name)."&tr=".implode("&tr=", $this->opts->torrent_trackers); + $seeders = sanitize_numeric(sanitize($torrent['seeds'])); + $leechers = sanitize_numeric(sanitize($torrent['peers'])); $size = sanitize($torrent['size']); // Ignore results with 0 seeders? @@ -61,17 +62,17 @@ public function parse_results($response) { // Get extra data $quality = sanitize($torrent['quality']); - + $id = uniqid(rand(0, 9999)); + $results[] = array ( // Required - "source" => "yts.mx", "name" => $name, "magnet" => $magnet, "seeders" => $seeders, "leechers" => $leechers, "size" => $size, + "id" => $id, "source" => "yts.mx", "name" => $name, "magnet" => $magnet, "hash" => $hash, "seeders" => $seeders, "leechers" => $leechers, "size" => $size, // Extra "quality" => $quality, "year" => $year, "category" => $category, "runtime" => $runtime, "url" => $url, "date_added" => $date_added ); } - - unset($name, $magnet, $seeders, $leechers, $size, $quality, $year, $category, $runtime, $url, $date_added); } + unset($json_response); return $results; } diff --git a/functions/index.php b/functions/index.php new file mode 100644 index 0000000..9abd899 --- /dev/null +++ b/functions/index.php @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/functions/search_engine.php b/functions/search_engine.php index fd49199..b920d61 100644 --- a/functions/search_engine.php +++ b/functions/search_engine.php @@ -23,7 +23,7 @@ function __construct($opts, $mh) { if(!$this->url) return; // Skip if there is a cached result (from earlier search) - if($this->opts->cache == "on" && has_cached_results($this->url, $this->opts->hash)) return; + if($this->opts->cache == "on" && has_cached_results($this->opts->cache_type, $this->opts->hash, $this->url, (intval($this->opts->cache_time) * 60))) return; // Curl $this->ch = curl_init(); @@ -43,7 +43,7 @@ public function get_request_url() { // Check if a request to a search engine was successful --------------------------------------*/ public function request_successful() { - if((isset($this->ch) && curl_getinfo($this->ch)['http_code'] == '200') || has_cached_results($this->url, $this->opts->hash)) { + if((isset($this->ch) && curl_getinfo($this->ch)['http_code'] == '200') || has_cached_results($this->opts->cache_type, $this->opts->hash, $this->url, (intval($this->opts->cache_time) * 60))) { return true; } @@ -56,21 +56,31 @@ abstract function parse_results($response); // Load search results --------------------------------------*/ public function get_results() { - - // It's a torrent search? - if(!isset($this->url)) return $this->parse_results(null); + if(!isset($this->url)) { + return $this->parse_results(null); + } + + $ttl = intval($this->opts->cache_time) * 60; // If there is a cached result from an earlier search use that instead - if($this->opts->cache == "on" && has_cached_results($this->url, $this->opts->hash)) return fetch_cached_results($this->url, $this->opts->hash); + if($this->opts->cache == "on" && has_cached_results($this->opts->cache_type, $this->opts->hash, $this->url, $ttl)) { + return fetch_cached_results($this->opts->cache_type, $this->opts->hash, $this->url); + } // Curl request - if(!isset($this->ch)) return $this->parse_results(null); + if(!isset($this->ch)) { + return $this->parse_results(null); + } + $response = ($this->mh) ? curl_multi_getcontent($this->ch) : curl_exec($this->ch); $results = $this->parse_results($response) ?? array(); // Cache last request - if($this->opts->cache == "on" && !empty($results)) store_cached_results($this->url, $this->opts->hash, $results, (intval($this->opts->cache_time) * 60)); + if($this->opts->cache == "on") { + if(!empty($results)) store_cached_results($this->opts->cache_type, $this->opts->hash, $this->url, $results, $ttl); + if($this->opts->cache_type == "file") delete_cached_results($ttl); + } return $results; } @@ -90,9 +100,11 @@ function load_opts() { $opts->user_auth = (isset($_REQUEST['a'])) ? sanitize($_REQUEST['a']) : ""; // Force a few defaults and safeguards + if($opts->cache_type == "file" && !is_dir(dirname(__DIR__).'/cache/')) $opts->cache = "off"; + if($opts->cache_type == "apcu" && !function_exists("apcu_exists")) $opts->cache = "off"; if($opts->enable_image_search == "off" && $opts->type == 1) $opts->type = 0; if($opts->enable_torrent_search == "off" && $opts->type == 9) $opts->type = 0; - if(!is_numeric($opts->cache_time) || ($opts->cache_time > 30 || $opts->cache_time < 1)) $opts->cache_time = 30; + if(!is_numeric($opts->cache_time) || ($opts->cache_time > 720 || $opts->cache_time < 1)) $opts->cache_time = 30; if(!is_numeric($opts->social_media_relevance) || ($opts->social_media_relevance > 10 || $opts->social_media_relevance < 0)) $opts->social_media_relevance = 8; // Remove ! at the start of queries to prevent DDG Bangs (!g, !c and crap like that) diff --git a/functions/tools.php b/functions/tools.php index 8d92247..8c32785 100644 --- a/functions/tools.php +++ b/functions/tools.php @@ -53,14 +53,13 @@ function set_curl_options($curl, $url, $user_agents) { // Load pages into a DOM --------------------------------------*/ function get_xpath($response) { - if(!$response) - return null; - - $htmlDom = new DOMDocument; - @$htmlDom->loadHTML($response); - $xpath = new DOMXPath($htmlDom); - - return $xpath; + if(!$response) return null; + + $htmlDom = new DOMDocument; + @$htmlDom->loadHTML($response); + $xpath = new DOMXPath($htmlDom); + + return $xpath; } /*-------------------------------------- @@ -68,38 +67,73 @@ function get_xpath($response) { --------------------------------------*/ function get_formatted_url($url) { $url = parse_url($url); - + $formatted_url = $url['scheme'] . "://" . $url['host']; $formatted_url .= str_replace('/', ' › ', urldecode(str_replace('%20', ' ', rtrim($url['path'], '/')))); - + return $formatted_url; } /*-------------------------------------- -// APCu Caching +// Result Caching --------------------------------------*/ -function has_cached_results($url, $hash) { - if(function_exists("apcu_exists")) { +function has_cached_results($cache_type, $hash, $url, $ttl) { + if($cache_type == "apcu") { return apcu_exists("$hash:$url"); } + if($cache_type == "file") { + $cache_file = dirname(__DIR__).'/cache/'.md5("$hash:$url").'.data'; + if(is_file($cache_file)) { + if(filemtime($cache_file) >= (time() - $ttl)) { + return true; + } + } + } + return false; } -function store_cached_results($url, $hash, $results, $ttl = 0) { - if(function_exists("apcu_store") && !empty($results)) { - return apcu_store("$hash:$url", $results, $ttl); +function store_cached_results($cache_type, $hash, $url, $results, $ttl) { + if($cache_type == "apcu" && !empty($results)) { + apcu_store("$hash:$url", $results, $ttl); + } + + if($cache_type == "file") { + $cache_file = dirname(__DIR__).'/cache/'.md5("$hash:$url").'.data'; + file_put_contents($cache_file, serialize($results)); } } -function fetch_cached_results($url, $hash) { - if(function_exists("apcu_fetch")) { +function fetch_cached_results($cache_type, $hash, $url) { + if($cache_type == "apcu") { return apcu_fetch("$hash:$url"); } - + + if($cache_type == "file") { + $cache_file = dirname(__DIR__).'/cache/'.md5("$hash:$url").'.data'; + if(is_file($cache_file)) { + return unserialize(file_get_contents($cache_file)); + } + } + return array(); } +function delete_cached_results($ttl) { + $folder = opendir(dirname(__DIR__).'/cache/'); + while($file_name = readdir($folder)) { + $extension = pathinfo($file_name, PATHINFO_EXTENSION); + if($file_name == "." OR $file_name == ".." OR $extension != "data") continue; + + if(is_file($folder.$file_name)) { + if(filemtime($folder.$file_name) < (time() - $ttl)) { + unlink($folder.$file_name); + } + } + } +} + /*-------------------------------------- // Sanitize variables --------------------------------------*/ @@ -119,6 +153,13 @@ function sanitize($variable) { return $variable; } +function sanitize_numeric($variable) { + $variable = preg_replace('/[^0-9]/', '', $variable); + if(strlen($variable) == 0) $variable = 0; + + return $variable; +} + /*-------------------------------------- // Search result match counter --------------------------------------*/ @@ -200,7 +241,7 @@ function search_sources($results) { $sources = replace_last_comma(implode(', ', $sources)); - echo "
  • ".$sources.".
    • "; + echo "
  • Includes ".$sources.".
    • "; unset($sources); } @@ -280,21 +321,22 @@ function string_generator() { function show_version($opts) { $cache_file = dirname(__DIR__).'/version.data'; + // Currently installed version + $current_version = "1.2.1"; + if(!is_file($cache_file)){ // Create update cache file - $version = array('version' => "1.2", 'latest' => "0.0", "checked" => 0, "url" => ""); + $version = array('latest' => "0.0", "checked" => 0, "url" => ""); file_put_contents($cache_file, serialize($version)); } else { // Get update information $version = unserialize(file_get_contents($cache_file)); } - // Current version - $show_version = "Goosle ".$version['version']."."; - + // Update check, every week if($version['checked'] < time() - 604800) { $ch = curl_init(); - set_curl_options($ch, "https://api.github.com/repos/adegans/goosle/releases/latest", $opts->user_agents); + set_curl_options($ch, "https://api.github.com/repos/adegans/goosle/releases/latest", array("goosle/".$current_version.";")); $response = curl_exec($ch); curl_close($ch); @@ -304,12 +346,15 @@ function show_version($opts) { if(empty($json_response)) return $show_version; // Update version info - $version = array('version' => $version['version'], 'latest' => $json_response['tag_name'], "checked" => time(), "url" => $json_response['html_url']); + $version = array('latest' => $json_response['tag_name'], "checked" => time(), "url" => $json_response['html_url']); file_put_contents($cache_file, serialize($version)); } + // Format version for footer + $show_version = "Goosle ".$current_version."."; + // Check if a newer version is available and add it to the version display - if(version_compare($version['version'], $version['latest'], "<")) { + if(version_compare($current_version, $version['latest'], "<")) { $show_version .= " Version ".$version['latest']." is available!"; } diff --git a/readme.md b/readme.md index 278deb4..b649281 100644 --- a/readme.md +++ b/readme.md @@ -14,7 +14,7 @@ Host for yourself and friends, with a access hash key. Or set up a public search After-all, finding things should be easy and not turn into a chore. -[![Goosle Mainpage](https://ajdg.solutions/assets/goosle/goosle-main.jpg)](https://ajdg.solutions/assets/goosle/goosle-main.jpg) +[![Goosle Mainpage](https://ajdg.solutions/wp-content/uploads/2023/12/goosle-mainpage-960x593.png)](https://ajdg.solutions/wp-content/uploads/2023/12/goosle-mainpage.png) ## Features - Works on **any** hosting package that does PHP7.4 or newer @@ -44,9 +44,9 @@ And yet it just works... fast! If you like Goosle, or found a use for it, please support my work and [donate](https://www.arnan.me/donate.html?mtm_campaign=goosle_readme) and tell everyone about its existence. ## Screenshots -[![Goosle Search results](https://ajdg.solutions/assets/goosle/goosle-search-150x150.jpg)](https://ajdg.solutions/assets/goosle/goosle-search.jpg) -[![Goosle Image results](https://ajdg.solutions/assets/goosle/goosle-images-150x150.jpg)](https://ajdg.solutions/assets/goosle/goosle-images.jpg) -[![Goosle Torrent results](https://ajdg.solutions/assets/goosle/goosle-torrents-150x150.jpg)](https://ajdg.solutions/assets/goosle/goosle-torrents.jpg) +[![Goosle Mainpage](https://ajdg.solutions/wp-content/uploads/2023/12/goosle-mainpage-150x150.png)](https://ajdg.solutions/wp-content/uploads/2023/12/goosle-mainpage.png) +[![Goosle Search results](https://ajdg.solutions/wp-content/uploads/2023/12/goosle-search-150x150.png)](https://ajdg.solutions/wp-content/uploads/2023/12/goosle-search.png) +[![Goosle Torrent results](https://ajdg.solutions/wp-content/uploads/2023/12/goosle-torrentsearch-150x150.png)](https://ajdg.solutions/wp-content/uploads/2023/12/goosle-torrentsearch.png) ## Requirements Any basic webserver/webhosting package with PHP7.4 or newer. \ @@ -56,16 +56,16 @@ Tested to work on Apache with PHP8.0.24-8.2.x. ## Installation 1. Unzip the download. 2. In the main directory. Copy config.default.php to config.php. -3. Edit the config.php file and set your preferences. -4. Upload all files to your webserver, for example to the root folder of a domain (eg. example.com), subdomain (eg. search.example.com) or a sub-folder on your main domain (eg. example.com/search/) +3. Edit config.php file and set your preferences. +4. Upload all files to your webserver, for example to the root folder of a subdomain (eg. search.example.com) or a sub-folder on your main site (eg. example.com/search/) 5. Rename goosle.htaccess to .htaccess -6. Load the site in your browser. If you've enabled the access hash add *?a=YOURHASH* to the url. +6. Load the site in your browser. If you've enabled the access hash add ?a=YOURHASH to the url. 7. Let me know where you installed Goosle :-) ## Updates 1. Unzip the download. -2. Check your config.php file and go over your preferences. Make sure any new settings are present in your config.php. (Or reconfigure Goosle with a new copy from config.default.php) -3. Upload all files to your webserver, overwriting the current Goosle files. +2. Check your config.php file and go over your preferences. Make sure any new settings or changed values are present in your config.php. (Or reconfigure Goosle with a new copy from config.default.php) +3. Upload all files to your webserver, overwriting all files except perhaps config.php. 4. Load the site in your browser. If you've enabled the access hash don't forget to add *?a=YOURHASH* to the url. 5. Enjoy your updated search experience! @@ -82,6 +82,14 @@ You can post your questions on Github Discussions or on my support forum on [ajd Or say hi on [Mastodon](https://mas.to/@arnan) or [Telegram](https://t.me/arnandegans). ## Changelog +1.2.1 - January 15, 2024 +- [new] Merge identical downloads (determined by info hash) from different torrent sites that provide hashes +- [new] Option to cache to flat files instead of APCu, files stored in /cache/ folder +- [new] Blank index.php files in all subfolders to shield from prying eyes +- [tweak] Improved version check +- [fix] Stray periods in some Limetorrent categories +- [fix] Inconsistent size indication for torrent results + 1.2 - January 2, 2024 - [new] Preferred language setting for DuckDuckGo results in config.php. - [new] Preferred language setting for Wikipedia results in config.php.