New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Sniper] Fix cache, and name matching tweak #5674

Merged

Gobberwart merged 18 commits into PokemonGoF:dev from Gobberwart:gobb_snipesource

Sep 26, 2016

Contributor

Gobberwart commented Sep 25, 2016

Short Description:

Two issues fixed here.

Existing cache is not persistent, and gets reset every time the task runs, which results in repeated failed attempts to catch non-existent/otherwise failed pokemon. Even if cache hits, still generates an unnecessary log message. Changed to persistent cache, and check before attempting to snipe.
Name matching can be flaky depending on sources. Added method to manually fix names we know about, or use difflib (as with pokemonoptimizer) to try and match any we haven't already caught. Note that difflib alone is not sufficient for this, eg. it thinks source name "mr-mime" is a Grimer. Close but no cigar.

Gobberwart and others added 17 commits

September 24, 2016 14:33


          Merge pull request #5645 from PokemonGoF/dev

fd49544

Dev Merge to Master


          Added to fixname method

842ca3b

Respond to possible name == None
Added Nidoran M/F


          Merge branch 'master' into gobb_snipesource

4940a96


          Added to fixname method

661468d

Respond to possible name == None
Added Nidoran M/F


          Added further unknown name fixing

980df53


          return name if no match

49ac3f7


          Merge branch 'gobb_snipesource' of https://github.com/Gobberwart/Poke…

c1cc04a

…monGo-Bot into gobb_snipesource


          Removed print statement

cd9fadb


          Added to fixname method

7ae5681

Respond to possible name == None
Added Nidoran M/F


          Added further unknown name fixing

a35bbc1


          return name if no match

48949c5


          Removed print statement

410c0e7


          Merge branch 'gobb_snipesource' of https://github.com/Gobberwart/Poke…

07a2151

…monGo-Bot into gobb_snipesource


          Fixed cache issue

7dc98fa

Previously using self.cache which gets reset every time this task is
called. Moved to self.bot.sniper_cache and added check in main snipe
thread to avoid unnecessary log spam.


          Removed debug prints

6e9a7ff


          Remove hashlib

32a1f7a

Not required


          Further improvements to _cache and _is_cached

0e89d08

Uses a list of uniqueid strings instead of list of dicts. Unique id is
an md5 hash of (pokemon_id, latitude, longitude and expiration)

mention-bot commented Sep 25, 2016

@Gobberwart, thanks for your PR! By analyzing the annotation information on this pull request, we identified @YvesHenri and @GhosterBot to be potential reviewers

YvesHenri requested changes

View reviewed changes

pokemongo_bot/cell_workers/sniper.py

+                              self.bot.sniper_cache.pop(0)
+                          self.bot.sniper_cache.append(uniqueid)
+                  def _build_unique_id(self, pokemon):

YvesHenri Sep 25, 2016

This is totally unecessary. Use _hash instead.

Contributor Author

Gobberwart Sep 25, 2016

How is it unnecessary? If there's an easy way to generate MD5 from string, then awesome. Please share your alternative code.

Contributor Author

Gobberwart Sep 26, 2016

Unless you're talking about the len(self.bot.sniper_cache) thing... which is your code so you tell me why it's necessary.

YvesHenri reviewed

View reviewed changes

pokemongo_bot/cell_workers/sniper.py

+                          name = name.replace("mr-mime","mr. mime")
+                          name = name.replace("farfetchd","farfetch'd")
+                          name = name.replace("Nidoran\u2642","nidoran m")
+                          name = name.replace("Nidoran\u2640","nidoran f")

YvesHenri Sep 25, 2016

Seriously? Are you going to handle every name combination possible?

Contributor Author

Gobberwart Sep 25, 2016 •

edited

Loading

Read the description. Obviously not, but if we KNOW about something that's causing a problem (and difflib can't handle it), then why not replace it?

YvesHenri reviewed

View reviewed changes

pokemongo_bot/cell_workers/sniper.py

		return name

		def _get_closest_name(self, name):

YvesHenri Sep 25, 2016

What's this?

Contributor Author

Gobberwart Sep 25, 2016 •

edited

Loading

@Again, read the description. Difflib is already used in pokemonoptimizer to TRY and get a real pokemon name from a supplied string. Eg. if source is "pigeot", it should return "pidgeot"... in many cases, it seems to work well. As per suggestion from @anakin5

YvesHenri reviewed

View reviewed changes

pokemongo_bot/cell_workers/sniper.py

@@ @@ -210,13 +227,12 @@ class Sniper(BaseTask): @@
                   MIN_SECONDS_ALLOWED_FOR_CELL_CHECK = 10
                   MIN_SECONDS_ALLOWED_FOR_REQUESTING_DATA = 5
                   MIN_BALLS_FOR_CATCHING = 10
-                  MAX_CACHE_LIST_SIZE = 200
+                  MAX_CACHE_LIST_SIZE = 500

YvesHenri Sep 25, 2016

You do realize people bot on raspberry devices, right? Increasing this this high might not be a good idea.

Contributor Author

Gobberwart Sep 25, 2016

You do realise that replacing dict with an md5 string makes it take FARRRRR less memory to store.

Contributor Author

Gobberwart Sep 26, 2016 •

edited

Loading

But OK, sure... I'll compromise at 300. See latest commit.

YvesHenri reviewed

View reviewed changes

pokemongo_bot/cell_workers/sniper.py

@@ @@ -232,6 +248,9 @@ def initialize(self): @@
                       self.altitude = uniform(self.bot.config.alt_min, self.bot.config.alt_max)
                       self.sources = [SniperSource(data) for data in self.config.get('sources', [])]
+                      if not hasattr(self.bot,"sniper_cache"):
+                          self.bot.sniper_cache = []

YvesHenri Sep 25, 2016

What's this and what's it for?

Contributor Author

Gobberwart Sep 25, 2016

You had a "self.cache" which is nice but not persistent. Every time the task runs, it reset it to []. This creates a bot-global cache which will persist through task re-inits.

YvesHenri requested changes

View reviewed changes

pokemongo_bot/cell_workers/sniper.py

+                          # Check if already in list of pokemon we've tried
+                          uniqueid = self._build_unique_id(pokemon)
+                          if self._is_cached(uniqueid):

YvesHenri Sep 25, 2016

is_snipeable should check whether its been cached or not.

Contributor Author

Gobberwart Sep 25, 2016

No need. It checks before it calls is_snipeable. Saves a wasted log message.

Contributor Author

Gobberwart Sep 26, 2016

is_snipeable currently lumps everything together as a general fail. I've removed one of those for clarity.

Contributor Author

Gobberwart commented Sep 25, 2016

OK @YvesHenri I've answered all your questions. Talk to me in chat if something is still unclear.


          Reduce cache size to 300

1d71fae

Contributor Author

Gobberwart commented Sep 26, 2016

Does anyone else have feedback for this?

Contributor Author

Gobberwart commented Sep 26, 2016

No feedback received and YH's requested changes are irrelevant. Merging.

Gobberwart merged commit f60a4b1 into PokemonGoF:dev

Gobberwart deleted the gobb_snipesource branch

November 9, 2016 22:33

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet