add sqlite pragma settings to sql_storage.py - change distinct to aggregate in mongodb.py #916

lesleslie · 2017-08-09T10:00:38Z

This commit is in response to #873. It allows sqlite pragma settings to be set. This commit sets

PRAGMA journal_mode=WAL
PRAGMA synchronous=NORMAL

when sqlite is used for the database. These settings should speed things up while still maintaining data integrity.

See #873 for more details.

lesleslie · 2017-08-09T10:34:00Z

Also, change default_session.id to default_conversation_id
for examples learning_feedback_example.py and learning_new_response.py.

Those examples were broken somewhere between 7.4 and 7.6.

lesleslie · 2017-08-09T14:42:47Z

change distinct to aggregate for response_query in mongodb.py get_response_statements.

This probably closes #747, #697, #686.

I didn't actually clock it, but the query now seems to be running about at least 4x faster on an 18MB collection.

and yes, you should be able to break the 16MB BSON limit?.

vkosuri · 2017-08-10T07:02:58Z

chatterbot/chatterbot.py

@@ -116,7 +116,7 @@ def get_response(self, input_item, conversation_id=None):

        if not self.read_only:
            self.learn_response(statement, previous_statement)
-            self.storage.add_to_converation(conversation_id, statement, response)
+            self.storage.add_to_conversation(conversation_id, statement, response)


gunthercox · 2017-08-10T21:43:07Z

@lesleslie There are some great changes here, thank you. Once the tests are passing I'll be happy to merge in your changes.

Thank you again.

lesleslie · 2017-08-11T01:15:25Z

Thank you. This is a great app and I'm thrilled to be able to contribute to it.

…

-- les

On Aug 10, 2017 2:43 PM, "Gunther Cox" ***@***.***> wrote: @lesleslie <https://github.com/lesleslie> There are some great changes here, thank you. Once the tests are passing I'll be happy to merge in your changes. Thank you again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#916 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOUGNzhIOkcQoBp1HBlwTPDe_1VfzKETks5sW3l0gaJpZM4Ox279> .

lesleslie · 2017-08-11T01:17:38Z

Just fyi, you should expect contributions from me for the kensor/tensor flow logic adapter as well as Facebook messenger input and output adapters.

…

On Aug 10, 2017 6:15 PM, "les" ***@***.***> wrote: Thank you. This is a great app and I'm thrilled to be able to contribute to it. -- les On Aug 10, 2017 2:43 PM, "Gunther Cox" ***@***.***> wrote: > @lesleslie <https://github.com/lesleslie> There are some great changes > here, thank you. Once the tests are passing I'll be happy to merge in your > changes. > > Thank you again. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#916 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AOUGNzhIOkcQoBp1HBlwTPDe_1VfzKETks5sW3l0gaJpZM4Ox279> > . >

lesleslie · 2017-08-11T07:59:57Z

Ok - all tests pass!

Sorry about the sloppy pull request and all the commits. I have a better idea of the process now. This was also my first experience with Travis-CI but I think I have a good handle on that now too.
I'll be much cleaner about my pull requests moving forward.

I will note it hear that increasing the sort buffer size using:

self.client.admin.command({'setParameter': 1, 'internalQueryExecMaxBlockingSortBytes': 44040192})

does not currently work for Python 2 on Linux or Python 3 on Windows.

vkosuri · 2017-08-11T09:26:17Z

examples/learning_feedback_example.py

@@ -22,7 +22,7 @@
    output_adapter='chatterbot.output.TerminalAdapter'
 )

-DEFAULT_SESSION_ID = bot.default_session.id
+DEFAULT_SESSION_ID = bot.default_conversation_id


I think this could also rephrase to DEFAULT_CONVERSATION_ID

vkosuri · 2017-08-11T09:26:49Z

examples/learning_new_response.py

@@ -16,7 +16,7 @@

 bot.train("chatterbot.corpus.english")

-DEFAULT_SESSION_ID = bot.default_session.id
+DEFAULT_SESSION_ID = bot.default_conversation_id


I think this could also rephrase to DEFAULT_CONVERSATION_ID

gunthercox · 2017-08-10T21:41:29Z

chatterbot/storage/mongodb.py

@@ -92,6 +92,9 @@ def __init__(self, **kwargs):
        # Use the default host and port
        self.client = MongoClient(self.database_uri)

+        # Increase the sort buffer to 42M
+        self.client.admin.command({'setParameter': 1, 'internalQueryExecMaxBlockingSortBytes': 44040192})


It looks like tests are failing after this change;

pymongo.errors.OperationFailure: no such command: 'internalQueryExecMaxBlockingSortBytes', bad cmd: '{ internalQueryExecMaxBlockingSortBytes: 44040192, setParameter: 1 }'

gunthercox · 2017-08-11T20:17:32Z

Thank you again. Fantastic job @lesleslie putting all this together. It is greatly appreciated!

lesleslie added 2 commits August 9, 2017 02:53

add sqlite pragma settings

af76b8c

change default_session.id to default_conversation_id

99b14ed

vkosuri approved these changes Aug 9, 2017

View reviewed changes

lesleslie added 2 commits August 9, 2017 07:17

switch statement for response line 118

957a970

work

25a2f1a

lesleslie added 2 commits August 9, 2017 08:40

bug fix

29cd3fe

undo statement to response

9c202a8

lesleslie changed the title ~~add sqlite pragma settings~~ add sqlite pragma settings to sql_storage.py - change distinct to aggregate in mongodb.py Aug 9, 2017

lesleslie added 2 commits August 9, 2017 23:52

increase sort buffer to 42M

16beccd

converation --> conversation

a86df43

vkosuri reviewed Aug 10, 2017

View reviewed changes

This was referenced Aug 10, 2017

Feedback example #920

Closed

bot.get_response(input_msg) is very slow when used to get response from mongodb #697

Closed

lesleslie added 9 commits August 10, 2017 21:48

add mock to requirements.txt - needed for tests

17f0a1b

add package requirements for testing, bump pymongo version to 3.4

01a43cc

remove those last requirements - found dev-requirements.txt

73295bc

and again for coverage

f844bbf

self.client.admin.command({'setFeatureCompatibilityVersion': '3.4'})

a58f3e0

self.client.admin.command({'setFeatureCompatibilityVersion': '3.2'})

bfbadc4

back out few commits - see travis errors again

5066b2c

bump pymongo to 3.4

0ba10a6

reset pymongo version to 3.3, import OperationFailure

0346efd

vkosuri reviewed Aug 11, 2017

View reviewed changes

DEFAULT_SESSION_ID --> DEFAULT_CONVERSATION_ID

11272e9

gunthercox approved these changes Aug 11, 2017

View reviewed changes

gunthercox merged commit 2ff9a2c into gunthercox:master Aug 11, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add sqlite pragma settings to sql_storage.py - change distinct to aggregate in mongodb.py #916

add sqlite pragma settings to sql_storage.py - change distinct to aggregate in mongodb.py #916

lesleslie commented Aug 9, 2017

lesleslie commented Aug 9, 2017

lesleslie commented Aug 9, 2017 •

edited

Loading

vkosuri Aug 10, 2017

gunthercox commented Aug 10, 2017

lesleslie commented Aug 11, 2017 via email

lesleslie commented Aug 11, 2017 via email

lesleslie commented Aug 11, 2017

vkosuri Aug 11, 2017

lesleslie Aug 11, 2017

vkosuri Aug 11, 2017

lesleslie Aug 11, 2017

gunthercox Aug 10, 2017

gunthercox commented Aug 11, 2017

add sqlite pragma settings to sql_storage.py - change distinct to aggregate in mongodb.py #916

add sqlite pragma settings to sql_storage.py - change distinct to aggregate in mongodb.py #916

Conversation

lesleslie commented Aug 9, 2017

lesleslie commented Aug 9, 2017

lesleslie commented Aug 9, 2017 • edited Loading

vkosuri Aug 10, 2017

Choose a reason for hiding this comment

gunthercox commented Aug 10, 2017

lesleslie commented Aug 11, 2017 via email

lesleslie commented Aug 11, 2017 via email

lesleslie commented Aug 11, 2017

vkosuri Aug 11, 2017

Choose a reason for hiding this comment

lesleslie Aug 11, 2017

Choose a reason for hiding this comment

vkosuri Aug 11, 2017

Choose a reason for hiding this comment

lesleslie Aug 11, 2017

Choose a reason for hiding this comment

gunthercox Aug 10, 2017

Choose a reason for hiding this comment

gunthercox commented Aug 11, 2017

lesleslie commented Aug 9, 2017 •

edited

Loading