Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev #67

Merged
merged 48 commits into from
Jun 27, 2023
Merged

Dev #67

merged 48 commits into from
Jun 27, 2023

Commits on Mar 30, 2023

  1. Moving conf env vars to settings.yaml

    @renan-souza
    renan-souza committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    153711b View commit details
    Browse the repository at this point in the history

  2. change resource/settings.yaml

    @renan-souza
    renan-souza committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    eb5a0cd View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2023

  1. Small fixes after running on Summit

    @renan-souza
    renan-souza committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    b3c16fa View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2023

  1. Checkpoint to run dask deepmd

    @renan-souza
    renan-souza committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    d604bf4 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. Checkpoint to run dask

    @renan-souza
    renan-souza committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    c86c52d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' of github.com:ORNL/flowcept into dev

    @renan-souza
    renan-souza committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    1ca9374 View commit details
    Browse the repository at this point in the history

Commits on May 4, 2023

  1. Checkpoint for extra metadata

    @renan-souza
    renan-souza committed May 4, 2023
    Configuration menu
    Copy the full SHA
    6f207a2 View commit details
    Browse the repository at this point in the history

  2. checkpoint from summit

    Renan Souza committed May 4, 2023
    Configuration menu
    Copy the full SHA
    01baced View commit details
    Browse the repository at this point in the history

  3. Checkpoint after merge

    @renan-souza
    renan-souza committed May 4, 2023
    Configuration menu
    Copy the full SHA
    a176ca1 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2023

  1. Update for more flexibility to mlflow capture

    @renan-souza
    renan-souza committed May 8, 2023
    Configuration menu
    Copy the full SHA
    ae417f6 View commit details
    Browse the repository at this point in the history

  2. Improving exception handling in mlflow interception

    @renan-souza
    renan-souza committed May 8, 2023
    Configuration menu
    Copy the full SHA
    e14a8fe View commit details
    Browse the repository at this point in the history

  3. checkpoint mlflow

    @renan-souza
    renan-souza committed May 8, 2023
    Configuration menu
    Copy the full SHA
    cd0e3ae View commit details
    Browse the repository at this point in the history

Commits on May 10, 2023

  1. Updates

    @renan-souza
    renan-souza committed May 10, 2023
    Configuration menu
    Copy the full SHA
    b7bc0b2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' of github.com:ORNL/flowcept into dev

    @renan-souza
    renan-souza committed May 10, 2023
    Configuration menu
    Copy the full SHA
    62fb266 View commit details
    Browse the repository at this point in the history

  3. Moving dask_scheduler_setup to resources

    @renan-souza
    renan-souza committed May 10, 2023
    Configuration menu
    Copy the full SHA
    4271587 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2023

  1. Major changes to buffer Redis insertions

    @renan-souza
    renan-souza committed May 15, 2023
    Configuration menu
    Copy the full SHA
    e85f7c7 View commit details
    Browse the repository at this point in the history

  2. Adding while to doc inserter

    @renan-souza
    renan-souza committed May 15, 2023
    Configuration menu
    Copy the full SHA
    6330f5e View commit details
    Browse the repository at this point in the history

  3. changes in db inserter to avoid statuses inconsistencies

    @renan-souza
    renan-souza committed May 15, 2023
    Configuration menu
    Copy the full SHA
    cd6d843 View commit details
    Browse the repository at this point in the history

  4. Improving logging

    @renan-souza
    renan-souza committed May 15, 2023
    Configuration menu
    Copy the full SHA
    b0a0cb6 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2023

  1. debugging doc upsert

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    610374e View commit details
    Browse the repository at this point in the history

  2. debugging doc insert

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    b378650 View commit details
    Browse the repository at this point in the history

  3. Adding locks to flush operations

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    eafdeb2 View commit details
    Browse the repository at this point in the history

  4. Debug msgs

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    4919c52 View commit details
    Browse the repository at this point in the history

  5. adding sleep time

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    0be17af View commit details
    Browse the repository at this point in the history

  6. adding long sleep time in stop. Worked with small case

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    d6deefa View commit details
    Browse the repository at this point in the history

  7. removing debug msg

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    2beb5f0 View commit details
    Browse the repository at this point in the history

  8. Fixing sleep time for consumer.stopg

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    3c0d9a5 View commit details
    Browse the repository at this point in the history

  9. https://github.com/ORNL/flowcept/issues/66

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    8284296 View commit details
    Browse the repository at this point in the history

  10. https://github.com/ORNL/flowcept/issues/66

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    19591ba View commit details
    Browse the repository at this point in the history

  11. Adding perf logging + task_id index

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    7edb41f View commit details
    Browse the repository at this point in the history

  12. Improve logging

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    75de307 View commit details
    Browse the repository at this point in the history

  13. hostname to logging

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    accaf44 View commit details
    Browse the repository at this point in the history

  14. fix to perf logging

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    643f1b8 View commit details
    Browse the repository at this point in the history

  15. Adding submission_time to dask interceptor and time at insertion to m… 

    …ongo insertion
    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    9e976ad View commit details
    Browse the repository at this point in the history

  16. adding adaptive buffer in mongo insertion

    @renan-souza
    renan-souza committed May 16, 2023
    Configuration menu
    Copy the full SHA
    1879ce3 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2023

  1. Improving adaptive buffer size

    @renan-souza
    renan-souza committed May 17, 2023
    Configuration menu
    Copy the full SHA
    cd3f906 View commit details
    Browse the repository at this point in the history

  2. adding if len check into lock() block

    @renan-souza
    renan-souza committed May 17, 2023
    Configuration menu
    Copy the full SHA
    70c03e2 View commit details
    Browse the repository at this point in the history

  3. Major changes to allow a graceful stop without too much sleep time 

    This is the sequence.

0. User starts the interceptors using the API -- This resets the number of running MQ consumption threads
1. Each interceptor increments a counter of consumption threads
2. After execution, user sends a "master stop" command using the API
3. Each MQ thread executes a stop command, flushes the remainder of the buffer, and sends a stop event to the queue
4. The Document Inserter receives these events and counts. When the number of stop events coming from the MQ threads is equal to the number of MQ threads, the DocInserter sends a stop command.
5. The stop command flushes all buffers and close.
    @renan-souza
    renan-souza committed May 17, 2023
    Configuration menu
    Copy the full SHA
    844bfe1 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. Minor fixes for better terminology

    @renan-souza
    renan-souza committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    0e54ab3 View commit details
    Browse the repository at this point in the history

  2. code format

    @renan-souza
    renan-souza committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    8381233 View commit details
    Browse the repository at this point in the history

  3. Enhancing MD files

    @renan-souza
    renan-souza committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    6e3e8c3 View commit details
    Browse the repository at this point in the history

  4. running tests

    @renan-souza
    renan-souza committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    9614ebc View commit details
    Browse the repository at this point in the history

  5. Fixing pip install

    @renan-souza
    renan-souza committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    bdf38b3 View commit details
    Browse the repository at this point in the history

  6. Changing tests

    @renan-souza
    renan-souza committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    ec6e64d View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2023

  1. Notebooks test on hold

    @renan-souza
    renan-souza committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    f358c36 View commit details
    Browse the repository at this point in the history

  2. Testing notebooks

    @renan-souza
    renan-souza committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    c054c95 View commit details
    Browse the repository at this point in the history

  3. Fixing MLFLow notebook

    @renan-souza
    renan-souza committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    fc8104e View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into dev

    @renan-souza
    renan-souza authored Jun 27, 2023
    Configuration menu
    Copy the full SHA
    d4c6eb0 View commit details
    Browse the repository at this point in the history