Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

062 scale up #68

Merged
merged 43 commits into from
May 24, 2020
Merged

062 scale up #68

merged 43 commits into from
May 24, 2020

Commits on Apr 25, 2020

  1. create node pool CRUD

    @rivernews
    rivernews committed Apr 25, 2020
    Configuration menu
    Copy the full SHA
    36acce0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into 062_scale_up

    @rivernews
    rivernews committed Apr 25, 2020
    Configuration menu
    Copy the full SHA
    ef67cf5 View commit details
    Browse the repository at this point in the history

  3. change auth method from travis to slack

    @rivernews
    rivernews committed Apr 25, 2020
    Configuration menu
    Copy the full SHA
    344c1a0 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2020

  1. implement elastic scaling scraper worker node and selenium microservi… 

    …ce; expose API
    @rivernews
    rivernews committed Apr 29, 2020
    Configuration menu
    Copy the full SHA
    f88edc9 View commit details
    Browse the repository at this point in the history

  2. change to provision scraper job on worker node

    @rivernews
    rivernews committed Apr 29, 2020
    Configuration menu
    Copy the full SHA
    15433e2 View commit details
    Browse the repository at this point in the history

  3. return delete response array for clean node API

    @rivernews
    rivernews committed Apr 29, 2020
    Configuration menu
    Copy the full SHA
    bf6d757 View commit details
    Browse the repository at this point in the history

  4. support configurable node size

    @rivernews
    rivernews committed Apr 29, 2020
    Configuration menu
    Copy the full SHA
    9334bcb View commit details
    Browse the repository at this point in the history

  5. switch back bull scraper job run on primary node pool since scraper w… 

    …orker node often has memory pressure AND primary node pool now use larger 4G one so has lots of capacity
    @rivernews
    rivernews committed Apr 29, 2020
    Configuration menu
    Copy the full SHA
    78e9031 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2020

  1. take safe - just 3 concurrent jobs; 5 sec / job dispathing, slow but … 

    …should be steady
    @rivernews
    rivernews committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    0ba6f3c View commit details
    Browse the repository at this point in the history

  2. change to use scraper node for k8s job; add more node size options; i… 

    …ncrease redis timeout
    @rivernews
    rivernews committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    6f673e0 View commit details
    Browse the repository at this point in the history

  3. let k8s job be in same namespace with selenium for easy clean up

    @rivernews
    rivernews committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    489b603 View commit details
    Browse the repository at this point in the history

  4. maximize capacity based on benchmark result

    @rivernews
    rivernews committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    be10946 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2020

  1. cancel s3 dispatch delay for better debugging

    @rivernews
    rivernews committed May 1, 2020
    Configuration menu
    Copy the full SHA
    735acc5 View commit details
    Browse the repository at this point in the history

  2. remove s3 delay dispatching job

    @rivernews
    rivernews committed May 1, 2020
    Configuration menu
    Copy the full SHA
    fd65c23 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2020

  1. add s3 job dispatching delay; disable travis job since redis keeps ti… 

    …meout
    @rivernews
    rivernews committed May 3, 2020
    Configuration menu
    Copy the full SHA
    13f4df3 View commit details
    Browse the repository at this point in the history

  2. add firewall tag to scraper worker nodes and see if this can mitigate… 

    … redis timed out issue
    @rivernews
    rivernews committed May 3, 2020
    Configuration menu
    Copy the full SHA
    7888612 View commit details
    Browse the repository at this point in the history

  3. add cpu opt node size; add travis but just one

    @rivernews
    rivernews committed May 3, 2020
    Configuration menu
    Copy the full SHA
    3621f5f View commit details
    Browse the repository at this point in the history

  4. increase delay

    @rivernews
    rivernews committed May 3, 2020
    Configuration menu
    Copy the full SHA
    950907e View commit details
    Browse the repository at this point in the history

  5. add http cors too when https not available

    @rivernews
    rivernews committed May 3, 2020
    Configuration menu
    Copy the full SHA
    bbad80d View commit details
    Browse the repository at this point in the history

Commits on May 10, 2020

  1. scalable selenium - using hub - node

    @rivernews
    rivernews committed May 10, 2020
    Configuration menu
    Copy the full SHA
    1af4096 View commit details
    Browse the repository at this point in the history

  2. let node scaling more configurable to better prepare for large scale up

    @rivernews
    rivernews committed May 10, 2020
    Configuration menu
    Copy the full SHA
    d59adc1 View commit details
    Browse the repository at this point in the history

  3. simplify s3 job workflow, remove redundant concurrency checks

    @rivernews
    rivernews committed May 10, 2020
    Configuration menu
    Copy the full SHA
    f1c7e11 View commit details
    Browse the repository at this point in the history

  4. let hub be on primary node pool since it does not scale

    @rivernews
    rivernews committed May 10, 2020
    Configuration menu
    Copy the full SHA
    3f000b8 View commit details
    Browse the repository at this point in the history

Commits on May 11, 2020

  1. add pod-standalone mode

    @rivernews
    rivernews committed May 11, 2020
    Configuration menu
    Copy the full SHA
    a5adc35 View commit details
    Browse the repository at this point in the history

  2. use sidecar container in k8 so we don't have to delete job and can ch… 

    …eck back log if error occur in k8s job
    @rivernews
    rivernews committed May 11, 2020
    Configuration menu
    Copy the full SHA
    95b4281 View commit details
    Browse the repository at this point in the history

  3. simplify log for fetch response; solve problem of s3 job, some superv… 

    …isor job not resolved when manually terminated
    @rivernews
    rivernews committed May 11, 2020
    Configuration menu
    Copy the full SHA
    7fc8bcf View commit details
    Browse the repository at this point in the history

Commits on May 15, 2020

  1. let s3 be able to stop dispatching immediately after termination order

    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    00434f7 View commit details
    Browse the repository at this point in the history

  2. decrease slack sending freq in scraper jobs; use try-catch block for … 

    …slack request at the end of s3 job
    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    91640ea View commit details
    Browse the repository at this point in the history

  3. add comment

    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    6f1b5f4 View commit details
    Browse the repository at this point in the history

  4. let s3 job auto scale up

    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    86aa260 View commit details
    Browse the repository at this point in the history

  5. let s3 job wait for all supervisor job to end including manually re-t… 

    …ry ones, before scaling down
    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    8c45c53 View commit details
    Browse the repository at this point in the history

  6. patch additional change

    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    40bc3f7 View commit details
    Browse the repository at this point in the history

  7. fix dispatch interval not using configuration value

    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    d12ee9d View commit details
    Browse the repository at this point in the history

  8. larger polling interval for node pool

    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    c849141 View commit details
    Browse the repository at this point in the history

  9. avoid mem over allocating on the same node

    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    dd4c794 View commit details
    Browse the repository at this point in the history

  10. conservative approach; avoid too many job being assigned on one node

    @rivernews
    rivernews committed May 15, 2020
    Configuration menu
    Copy the full SHA
    071b751 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2020

  1. implement node pool semaphore

    @rivernews
    rivernews committed May 16, 2020
    Configuration menu
    Copy the full SHA
    5dbd3de View commit details
    Browse the repository at this point in the history

Commits on May 18, 2020

  1. implement anti-affinity ourself and make sure no job memory utilizati… 

    …on overlap and memory spikes occur
    @rivernews
    rivernews committed May 18, 2020
    Configuration menu
    Copy the full SHA
    2fd19cc View commit details
    Browse the repository at this point in the history

  2. unlobck test tempararily

    @rivernews
    rivernews committed May 18, 2020
    Configuration menu
    Copy the full SHA
    4aad87d View commit details
    Browse the repository at this point in the history

Commits on May 19, 2020

  1. testing for SLK scalability

    @rivernews
    rivernews committed May 19, 2020
    Configuration menu
    Copy the full SHA
    e6e00f4 View commit details
    Browse the repository at this point in the history

  2. try to normalize job concurrency under multi replica situation

    @rivernews
    rivernews committed May 19, 2020
    Configuration menu
    Copy the full SHA
    6e55b47 View commit details
    Browse the repository at this point in the history

  3. try to factor in replica to realize the correct concurrency

    @rivernews
    rivernews committed May 19, 2020
    Configuration menu
    Copy the full SHA
    ec6b9b5 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2020

  1. send slack message after s3 job finalizer completes

    @rivernews
    rivernews committed May 23, 2020
    Configuration menu
    Copy the full SHA
    fba3697 View commit details
    Browse the repository at this point in the history