Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge branch 'boltdb' into dev #2562

Merged
merged 22 commits into from
Aug 12, 2020
Merged

Merge branch 'boltdb' into dev #2562

merged 22 commits into from
Aug 12, 2020

Commits on Jun 17, 2020

  1. Configuration menu
    Copy the full SHA
    64a8953 View commit details
    Browse the repository at this point in the history
  2. Added new data interface and dummy implementation.

    Added the new data interface for data management in agent.
    Added dummy boltdb implementation of the interface and its initialization.
    fenxiong committed Jun 17, 2020
    Configuration menu
    Copy the full SHA
    cce6787 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2020

  1. Implemented task/container related methods for boltdb data client.

    Details:
    * data: implemented SaveContainer, SaveDockerContainer, DelContainer, GetContainers, SaveTask, DelTask, GetTasks. Added a few helper functions to cover common boltdb interaction.
    * api/container: added a new field TaskARN to allow easier generation of key when saving a container to db. This field will be populated with correct value in PostUnmarshalTask in a later code change.
    * utils: added a helper function to get task id from task arn.
    fenxiong committed Jun 18, 2020
    Configuration menu
    Copy the full SHA
    b72826b View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2020

  1. Save task and container data to boltdb.

    Changes are made in following packages:
    * acs/handler: save task to boltdb after adding the task to task engine;
    * api/task: populate task arn to container in PostUnmarshalTask. this is used to generate database key when saving the container;
    * app/agent: initialize boltdb data client and pass it to task engine, acs handler and event handler;
    * data: added a no-op client which is used in testing and when ECS_CHECKPOINT is set to false;
    * engine:
      - added a file data.go which covers interaction with boltdb;
      - task engine: remove task and containers data from database when cleaning up the task; added a method SetDataClient to set the client similar to how the state manager was set;
      - task manager: save task to boltdb when its desired/known status changes and when resource known status changes; save container to boltdb when its desired/known status changes and when updating its metadata. these are done in handleDesiredStatusChange, handleContainerChange and handleResourceStateChange in task_manager.go;
    * eventhandler: save task/container in boltdb after updating their sent status.
    fenxiong committed Jun 25, 2020
    Configuration menu
    Copy the full SHA
    c6c4ea6 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2020

  1. Implement metadata related methods and save agent metadata to boltdb.

    Save various metadata to the metadata bucket in boltdb. Details:
    * acs/handler: made changes to save task manifest seq num to boltdb;
    * app: made changes to save agent version, availability zone, cluster name, container instance arn and ec2 instance id to boltdb; removed a redundant unit test TestDoStartHappyPath from agent_unix_test.go as it is covered by TestDoStartRegisterAvailabilityZone in agent_test.go which is basically the same, and renamed the latter as TestDoStartHappyPath;
    * data: implemented SaveMetadata and GetMetadata.
    fenxiong committed Jun 29, 2020
    Configuration menu
    Copy the full SHA
    1d46a83 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2020

  1. Added boltdb implementation to get, update and delete image state. Up…

    …dated ImageManager to use data Client instead of state manager to persist image states.
    mythri-garaga committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    0a1b4e3 View commit details
    Browse the repository at this point in the history
  2. Add and save task local ip address in task struct.

    For a task in awsvpc network mode, the task engine state holds a mapping between the task's local ip address and the task, and the mapping is saved as part of the state via state manager. With migration to boltdb, this mapping is not saved. So to maintain this information in boltdb, the ip address is added as a field of the task struct and it is saved together with the task.
    fenxiong committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    8f6c9b7 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2020

  1. Added boltdb implementation to get, update and delete eni attachments…

    …. Updated agent to use data Client instead of state manager to persist eni attachment data
    mythri-garaga committed Jul 10, 2020
    Configuration menu
    Copy the full SHA
    8e4f86b View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2020

  1. Configuration menu
    Copy the full SHA
    bb836cc View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2020

  1. Configuration menu
    Copy the full SHA
    a59d9ea View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2020

  1. Load data from boltdb upon startup.

    Implemented logic for loading data from boltdb upon startup, while preseving backward compatibility by falling back to loading from state file. Details:
    * app:
     - In data.go, implement method `loadData` that loads data from previous data file, either boltdb or state file. In the later case, data is migrated to boltdb after loading. Behavior of three cases are considered:
        1. Agent starts from fresh instance (no previous state):
           (1) Try to load from boltdb, get nothing;
           (2) Try to load from state file, get nothing;
           (3) Return empty data.
    
        2. Agent starts with previous state stored in boltdb:
           (1) Try to load from boltdb, get the data;
           (2) Return loaded data.
    
        3. Agent starts with previous state stored in state file (i.e. it was just upgraded from an old agent that uses state file):
           (1) Try to load from boltdb, get nothing;
           (2) Try to load from state file, get something;
           (3) Save loaded data to boltdb;
           (4) Return loaded data.
    
     - In agent.go, invoke `loadData` method to load data, replacing the existing few lines of code that uses state manager to load data.
     - Update a few unit tests to use actual task engine state instead of mock one because with the changes, it would be tedious to list all the expected calls to the engine state.
    * engine: added a method SaveState which saves the whole task engine state to boltdb.
    fenxiong committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    40e4277 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0209f31 View commit details
    Browse the repository at this point in the history
  3. Merge pull request aws#2532 from fenxiong/boltdb-merge

    Merge branch 'dev' into boltdb
    fenxiong committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    000f6e4 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2020

  1. Remove usages of state manager in acs and event handler.

    Also added logic to save attachment sent status as part of task state change, in agent/eventhandler/task_handler.go.
    fenxiong committed Jul 22, 2020
    Configuration menu
    Copy the full SHA
    482fd46 View commit details
    Browse the repository at this point in the history
  2. Add more unit tests to api/eni package.

    Previous commit lowers the test coverage by 0.1% without obvious reason. Raising code coverage in a poorly covered package instead.
    fenxiong committed Jul 22, 2020
    Configuration menu
    Copy the full SHA
    6bd4e0f View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2020

  1. Configuration menu
    Copy the full SHA
    0f72765 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2020

  1. Configuration menu
    Copy the full SHA
    ce953c6 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2020

  1. Configuration menu
    Copy the full SHA
    cbb6d2f View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2020

  1. Merge pull request aws#2553 from fenxiong/boltdb-merge

     Merge branch 'dev' into boltdb
    fenxiong committed Aug 4, 2020
    Configuration menu
    Copy the full SHA
    ca26790 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    876425a View commit details
    Browse the repository at this point in the history
  3. Merge pull request aws#2552 from fenxiong/boltdb-update

    Remove unnecessary db saves and use batch for db update
    fenxiong committed Aug 4, 2020
    Configuration menu
    Copy the full SHA
    b196631 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2020

  1. Configuration menu
    Copy the full SHA
    ad61341 View commit details
    Browse the repository at this point in the history