NSFS | NC | IAM service - root accounts manager (root account operati…

…ons using the IAM API)

1. Add more properties to nsfs_account_schema (not required):
   - iam_operate_on_root_account = boolean (if an account was set with true. then it is a roots accounts manager).
2. Add the option of iam_operate_on_root_account in noobaa cli.
3. Edit all the functions in AccountSpaceFS (CRUD) to operate on root accounts when the requesting account is a root accounts manager.

Signed-off-by: shirady <57721533+shirady@users.noreply.github.com>

docs/design/iam.md

@@ -88,3 +88,18 @@ Source: AccessKeys
   - AccessKey (Create, Update, Delete, List)
     - root account
     - all IAM users only for themselves (except the first creation that can be done only by the root account).
+
+### Root Accounts Manager
+The root accounts managers are a solution for creating root accounts using the IAM API.
+
+- The root accounts managers will be created only using the CLI (can have more than one root account manager).
+- It is not mandatory to have a root account manager, it is only for allowing the IAM API for creating new root accounts, but this account does not owns the root accounts.
+- The root accounts manager functionality is like root account in the IAM API perspective:
+  - We use root accounts to create IAM users: We use root accounts manager to create root accounts
+  - We use root accounts to create the first access key of an IAM user: We use root accounts manager to create the first access key of a root account.
+- When using IAM users API:
+  - root accounts manager can run IAM users create/update/delete/list - only on root accounts (not on other IAM users).
+root accounts manager can run IAM access keys create/update/delete/list - only on root accounts and himself.
+
+Here attached a diagram with all the accounts that we have in our system:
+![All accounts diagram](https://github.com/noobaa/noobaa-core/assets/57721533/c4395c06-3ab3-4425-838b-c020ef7cc38a)

docs/dev_guide/nc_nsfs_iam_developer_doc.md

@@ -13,7 +13,7 @@ This will be the argument for:
   - `new_buckets_path` flag  `/tmp/nsfs_root1` (that we will use in the account commands)
   - `path` in the buckets commands `/tmp/nsfs_root1/my-bucket` (that we will use in bucket commands).
 2. Create the root user account with the CLI:
-`sudo node src/cmd/manage_nsfs account add --name <name>> --new_buckets_path /tmp/nsfs_root1 --access_key <access-key> --secret_key <secret-key> --uid <uid> --gid <gid>`.
+`sudo node src/cmd/manage_nsfs account add --name <name> --new_buckets_path /tmp/nsfs_root1 --access_key <access-key> --secret_key <secret-key> --uid <uid> --gid <gid>`.
 3. Start the NSFS server (using debug mode and the port for IAM): `sudo node src/cmd/nsfs --debug 5 --https_port_iam 7005`
 Note: before starting the server please add this line: `process.env.NOOBAA_LOG_LEVEL = 'nsfs';` in the endpoint.js (before the condition `if (process.env.NOOBAA_LOG_LEVEL) {`)
 4. Create the alias for IAM service: 
@@ -40,7 +40,7 @@ Create the alias for IAM service for the user that was created (with its access
 1. Use the root account credentials to create a user: `nc-user-1-iam iam create-user --user-name <username>`
 2. Use the root account credentials to create access keys for the user: `nc-user-1-iam iam create-access-key --user-name <username>`
 3. The alias for s3 service: `alias nc-user-1-s3-regular='AWS_ACCESS_KEY_ID=<access-key> AWS_SECRET_ACCESS_KEY=<secret-key> aws --no-verify-ssl --endpoint-url https://localhost:6443'` 
-2. Create a bucket (so we can list it) `nc-user-1-s3-regular s3 mb s3://<bucket-name>>`
+2. Create a bucket (so we can list it) `nc-user-1-s3-regular s3 mb s3://<bucket-name`
 3. List bucket (use s3 service)`nc-user-1-s3-regular s3 ls`
 4. List access keys (use IAM service) `nc-user-1-iam-regular iam list-access-keys`
 5. Deactivate access keys: `nc-user-1-iam iam update-access-key --access-key-id <access-key> --user-name <username> --status Inactive`
@@ -51,4 +51,13 @@ Note: Currently we clean the cache after update, but it happens for the specific
 1. Use the root account credentials to create a user: `nc-user-1-iam iam create-user --user-name <username>` (You should see the config file in under the accounts directory).
 2. Use the root account credentials to create access keys for the user:(first time): `nc-user-1-iam iam create-access-key --user-name <username>` (You should see the first symbolic link in under the access_keys directory).
 3. Use the root account credentials to create access keys for the user (second time): `nc-user-1-iam iam create-access-key --user-name <username>` (You should see the second symbolic link in under the access_keys directory).
-4. Update the username: `nc-user-1-iam iam update-user --user-name <username> --new-user-name <new-username>` (You should see the following changes: config file name updated, symlinks updated according to the current config).
+4. Update the username: `nc-user-1-iam iam update-user --user-name <username> --new-user-name <new-username>` (You should see the following changes: config file name updated, symlinks updated according to the current config).
+
+#### Create root account using the IAM API (requesting account is root accounts manager):
+1. Create the root accounts manager with the CLI:
+`sudo node src/cmd/manage_nsfs account add --name <name> --new_buckets_path /tmp/nsfs_root1 --access_key <access-key> --secret_key <secret-key> --uid <uid> --gid <gid> --iam_operate_on_root_account`.
+2. Use the root accounts manager details in the alias:
+`alias nc-user-manager-iam='AWS_ACCESS_KEY_ID=<access-key> AWS_SECRET_ACCESS_KEY=<secret-key> aws --no-verify-ssl --endpoint-url https://localhost:7005'`.
+3. Use the root accounts manager account credentials to create a root account:
+ `nc-user-manager-iam create-user --user-name <username>`
+4. Use the root account credentials to create access keys for the root account: `nc-user-manager-iam iam create-access-key --user-name <username>`

src/cmd/manage_nsfs.js

@@ -341,6 +341,8 @@ async function fetch_account_data(action, user_input) {
         new_access_key,
         access_keys,
         force_md5_etag: _.isUndefined(user_input.force_md5_etag) || user_input.force_md5_etag === '' ? user_input.force_md5_etag : get_boolean_or_string_value(user_input.force_md5_etag),
+        iam_operate_on_root_account: _.isUndefined(user_input.iam_operate_on_root_account) ?
+                undefined : get_boolean_or_string_value(user_input.iam_operate_on_root_account),
         nsfs_account_config: {
             distinguished_name: user_input.user,
             uid: user_input.user ? undefined : user_input.uid,
@@ -371,7 +373,6 @@ async function fetch_account_data(action, user_input) {
     data.nsfs_account_config.new_buckets_path = data.nsfs_account_config.new_buckets_path || undefined;
     // force_md5_etag deletion specified with empty string '' checked against user_input.force_md5_etag because data.force_md5_etag is boolean
     data.force_md5_etag = data.force_md5_etag === '' ? undefined : data.force_md5_etag;
-    // allow_bucket_creation either set by user or infer from new_buckets_path
     if (_.isUndefined(user_input.allow_bucket_creation)) {
         data.allow_bucket_creation = !_.isUndefined(data.nsfs_account_config.new_buckets_path);
     } else if (typeof user_input.allow_bucket_creation === 'boolean') {
@@ -421,7 +422,7 @@ async function fetch_existing_account_data(action, target, decrypt_secret_key) {
 }
 
 async function add_account(data) {
-    await manage_nsfs_validations.validate_account_args(data, ACTIONS.ADD);
+    await manage_nsfs_validations.validate_account_args(data, ACTIONS.ADD, config_root_backend, accounts_dir_path, undefined);
 
     const fs_context = native_fs_utils.get_process_fs_context(config_root_backend);
     const access_key = has_access_keys(data.access_keys) ? data.access_keys[0].access_key : undefined;
@@ -456,8 +457,9 @@ async function add_account(data) {
 }
 
 
-async function update_account(data) {
-    await manage_nsfs_validations.validate_account_args(data, ACTIONS.UPDATE);
+async function update_account(data, is_flag_iam_operate_on_root_account) {
+    await manage_nsfs_validations.validate_account_args(data, ACTIONS.UPDATE,
+        config_root_backend, accounts_dir_path, is_flag_iam_operate_on_root_account);
 
     const fs_context = native_fs_utils.get_process_fs_context(config_root_backend);
     const cur_name = data.name;
@@ -521,7 +523,7 @@ async function update_account(data) {
 }
 
 async function delete_account(data) {
-    await manage_nsfs_validations.validate_account_args(data, ACTIONS.DELETE);
+    await manage_nsfs_validations.validate_account_args(data, ACTIONS.DELETE, config_root_backend, accounts_dir_path, undefined);
     await manage_nsfs_validations.validate_delete_account(config_root_backend, buckets_dir_path, data.name);
 
     const fs_context = native_fs_utils.get_process_fs_context(config_root_backend);
@@ -535,7 +537,7 @@ async function delete_account(data) {
 }
 
 async function get_account_status(data, show_secrets) {
-    await manage_nsfs_validations.validate_account_args(data, ACTIONS.STATUS);
+    await manage_nsfs_validations.validate_account_args(data, ACTIONS.STATUS, config_root_backend, accounts_dir_path, undefined);
     try {
         const account_path = _.isUndefined(data.name) ?
             get_symlink_config_file_path(access_keys_dir_path, data.access_keys[0].access_key) :
@@ -559,7 +561,8 @@ async function manage_account_operations(action, data, show_secrets, user_input)
     } else if (action === ACTIONS.STATUS) {
         await get_account_status(data, show_secrets);
     } else if (action === ACTIONS.UPDATE) {
-        await update_account(data);
+        const is_flag_iam_operate_on_root_account = get_boolean_or_string_value(user_input.iam_operate_on_root_account);
+        await update_account(data, is_flag_iam_operate_on_root_account);
     } else if (action === ACTIONS.DELETE) {
         await delete_account(data);
     } else if (action === ACTIONS.LIST) {

src/manage_nsfs/manage_nsfs_cli_errors.js

@@ -198,6 +198,20 @@ ManageCLIError.AccountDeleteForbiddenHasBuckets = Object.freeze({
     http_code: 403,
 });
 
+ManageCLIError.AccountCannotCreateRootAccountsRequesterIAMUser = Object.freeze({
+    code: 'AccountCannotCreateRootAccounts',
+    message: 'Cannot update account to have iam_operate_on_root_account. ' +
+        'You must use root account for this action',
+    http_code: 409,
+});
+
+ManageCLIError.AccountCannotBeRootAccountsManager = Object.freeze({
+    code: 'AccountCannotBeRootAccountsManager',
+    message: 'Cannot update account to have iam_operate_on_root_account. ' +
+        'You must delete all IAM accounts before update or ' +
+        'use root accounts that does not owns any IAM accounts',
+    http_code: 409,
+});
 
 //////////////////////////////////
 //// ACCOUNT ARGUMENTS ERRORS ////

src/manage_nsfs/manage_nsfs_cli_utils.js

@@ -142,6 +142,17 @@ function generate_id() {
     return mongo_utils.mongoObjectId();
 }
 
+/**
+ * check_root_account_owns_user checks if an account is owned by root account
+ * @param {object} root_account
+ * @param {object} account
+ */
+function check_root_account_owns_user(root_account, account) {
+    if (account.owner === undefined) return false;
+    return root_account._id === account.owner;
+}
+
+
 // EXPORTS
 exports.throw_cli_error = throw_cli_error;
 exports.write_stdout_response = write_stdout_response;
@@ -154,3 +165,4 @@ exports.get_options_from_file = get_options_from_file;
 exports.has_access_keys = has_access_keys;
 exports.generate_id = generate_id;
 exports.set_debug_level = set_debug_level;
+exports.check_root_account_owns_user = check_root_account_owns_user;

src/manage_nsfs/manage_nsfs_constants.js

@@ -35,8 +35,8 @@ const FROM_FILE = 'from_file';
 const ANONYMOUS = 'anonymous';
 
 const VALID_OPTIONS_ACCOUNT = {
-    'add': new Set(['name', 'uid', 'gid', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', FROM_FILE, ...GLOBAL_CONFIG_OPTIONS]),
-    'update': new Set(['name', 'uid', 'gid', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'new_name', 'regenerate', ...GLOBAL_CONFIG_OPTIONS]),
+    'add': new Set(['name', 'uid', 'gid', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', FROM_FILE, ...GLOBAL_CONFIG_OPTIONS]),
+    'update': new Set(['name', 'uid', 'gid', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', 'new_name', 'regenerate', ...GLOBAL_CONFIG_OPTIONS]),
     'delete': new Set(['name', ...GLOBAL_CONFIG_OPTIONS]),
     'list': new Set(['wide', 'show_secrets', 'gid', 'uid', 'user', 'name', 'access_key', ...GLOBAL_CONFIG_OPTIONS]),
     'status': new Set(['name', 'access_key', 'show_secrets', ...GLOBAL_CONFIG_OPTIONS]),
@@ -91,6 +91,7 @@ const OPTION_TYPE = {
     fs_backend: 'string',
     allow_bucket_creation: 'boolean',
     force_md5_etag: 'boolean',
+    iam_operate_on_root_account: 'boolean',
     config_root: 'string',
     from_file: 'string',
     config_root_backend: 'string',
@@ -113,7 +114,7 @@ const OPTION_TYPE = {
 
 const BOOLEAN_STRING_VALUES = ['true', 'false'];
 const BOOLEAN_STRING_OPTIONS = new Set(['allow_bucket_creation', 'regenerate', 'wide', 'show_secrets', 'force',
-    'force_md5_etag', 'all_account_details', 'all_bucket_details', 'anonymous']);
+    'force_md5_etag', 'iam_operate_on_root_account', 'all_account_details', 'all_bucket_details', 'anonymous']);
 
 //options that can be unset using ''
 const LIST_UNSETABLE_OPTIONS = ['fs_backend', 's3_policy', 'force_md5_etag'];

src/manage_nsfs/manage_nsfs_help_utils.js

@@ -80,6 +80,7 @@ Flags:
 --fs_backend <none | GPFS | CEPH_FS | NFSv4>          (optional)        Set the filesystem type of new_buckets_path (default config.NSFS_NC_STORAGE_BACKEND)
 --allow_bucket_creation <true | false>                (optional)        Set the account to explicitly allow or block bucket creation
 --force_md5_etag <true | false>                       (optional)        Set the account to force md5 etag calculation. (unset with '') (will override default config.NSFS_NC_STORAGE_BACKEND)
+--iam_operate_on_root_account <true | false>          (optional)        Set the account to create root accounts instead of IAM users in IAM API requests.
 --from_file <string>                                  (optional)        Use details from the JSON file, there is no need to mention all the properties individually in the CLI
 `;
 
@@ -100,6 +101,7 @@ Flags:
 --fs_backend <none | GPFS | CEPH_FS | NFSv4>          (optional)        Update the filesystem type of new_buckets_path (default config.NSFS_NC_STORAGE_BACKEND)
 --allow_bucket_creation <true | false>                (optional)        Update the account to explicitly allow or block bucket creation
 --force_md5_etag <true | false>                       (optional)        Update the account to force md5 etag calculation (unset with '') (will override default config.NSFS_NC_STORAGE_BACKEND)
+--iam_operate_on_root_account <true | false>          (optional)        Update the account to create root accounts instead of IAM users in IAM API requests.
 `;
 
 const ACCOUNT_FLAGS_DELETE = `

src/manage_nsfs/manage_nsfs_validations.js

@@ -13,7 +13,8 @@ const native_fs_utils = require('../util/native_fs_utils');
 const ManageCLIError = require('../manage_nsfs/manage_nsfs_cli_errors').ManageCLIError;
 const bucket_policy_utils = require('../endpoint/s3/s3_bucket_policy_utils');
 const { throw_cli_error, get_config_file_path, get_bucket_owner_account,
-    get_config_data, get_options_from_file, get_boolean_or_string_value } = require('../manage_nsfs/manage_nsfs_cli_utils');
+    get_config_data, get_options_from_file, get_boolean_or_string_value,
+    check_root_account_owns_user } = require('../manage_nsfs/manage_nsfs_cli_utils');
 const { TYPES, ACTIONS, VALID_OPTIONS, OPTION_TYPE, FROM_FILE, BOOLEAN_STRING_VALUES, BOOLEAN_STRING_OPTIONS,
     GLACIER_ACTIONS, LIST_UNSETABLE_OPTIONS, ANONYMOUS } = require('../manage_nsfs/manage_nsfs_constants');
 
@@ -361,7 +362,8 @@ function validate_account_identifier(action, input_options) {
  * @param {object} data
  * @param {string} action
  */
-async function validate_account_args(data, action) {
+async function validate_account_args(data, action, config_root_backend, accounts_dir_path,
+        is_flag_iam_operate_on_root_account_update_action) {
     if (action === ACTIONS.ADD || action === ACTIONS.UPDATE) {
         if (data.nsfs_account_config.gid && data.nsfs_account_config.uid === undefined) {
             throw_cli_error(ManageCLIError.MissingAccountNSFSConfigUID, data.nsfs_account_config);
@@ -392,6 +394,9 @@ async function validate_account_args(data, action) {
         if (!accessible) {
             throw_cli_error(ManageCLIError.InaccessibleAccountNewBucketsPath, data.nsfs_account_config.new_buckets_path);
         }
+        if (action === ACTIONS.UPDATE && is_flag_iam_operate_on_root_account_update_action) {
+            await validate_root_accounts_manager_update(config_root_backend, accounts_dir_path, data);
+        }
     }
 }
 
@@ -435,6 +440,41 @@ async function validate_delete_account(config_root_backend, buckets_dir_path, ac
     });
 }
 
+    // TODO - when we have the structure of config we can check easily which IAM users are owned by the root account
+    // currently, partial copy from _list_config_files_for_users
+    async function check_if_root_account_does_not_have_IAM_users(config_root_backend, accounts_dir_path, account_to_check) {
+        const fs_context = native_fs_utils.get_process_fs_context(config_root_backend);
+        const entries = await nb_native().fs.readdir(fs_context, accounts_dir_path);
+        await P.map_with_concurrency(10, entries, async entry => {
+            if (entry.name.endsWith('.json')) {
+                const full_path = path.join(accounts_dir_path, entry.name);
+                const account_data = await get_config_data(config_root_backend, full_path);
+                if (entry.name.includes(config.NSFS_TEMP_CONF_DIR_NAME)) return undefined;
+                const is_root_account_owns_user = check_root_account_owns_user(account_to_check, account_data);
+                if (is_root_account_owns_user) {
+                        const detail_msg = `Account ${account_to_check.name} has IAM account ${account_data.name}`;
+                        throw_cli_error(ManageCLIError.AccountCannotBeRootAccountsManager, detail_msg);
+                }
+                return account_data;
+            }
+        });
+    }
+
+/**
+ * validate_root_accounts_manager_update checks that an updated account that was set with iam_operate_on_root_account true:
+ * 1 - is not an IAM user
+ * 2 - the account does not owns IAM users
+ * @param {string} config_root_backend
+ * @param {string} accounts_dir_path
+ * @param {object} account
+ */
+async function validate_root_accounts_manager_update(config_root_backend, accounts_dir_path, account) {
+    if (account.owner) {
+        throw_cli_error(ManageCLIError.AccountCannotCreateRootAccountsRequesterIAMUser);
+    }
+    await check_if_root_account_does_not_have_IAM_users(config_root_backend, accounts_dir_path, account);
+}
+
 ///////////////////////////////////
 //// IP WhITE LIST VALIDATIONS ////
 ///////////////////////////////////
@@ -462,6 +502,7 @@ exports.validate_bucket_args = validate_bucket_args;
 exports.validate_account_args = validate_account_args;
 exports._validate_access_keys = _validate_access_keys;
 exports.validate_delete_account = validate_delete_account;
+exports.validate_root_accounts_manager_update = validate_root_accounts_manager_update;
 exports.validate_whitelist_arg = validate_whitelist_arg;
 exports.validate_whitelist_ips = validate_whitelist_ips;
 exports.validate_flags_combination = validate_flags_combination;