This repository enables you to automate the process of deploying MATLAB® Parallel Server™ and MATLAB Job Scheduler on Amazon® EC2® resources using your Amazon Web Services (AWS®) account.
Use this repository to deploy a compute cluster using compute, storage, and network resources hosted by AWS. For information about the architecture of this solution, see Learn About Cluster Architecture. Use this reference architecture to control every aspect of your cloud resources. Alternatively, for a simpler but less customizable method of deploying a MATLAB Parallel Server cluster in AWS, try MathWorks Cloud Center.
This reference architecture has been reviewed and qualified by AWS.
Before starting, you need the following:
- A MATLAB Parallel Server license. You can use either:
- A MATLAB Parallel Server license configured to use online licensing for MATLAB. For information on how to configure your license for cloud use, see Configure MATLAB Parallel Server Licensing for Cloud Platforms
- A network license manager for MATLAB hosting sufficient MATLAB Parallel Server licenses for your cluster. MathWorks® provides a reference architecture to deploy a suitable Network License Manager for MATLAB on Amazon Web Services or you can use an existing license manager.
- MATLAB and Parallel Computing Toolbox™ on your desktop.
- An AWS account with required permissions.
- A key pair for your AWS account in your chosen region. For instructions on how to create an SSH key pair, see the AWS documentation on Amazon EC2 key pairs.
You are responsible for the cost of the AWS services you use when you create cloud resources using this repository. Resource settings, such as instance type, affects the cost of deployment. For cost estimates, see the pricing pages for each AWS service you use. Prices are subject to change.
To view instructions for deploying the MATLAB Parallel Server reference architecture, select a MATLAB release:
| Linux | Windows |
|---|---|
| R2024b | R2024b |
| R2024a | R2024a |
| R2023b | R2023b |
| R2023a | R2023a |
| R2022b | R2022b |
| R2022a | R2022a |
| R2021b | R2021b |
| R2021a | |
| R2020b | |
| R2020a | |
| R2019b | |
| R2019a_and_older |
This diagram illustrates the cluster architecture created by the template. When you use the AWS CloudFormation templates in this repository, it creates the MATLAB Job Scheduler and the following resources. For more information about each resource, see AWS CloudFormation template reference.
- Security Group (AWS::EC2::SecurityGroup): A security group that defines the ports that are open for ingress to the cluster.
- Security Group Internal Traffic Rule (AWS::EC2::SecurityGroupIngress): A rule that opens access to network traffic between all cluster nodes internally.
- Headnode instance (AWS::EC2::Instance): An EC2 instance for the cluster headnode. The EBS volume attached to this instance stores the job database. The headnode communicates with the clients using a secure SSL connection.
- IAM Role for Cluster Instances (AWS::IAM::Role): A role allowing access to Amazon S3™ from services running in EC2.
- Instance Profile for cluster instances (AWS::IAM::InstanceProfile): A profile for the cluster instances that associates them with the IAM role above.
- Worker Auto Scaling Group (AWS::AutoScaling::AutoScalingGroup): A scaling group for worker instances to be deployed into.
- Worker Launch Configuration (AWS::EC2::LaunchTemplate): A launch template for one or more worker nodes which each run one or more worker MATLAB processes. Clients and workers communicate using a secure SSL connection.
- Cluster S3 Bucket (AWS::S3::Bucket): An S3 bucket that contains the shared secret required to register the workers and establish a secure connection with the job scheduler between the cluster nodes. The shared secret is encrypted in the bucket using server-side encryption. The bucket also contains the cluster profile required to connect to the cluster from the MATLAB client.
- IAM role for deletion of S3 bucket (AWS::IAM::Role): A role that allows an AWS lambda function to empty the S3 bucket during shut down of the cluster. CloudFormation can delete the S3 bucket only if it is empty.
- Lambda function to empty the S3 bucket (AWS::Lambda::Function): A lambda function to empty the S3 bucket created above to allow CloudFormation to successfully delete the S3 bucket when the cluster is shut down.
- Custom lambda dependency (Custom::LambdaDependency): A custom dependency to trigger the lambda function when the Cloud Formation stack is deleted.
Parallel Computing Toolbox and MATLAB Parallel Server software let you solve computationally and data-intensive programs using MATLAB and Simulink on computer clusters, clouds, and grids. Parallel processing constructs such as parallel-for loops and code blocks, distributed arrays, parallel numerical algorithms, and message-passing functions let you implement task-parallel and data-parallel algorithms at a high level in MATLAB. To learn more, see the documentation: Parallel Computing Toolbox and MATLAB Parallel Server.
MATLAB Job Scheduler is a built-in scheduler that ships with MATLAB Parallel Server. The scheduler coordinates the execution of jobs and distributes the tasks for evaluation to the server’s individual MATLAB sessions called workers. For more details, see How Parallel Computing Toolbox Runs a Job. The MATLAB Job Scheduler and the resources required by it are created using AWS CloudFormation templates.
AWS is a set of cloud services which allow you to build, deploy, and manage applications hosted in Amazon’s global network of data centers. Find out more about the range of cloud-based products offered by AWS. Services deployed in AWS can be created, managed, and deleted using the AWS Management Console. For more information about the AWS Management Console, see AWS Management Console.
No programming or cloud experience required.
If you already have an AWS account set up and ready to use, you can start a MATLAB Parallel Server Reference Architecture cluster in less than 15 minutes. Startup time varies depending on the size of your cluster.
To learn about setting quotas, see AWS Service Quotas.
You can copy the AMI for a certain MATLAB version to a target region of your choice.
- In the Releases folder of this repository, choose the MATLAB release that you want to copy. Download and open the CloudFormation template JSON file for that release.
- Locate the Mappings section in the CloudFormation template. Copy the AMI ID for one of the existing regions, for example, us-east-1.
- To copy the AMI to your target region, see the AWS documentation on Copy an AMI.
- In the Mappings section of the CloudFormation template, add a new RegionMap pair corresponding to your target region. Insert the new AMI ID of the AMI in the target region.
- In your AWS Console, change your region to your target region. In the CloudFormation menu, select Create Stack > With new resources option. Provide the modified CloudFormation template.
You can now deploy the AMI in your target region using the AMI that you copied.
Spot instances allow you to use AWS EC2 Instances at a reduced cost. AWS uses Spot Instances to sell unused instances within their data centers. However, AWS can reclaim these instances at any time. For more details, see the AWS documentation on Spot Instances.
Before enabling Spot Instances, consider these three aspects:
-
Pricing: Spot Instances offer discounts compared to On-Demand EC2 instances. The actual discount depends on the available unused capacity of the EC2 instance within the Availability Zone (AZ). For more details, refer to the AWS documentation.
-
Behavior of your cluster when AWS reclaims a Spot Instance: Spot Instances are used only for the worker nodes, whereas the head node always uses an On-Demand instance. This is to ensure that you do not lose any user job and task information when an EC2 instance is reclaimed by AWS. Jobs in the queue are run when a new worker instance is available. For details on how jobs are run, refer to How Parallel Computing Toolbox Runs a Job. If an EC2 Spot Instance for a worker is interrupted when it is running a task, the task is marked as failed. You can set the maximum number of times to rerun a failed task using the
MaximumRetriesproperty. For more details, see Access task properties and behaviors. -
Parameters for the VPC and subnet in this CloudFormation template: Each Availability Zone in a region has a different capacity available for Spot Instances. To increase the likelihood of obtaining Spot Instances for your cluster, ensure that your VPC has subnets in multiple Availability Zones in the region.
If you require assistance or have a request for additional features or capabilities, contact MathWorks Technical Support.
Copyright 2021-2024 The MathWorks, Inc.