![]() ![]() If you use a KMS key ID or an alias of your master key, the Amazon SageMaker execution role must include permissions to call kms:Encrypt. Amazon Resource Name (ARN) of a KMS Key Alias "arn:aws:kms:us-west-2:111122223333:alias/ExampleAlias" The KmsKeyId can be any of the following formats: The AWS Key Management Service (AWS KMS) key that Amazon SageMaker uses to encrypt the model artifacts at rest using Amazon S3 server-side encryption. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File ), this copies 1/ n of the number of objects. In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key. Keep this in mind when developing algorithms. This applies in both File and Pipe modes. If you do, some nodes won’t get any data and you will pay for nodes that aren’t getting any training data. In this case, model training on each machine uses only the subset of training data.ĭon’t choose more ML compute instances for training than available S3 objects. If there are n ML compute instances launched for a training job, each instance gets approximately 1/ n of the number of S3 objects. If you want Amazon SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key. If you want Amazon SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated. The object that each S3Uri points to must be readable by the IAM role that Amazon SageMaker uses to perform tasks on your behalf. s3://customer_bucket/some/prefix/relative/path/custdata-N The complete set of S3Uri in this manifest is the input data for the channel for this data source. The following code example shows a valid manifest format: This JSON is equivalent to the following S3Uri list: s3://customer_bucket/some/prefix/relative/path/to/custdata-1 s3://customer_bucket/some/prefix/relative/path/custdata-2. Note that the prefix must be a valid non-empty S3Uri that precludes users from specifying a manifest whose individual S3Uri is sourced from different S3 buckets. SageMaker appends the suffix elements to the prefix to get a full set of S3Uri. The first element is a prefix which is followed by one or more suffixes. For example:Ī key name prefix might look like this: s3://bucketname/exampleprefixĪ manifest might look like this: s3://bucketname/example.manifest A manifest is an S3 object which is a JSON file consisting of an array of elements. If the object sizes are skewed, training won’t be optimal as the data distribution is also skewed where one host in a training cluster is overloaded, thus becoming bottleneck in training.ĭepending on the value specified for the S3DataType, identifies either a key name prefix or a manifest. Amazon SageMaker does not split the files any further for model training. The algorithm container use ML storage volume to also store intermediate information, if any.įor distributed algorithms using File mode, training data is distributed uniformly, and your training duration is predictable if the input data objects size is approximately same. In addition to the training data, the ML storage volume also stores the output model. In File mode, make sure you provision ML storage volume with sufficient capacity to accommodate the data download from S3. ![]() If an algorithm supports the Pipe input mode, Amazon SageMaker streams data directly from S3 to the container. If an algorithm supports the File input mode, Amazon SageMaker downloads the training data from S3 to the provisioned ML storage Volume, and mounts the directory to docker volume for training container. For the input modes that Amazon SageMaker algorithms support, see Algorithms. The input mode that the algorithm supports. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |