PyTorch on AWS Customers

NEC uses a dedicated AWS environment located within NEC to infer a diverse set of PyTorch AI models trained on the AI supercomputer. Specifically, we utilize AWS SageMaker and Triton Inference Server to perform secure and on-demand AI inference. This allows us to operate and evaluate many PyTorch AI models in a secure environment in a low-cost and flexible environment.

Our first product, nodoca™, is an AI camera that uses a diagnostic AI model to predict influenza from pharyngeal images and other clinical information. The AI camera is connected to Amazon EC2 P3 instances via Wi-Fi and will expand the capabilities of telemedicine. Recently, we obtained approval to market this as a medical device in Japan. nodoca will be launched in December 2022 and is reimbursable through the national health insurance program. Our model is constructed with a large-scale ensembled AI powered by dozens of models. Initially, we faced problems with inference time and reliability on edge devices. Using PyTorch on AWS, we achieved ten times faster inference than with edge devices, even accounting for the network communication time. We are also constructing a secure database of over 500,000 pharyngeal images with Amazon S3 instances to improve our AI model continuously. We thank AWS and the PyTorch community for their unwavering support in constructing a high-performance, reliable, and secure PyTorch model server and database for medical use.

AIMINA is an AI Platform service that enables non-specialist users to easily learn, build and test various ML models. By running our PyTorch based BERT (Transformer) models supported on AIMINA, used for building chatbots, on AWS Inferentia based Amazon EC2 Inf1 instances, we immediately saw a 2x improvement in throughput and 3x reduction in cost compared to Amazon EC2 G4dn instances. Compiling models for Inferentia using the AWS Neuron SDK was quick and easy, and we look forward to migrating our other models running on GPU-based instances to Inf1 instances in the near future.

By training our large-scale PyTorch-based ML models on Amazon EC2 P4d UltraClusters, leveraging Amazon EKS, AWS EFA, and Amazon FSx for Lustre, we were able to take advantage of cloud scale for distributed training of our diffusion and transformer models. The ability to scale our training using cloud services accelerated our time to train these large models from months to weeks to release them to the open source community. AWS has been a trusted partner and worked with us in lockstep to unblock technical issues and provided a seamless deployment experience. We plan to continue to train our state-of-the-art ML models on AWS.

We execute design-make-test cycles on thousands of compounds per week, using PyTorch-based deep learning models running on AWS to generate molecular structures, perform property inferences, and fine-tune models on resulting bioassay data. We built a cloud scale inference pipeline using Amazon EKS, Karpenter, and KEDA to elastically scale our Amazon EC2 GPU powered compute instances based on custom metrics. This enables us to scale both model inference and fine-tuning, so that we can rapidly explore chemical space, discover differentiated drugs, and deliver them to clinic with unprecedented speed.

By continuing to increase our efficiency using AWS, we can unlock the possibilities of deep learning and artificial intelligence to benefit our customers.

We used Amazon EC2 G5 instances to run inference using a large PyTorch model, for protein sequence prediction. AWS offered agility and a strong partnership that helped us implement an architecture that quickly scaled up our workflow. Using this architecture, we were able to boost performance and operate on orders of magnitude more protein sequences.

Before engaging AWS, we couldn’t run a quarter of the workloads that we wanted in parallel. Using AWS Batch, we could solve a lot of the issues that we were facing. Running parallel tasks would be much simpler. On AWS Batch, we scaled a cluster to support 50,000 workloads in less than 1 hour.

By migrating to Amazon EC2 P4d instances, we improved our real-time inference speeds by 60%.

Amazon EC2 P4d instances offer 400-Gbps high-performance networking on EFA. The GPU-to-GPU networking speed directly impacts the ability to scale efficiently and remain cost effective when scaling to hundreds of GPUs.

For our text ad processing, we deploy PyTorch-based BERT models globally on AWS Inferentia based Inf1 instances. By moving to Inferentia from GPUs, we were able to lower our cost by 69% with comparable performance. Compiling and testing our models for AWS Inferentia took less than 3 weeks. Using Amazon SageMaker to deploy our models to Inf1 instances ensured our deployment was scalable and easy to manage. When I first analyzed the compiled models, the performance with AWS Inferentia was so impressive that I actually had to re-run the benchmarks to make sure they were correct! Going forward we plan to migrate our image ad processing models to Inferentia. We have already benchmarked 30% lower latency and 71% cost savings over comparable GPU-based instances for these models.

We continuously optimize and improve our computer vision models, which are critical to TRI-AD’s mission of achieving safe mobility for all with autonomous driving. Our models are trained with PyTorch on AWS, but until now PyTorch lacked a model serving framework. As a result, we spent significant engineering effort in creating and maintaining software for deploying PyTorch models to our fleet of vehicles and cloud servers. With TorchServe, we now have a performant and lightweight model server that is officially supported and maintained by AWS and the PyTorch community.