Moderna, mRNA and AWS

Transcript

Carlos Peralta: We’re trying to save lives here, right? We’re not doing online shopping, so we need the data to be completely secure and scalable.

Rahul Subramaniam: Just a few decades ago, something like this would have been impossible to even conceive of.

Carlos Peralta: We learned from day one that we have to think big and think what’s going to happen in the next 10 years.

Rahul Subramaniam: Would some of this even be possible if someone tried it on premise?

Carlos Peralta: It is just too demanding. It is just obsolete.

Hilary Doyle: This is AWS Insiders, an original podcast from CloudFix, bringing you what you need to know about AWS through the people and the companies that know it best. CloudFix is the nonstop automated way to find and fix AWS-recommended savings opportunities. It never stops. I’m Hilary Doyle. I’m the co-founder of Wealthie Works Daily.

Rahul Subramaniam: And I’m Rahul Subramaniam. I’m the founder at CloudFix.

Hilary Doyle: Rahul, how do you feel about needles?

Rahul Subramaniam: Well, not my favorite, but-

Hilary Doyle: Fair.

Rahul Subramaniam: … if needed, I’ll happily roll up my sleeves.

Hilary Doyle: That’s great because when you do, it turns out you are also benefiting from the magic of the cloud.

Rahul Subramaniam: Of course, I mean, everything is connected to AWS these days, isn’t it?

Hilary Doyle: Isn’t it just? Developing new vaccines and therapeutics takes years … experimenting, running trials, designing and building custom facilities that are needed for individual drugs. Then add to this, the data.

Rahul Subramaniam: That is a huge challenge, when data that is siloed doesn’t lead to collaboration or development of those key insights. Also, the on-premise data centers for supporting scientific workloads and high-performance computing are just notoriously inflexible, even if they do work at times.

Hilary Doyle: At times, yeah. So it is no surprise that in the history of vaccine development prior to 2020, the fastest vaccine development ever had taken four years. Now by March of 2020, in the midst of a global pandemic, four years sounded like an eternity, but some companies were perfectly positioned to respond to the need because they were already in the cloud. So within months, not years, we heard this news.

News clip: More hopeful news on the COVID-19 vaccine front, Moderna says, “New study results suggest its shot is more than 94% effective in preventing the virus,” even more promising-

Hilary Doyle: The cloud made Moderna possible. And today, we are speaking with Carlos Peralta, director of data engineering and cloud architecture at Moderna. We’ll speak with him about the essential role that AWS played and continues to play in Moderna’s innovative work. But first, because we like tradition, this week’s AWS news headlines …

I’m pretty sure we need a little mystery music to spin up this first news story. It’s very Hercule Poirot. Maybe it’s a little more Inspector Gadget? Customers can now search the Amazon Inspector vulnerability intelligence database for details about scanned vulnerabilities. Rahul, solve this news item for us.

Rahul Subramaniam: Hilary, it won’t tell you where the bodies are buried, but when you turn it on, it will scan all your resources for security vulnerabilities at best practice violations, and it allows you to plug these gaps quickly. So AWS does all the heavy lifting for you. Sometimes though, when it finds a vulnerability, it is so specific and niche that you have to take the time to research how and when that vulnerability is exploited and how that might impact you. This new announcement just makes that task really simple by allowing customers to search the Amazon Inspector database for all of these details.

Hilary Doyle: Go Go Gadget news item number two, some AWS AI news now, a family of new instances on SageMaker is now available to help achieve high performance at low cost for generative AI models, including LLMs and vision transformers. Rahul, how practically significant are these upgrades?

Rahul Subramaniam: It’s all about AI now, isn’t it?

Hilary Doyle: Yes.

Rahul Subramaniam: AWS just announced two new instance types, the ML Inf2 and the ML Trn1, specifically to handle generative AI workloads. These are instances that have 384 and 512 GB of shared accelerated memory, respectively, just for inferencing. When I look back at some of the conversations we’ve had on this show about on-prem versus cloud, I mean, I feel like the excuses for an on-prem setup, especially when you’re innovating and growing, are just disappearing really, really fast.

Hilary Doyle: No more excuses. Now onto the football turf … American football, Rahul, sorry. A few days ago, the NFL released its season schedule. Each season, we know that scientifically there are about a quadrillion possible schedule combinations considering primetime TV slots, travel, stadium availability, and obviously rivalries. We are looking at over 26,000 specific factors that need to be considered. AWS has a hand in helping the NFL do this. Rahul, lay out the play.

Rahul Subramaniam: That’s absolutely right. The NFL uses AWS to run high-performance computing workloads, and that means they can figure out the best possible schedule for the year. Simply put, here’s how they do it. The NFL uses thousands of instances on an hourly basis to run different schedule scenarios. Thanks to AWS, the NFL can put together and analyze over 100,000 possible schedules, something that would’ve taken massive amounts of people power to do manually, and then narrow it all down to the one perfect schedule.

Hilary Doyle: All we really want from AWS is to help on our fantasy football draft. Anyway, next season, guys, next season … That’s it for your AWS news headlines.

Rahul Subramaniam: Moderna is a platform-based pharmaceutical and tech company. It was born in the cloud. It leveraged the platforms for both AWS and mRNA to grow, scale, compete, and eventually outperform incumbents, and it has done this all in a highly regulated environment.

Hilary Doyle: Before we talk to Carlos, let me set the scene. It’s January of 2020 and employees at Moderna start hearing rumors of a virus in Asia that may be spreading to parts of Europe. Now, nobody really knows or understands the extent of the outbreak, but when a worldwide pandemic is declared by the WHO in March, Moderna pivots fast. They immediately decide that everyone will work remotely, and they scale up that work. Back in 2010, the company was actually founded in the cloud, and so working remotely wasn’t a challenge. The challenge they did have, the most pressing challenge in the world at the time, was developing a vaccine for this virus.

So Moderna receives the genetic sequence of the virus. It doesn’t physically arrive, it is emailed. And 45 days later, they’ve developed an mRNA vaccine for COVID-19. Now, to be clear, there had been years of development that led to this breakthrough. Not only had Moderna been working for a decade on mRNA vaccines, but by going all-in on the cloud, they could redirect instantly to work on a new challenge and then to ramp up the technology needed to address it.

Rahul Subramaniam: This was a massive amount of computing to do in such a short period of time, I mean, something that would have been virtually impossible just a short while ago. So let’s find out more about how Moderna leverages AWS to drive their innovation and development. Carlos Peralta is the director of data engineering at Moderna Therapeutics. Thank you so much for being on the show.

Carlos Peralta: No, thank you, Hilary, Rahul, for having me. I’m super excited to talk about this.

Rahul Subramaniam: Can you paint a picture for us about what the scale of the data and the compute looks like? What was the setup that you had, that you leveraged at AWS, that made all of this possible on your platform?

Carlos Peralta: So we developed, in-house, more than 70 apps, custom developed applications, to support different verticals of the company. So we had a bunch of EC2 instances for compute power. Then we migrated to Elastic Container Services, which are a way for us to give high availability. At the time, my manager was very focused on high availability, resilience on the applications that we were building.

So we had, again, more than 70 apps developed in-house, code in the cloud, these being deployed in a high-availability architecture databases for each one of those applications to make sure that we collect the data that we need. And on top of that, all the data that were coming from the clinical trials, which were not ingested by those apps, it was external data providers, right, all the entities that were supporting us on this process?

So we got all that data, and we use S3 because, of course, it’s like simple storage. And it give us the ability to scale that data in S3 with no size constraints or speed constraints. And then the data that was needed to be ingested, we use ETL processes, which is extract, transform, and load, and we put it into our data warehouses.

So centralizing the data from different angles, from the internal applications, from the internal datasets that we collected from designing vaccines, and then the data that we get from the clinical trials and put these on a centralized place where give us the ability to build all these contextualized datasets. So you can paint a picture of a single data point and its life cycle across the board.

Rahul Subramaniam: Got it.

Carlos Peralta: So scalability was one of the biggest things that we got from Amazon because, again, you can scale from two servers to 100 servers with just a couple of clicks. So we were always sure that the things that we were building were able to compute and analyze such large datasets.

Rahul Subramaniam: So could you just do me one more favor and take this one step further?

Carlos Peralta: Okay.

Rahul Subramaniam: If you contrasted this with something that, let’s say, somebody in your competition was trying to do on-prem, what would that even look like? Would some of this even be possible if someone tried it on premise?

Carlos Peralta: Yeah. It must be a very, very, very tough challenge, because when you say you go to Amazon and you want to spin up an EC2 instance, that is a P from the P3 family, which is super powerful compute, and you need that on prem, you’re going to have to go and buy hardware and then bring it to your office and put it in your server room and then plug it to your network. It is just too demanding. It is just obsolete.

Rahul Subramaniam: I think it would take 45 days just to do procurement on those GPUs, which are so hard to come by.

Carlos Peralta: Exactly. Exactly. And even worse, at the time where supply chain was at its worst, how do you get something? How do you get something like physical? Impossible. So I can’t imagine how somebody that lives in still on on-prem systems can do something like this.

Hilary Doyle: I’m just going to break into this interview for a second. Bear with me, because while we are talking about ways in which Moderna collaborated with AWS, I would be remiss not to mention Rahul’s side hustle. Now, you know I don’t like competition, but in this case, in this one case, I make an exception, because every week Rahul hosts a livestream with his fellow AWS nerd, the beloved Stephen Barr. The two of them discuss how to use all of the services from AWS to solve hard problems in the easiest way possible.

Rahul Subramaniam: And not only that, we also discuss how to do that most efficiently. We break down the latest news from AWS and offer our learnings and insights about them along with some amazing guests from AWS who are both with us in the virtual studio, as well as in the audience.

Hilary Doyle: That’s right. The livestream is called AWS Made Easy, and you can find out more about it at cloudfix.com/livestream. They stream live, expectedly, at LinkedIn, YouTube, Twitch, they’re very cool, Facebook. They’re keeping Twitter afloat. You can ask your questions live on the show, and this team and their amazing guests will do their very best to answer them all for you.

Rahul Subramaniam: Okay. Now back to our interview with Carlos, and in fact, back to when Moderna started.

Hilary Doyle: Moderna is obviously known as a pharmaceutical company that was born in the cloud, but when Moderna was born, 2010, AWS had only been around for four years. So how and why was this decision made?

Carlos Peralta: That’s a good question. I was not at the company at that time, to be fairly honest.

Hilary Doyle: I know, five years before. You were too young, too young.

Carlos Peralta: I would like to think that, yes. But I think that the vision from day one from our leadership was to make sure that we have a platform that can grow and scale exponentially without having to pivot too much into rebuilding and stuff. Of course, I’m oversimplifying, but that’s the whole purpose of being in the cloud, that you can scale easily, use the capabilities of the cloud and never be bound to any on-prem systems, when possible.

Rahul Subramaniam: Got it. So what has this birthed in AWS for Moderna? What has that changed relative to other pharma companies at that time or even today?

Carlos Peralta: Yeah, that’s perfect. That’s what I was going to say. It’s not even at the time. It’s even today, right?

Hilary Doyle: Yeah.

Carlos Peralta: Because pharma, being a highly regulated space, has always been on-prem systems, physical servers, physical equipment, so you can fulfill the requirements from a heavily regulated environment. So being in the cloud gave us the ability not only to build stuff in the larger scale, but also when priorities change, we were able to just pivot fearlessly and get into that path without affecting what we had before, but reusing all the technology and learnings from the past. So the cloud has enabled us to be faster, more efficient, and aligned with that vision from our leaders to digitize everywhere and be the first biotech that is a digital biotech company.

Rahul Subramaniam: That is really interesting. Moderna and general healthcare is, like you said, hugely regulated and requires crazy amounts of compliance. When you guys started with AWS in the cloud, a lot of this compliance itself did not exist across the vast majority of services. So I assume that you guys worked very closely with AWS to get all of these compliances in place, but that’s some really visionary thinking that I really love, where you made the big bet up front and took the big leap of faith that AWS would work with you to get all this compliance done. What was the thinking? Because that leap of faith doesn’t seem to be a common way of thinking across the industry.

Carlos Peralta: You are absolutely right, Rahul. We had a super strong partnership with Amazon and everything that we have been able to build and scale in large volumes have been in constant partnership and growing for both companies. We have been able to be part of the design of beta versions of their services that were not only designed for us, but also where we had great input.

So in that partnership, we were able to dictate what we needed, how we needed it, and when we needed it, especially because at Moderna, we struggled from day one because we were not able to find the solutions that we wanted off the shelf. We knew that when we get to the point of clinical trials and those highly regulated last stages of a vaccine, we knew that we had to face those challenges. But at the time when the decision was made, we were not there yet. Right? We were rushing into building stuff that were fully intended to help patients. And we knew that when we get to the point of actually having to file the regulations for the FDA and any other government entities that are required, I think that we were going to be able to accommodate all of that by the time that we get to that point. It’s just that, for us, everything happened super fast and we changed and grew and became such a big and important company in a very unexpected situation.

Rahul Subramaniam: I was just going to extend the conversation from compliance and regulation to security, because that’s one of the other concerns that a lot of highly regulated organizations have about being in the cloud, and a vast majority of these CIOs that I speak to have this notion that they feel much more secure in their own data centers on premise as, again, something that’s so-called public cloud. So how did you guys tackle that security concern, and how do you go about securing all of your data and all of your infrastructure?

Carlos Peralta: That’s a good question. Yeah, security has been on our roadmap since day one. We really know in advance with the type of data that we’re dealing with, right?

Rahul Subramaniam: Right.

Carlos Peralta: We’re not dealing with … I mean, I’m not saying that not every data point is important, but we’re dealing with data from patients. We’re dealing data from humans. We’re trying to save lives here, right?

Rahul Subramaniam: Right.

Carlos Peralta: We’re not doing online shopping. So we need the data to be completely secure and scalable. So yes, security has been on a top priority since day one. So we manage security in every layer of our architecture from the people who get involved in the conversations from day one, down to who has access to the database. Everything is controlled. Everything follows the standard procedures. Everything is audited and everything is locked. So we have the ability to go back to any point of time and know, what did you run every day at 10:00 AM in the morning? Or why did you run this? What was the dataset you extracted from? How many times did you use any of our visualizations, or how many data points were ingested on any given point of time?

Rahul Subramaniam: That sounds like literally a page out of the Well-Architected Framework. That is sage advice for the audience. Since all of you started in 2010 with AWS, that was the early days of the cloud. Right? A lot of the things that we take for granted … I think in 2010, even the AWS Console did not really exist.

Carlos Peralta: It’s true.

Rahul Subramaniam: And there was no IM. IM came in 2012. So without all of those pieces in place, I’m sure that there were tons of lessons you guys learned as you evolved and AWS evolved and you were trying to find solutions to some of these really, really hard problems. What are the lessons that you learned that others can learn from and make an easier decision to move to the cloud?

Carlos Peralta: Excellent. Yes, we keep learning lessons today. I mean, if you stop learning lessons, maybe you’re doing something wrong. So we’re still learning every day, and that’s super cool, honestly, because that motivate us to be better every day.

So I would say that one of the biggest things we have learned is that everything change and everything can change in the blink of an eye. So how are you going to be ready when a change is coming your way? So plan for that. Try to make sure that whatever you’re building, it’s good enough that you can switch and flip and then take an alternative path and still make it work.

We also learned that most of the things become obsolete in technology pretty fast. And working with a such a big partner like Amazon Web Services, we know that every service is evolving also with us. I mean, if we look at Redshift, Redshift, I don’t know, six years ago was like, yeah, the data warehouse, but it was-

Rahul Subramaniam: It was basically a Postgres database.

Carlos Peralta: Exactly. It was like, yeah, it’s okay. And that’s the offering. But the biggest strides and improving that Amazon made on Redshift in the last two, three years have been amazing. Right?

Rahul Subramaniam: I agree.

Carlos Peralta: We were thinking in times, back in the day … Again, everything changed. And I want to reiterate on that, because technology becomes very obsolete. We were doing ETL processes every four hours, and we were super excited, like, “Oh, awesome. We’re getting data refresh every four hours.” And today, the actual buzzword is no ETL at all, right?

Rahul Subramaniam: Yep.

Carlos Peralta: So how is that you’re going to adopt that new pattern or that new trend, but your systems are not going to be affected? Your end users are not going to be affected? And with Moderna, in particular, we want to always think like, “What happens if, in six months, we are 10 times bigger than we are today? What happens in 12 months, we have 10 vaccines in the market? How are we going to deal with it?” So we are trying to always take a step back from every project, analyze the expected outcome, and make sure that we have a plan just to make sure that we are going to be able to accommodate the growth.

Rahul Subramaniam: Carlos, a lot of the best learning comes from living through some failures.

Hilary Doyle: Speak for yourself.

Rahul Subramaniam: I’ve lived through failures, so I mean, that’s how I’ve come up. And I share them very generously. But Carlos, I’d love to get some stories about failures that others can learn from and hopefully make better, more resilient systems. So is there a specific technology failure that you could share that has changed the way you think about any specific AWS service or the way it’s put together or the way you scale it or plan for it?

Carlos Peralta: I don’t want to give names, just to be fully respectful, but yeah, we chose, in the past, ETL, for example, ETL tools that promised us to do real-time data and configuration to be super simple, so especially because they were born in the cloud, as well, is that what they said. So we chose those tools because they seemed to be okay, but unfortunately six months later, we realized they were really bad. We couldn’t accomplish what we were planned for.

So I always go back to the same mindset, but pivot fearlessly and behave like owners. So we accepted the failure. We chose a different tool. Find a better one, and not better because they’re better than the others, it’s just better because it’s what we need as a business.

Rahul Subramaniam: Got it.

Carlos Peralta: … and disregard what we did in the past, learn how to better assess new technologies and new vendors in that particular line of business of ETLing data, and then move forward with the new tools.

Rahul Subramaniam: Quick question, was this an AWS tool or a non-AWS tool?

Carlos Peralta: It was a non-AWS tool.

Rahul Subramaniam: Figures.

Hilary Doyle: And Carlos, talk about now. Where is Moderna’s focus today?

Carlos Peralta: We have 16 different vaccines in the pipeline. So if we talk about data, data’s going to keep coming. We still are running tons of clinical trials for all our vaccines. That doesn’t stop. That doesn’t scale down. That doesn’t go slow at any point of time. If anything, it’s just increasing.

Hilary Doyle: And so you haven’t seen a decline in terms of the size of the infrastructure that you are maintaining with AWS?

Carlos Peralta: No, no. Actually everything goes … Again, as I said before, it’s just increasing because we are becoming more and more proficient with our data. We’re getting more data for more clinical trials. Our data scientists are really building new AI/ML models, are going to bring even more impact to all the other vaccines that we have in our pipeline. So data is just increasing.

Hilary Doyle: Is there anything you’re still waiting for from AWS?

Carlos Peralta: I want to say that I am super on board with one of the latest announcement from Amazon last year in re:Invent with the no ETL. That’s their new promise. No, ETL. So I’m really, really looking forward to it.

Rahul Subramaniam: Sounds like magic for your world.

Carlos Peralta: Exactly. Exactly. That sounds like magic. And again, we’ve seen the big improvements they have done. And for my particular line of business, that will be a huge impact. That will bring a huge impact in saving time, people resources, and in getting us closer to what we are really going to. And it’s like real-time data everywhere.

Hilary Doyle: Well, Carlos, thank you for your work and for joining us on this podcast. It’s been great to have you here.

Carlos Peralta: No, it’s been a pleasure.

Rahul Subramaniam: It’s been an absolute pleasure.

Carlos Peralta: Thank you very much. I’m super excited.

Hilary Doyle: It was really interesting to speak with Carlos. I mean, yes, Moderna is more broad than the COVID vaccine alone, but that vaccine story was stunning to watch in real time. I mean, they were a company that had broken from convention, embraced a completely different paradigm in a rigid and extremely highly regulated industry. And then they found themselves in the middle of what was the ultimate proof moment. For AWS, it’s an incredible example of the cloud shaving years off of a tried-and-true process and literally helping to save millions of lives.

Rahul Subramaniam: I absolutely loved this conversation. I’m a sucker for stories where technology has such a life-changing impact.

Hilary Doyle: That’s lovely.

Rahul Subramaniam: I mean, there were three things I think that stayed with me till the very end. The first was that AWS and the cloud was the key enabler for Moderna moving to a remote setup when COVID hit. I mean, that resonated with me a lot because it was AWS that allowed us to become a completely remote and global organization way back in 2008.

Hilary Doyle: You’re so vanguard.

Rahul Subramaniam: The second one is that the kind of innovation that was needed in those dark COVID days could only have happened in the cloud and AWS specifically. I’m not sure companies could have even come up with these novel approaches had it not been for the cloud, and because this kind of processing is simply not possible in any on-premise setup. And lastly, the icing on the cake was the fact that Moderna believes that AWS and the cloud was literally the only secure and efficient way to go about running a modern, cutting edge and innovative business.

Hilary Doyle: Heard everything you said, but now can only think about cake. As always, we want to hear from you. Please send us your feedback. We thrive on criticism. Speaking of which, we speak with someone who is intensely critical of the cloud next time, so stay tuned. In the meantime, you can reach us at podcast@cloudfix.com.

Rahul Subramaniam: And if you’re loving the show-

Hilary Doyle: Only if you’re loving the show …

Rahul Subramaniam: … leave us a review. And don’t forget to follow the show to get new episodes as soon as they’re released.

Hilary Doyle: AWS Insiders is brought to you by CloudFix, an AWS cost optimization tool. You can learn more about them at cloudfix.com.

Rahul Subramaniam: Thanks for listening.

Hilary Doyle: We’ll see you next time.

Rahul Subramaniam: Bye-bye.