How do you ensure the security of a third of America’s web traffic? By comprehensively understanding the different attack vectors related to your cloud services, specifically AWS (Amazon web services). And being prepared to react fast.
If you're like most IT pros, you probably have many end users access and cloud services and virtual machines on a daily basis. Depending on the amount of activity, it may be hard for you to determine which IPs access in your business’s cloud are legit or elicit.
That’s why we got together with Will Bengtson, a security researcher with over a decade of experience in cybersecurity who works at Netflix.
So since your cloud provider isn’t going to hold your hand on this one, you need to know about these three tools Will uses.
Finding Good Tools
Working at Netflix, Will is used to handling a complicated cloud infrastructure.
Netflix as a streaming service at peak hours has a third of the Internet traffic in the US. “Which is insane,” Will said.
“We scale up or scale down depending on the time and where we think the peak traffic is going to be.”
They’ve got so many IPs that change so often that they can’t even describe them all before they’re different again.
That means that if they’re looking for a particular signal in monitoring, they can’t necessarily take a straightforward approach. The timing or the sheer amount of data causes them to think about their approach to security a little differently.
So in monitoring hundreds of thousands of different IPs for suspicious activity, Will makes use of a couple of really good tools.
Managing the threat landscape in the cloud? Not easy.
But some things can help.
IAM is AWS’s identity management, and Will calls it their bread and butter. “Within AWS we own permissions and kind of hand those out to teams. So we use the idea of least privilege to our advantage,” he explained.
CloudTrail is another helpful tool, risk auditing of AWS.
“So if you're describing what servers you have in your cloud environment, you'll see that actually end up in CloudTrail. ‘Hey, Will described our instances at this given time in this account,’” he explained.
Kind of like fingerprinting--keeping track so you can go back and check if you need to.
In addition to monitoring, they use CloudTrail for pulling back permissions, too.
“I’ve spent a lot of time in CloudTrail over the last few quarters diving into what each event means, what's logged or not, and then how we can determine whether an event in CloudTrail is suspicious,” he said.
Repokid is a tool that looks at what you're actually using from an API perspective and what you've been given.
“The combination of Repokid using that CloudTrail data to actually pull back permissions gives us a better stance. It allows us to be dynamic by deploying servers with a little increased permission than they might need and then pull it back to see what they're actually using,” Will said.
It’s a huge amount of work, but it’s actually pretty targeted.
He’s essentially building a database of whitelisted IPs and checking to see based on past events what the patterns are of how people are accessing it.
“Due to our size and how long it takes to describe our environment, we can't just ask the question to AWS, ‘Hey, what do we have at this given time?’” he said. “So we're having to try to build a table as we spin up services and spin them down.”
Think about it as calling dibs.
“The first time you have a credential and you use it, you're calling dibs on that IP address. And so we're making the assumption that first call wins, and anything that deviates from that ‘first called’ IP is potentially suspicious,” Will said.
Server Side Request Forgery (SSRF)
Just because something is flagged doesn’t necessarily mean it's suspicious, but it will require investigation. Is it something that isn’t right, or is it something you know about?
And here’s a relative new place to detect a compromise. It isn’t just modern IPs and API calls anymore.
Will shared about the server side request forgery (SSRF) attack method.
With SSRF, you’re basically finding a vulnerability in an application that allows you to have the application request a URL on your behalf.
“We've seen this before with the applications that are allowing you to configure where you should pull your icon from or things like that,” Will said.
Researchers will find a vulnerability and have the application reach out to the metadata service, which is attached to every instance in the cloud where the credentials are actually stored.
“They're tricking the app to reach out to the metadata service, and then they're pulling those credentials back to their local machines and using the credentials there.”
Will needed to be able to detect when those credentials were being used outside of their environment.
Here’s what it looks like. If SSRF is trying to exploit the credentials from an app, you’d see the IP change because the attackers are using the credentials from their own system or you’d see requests originating from the server itself as whatever URL library you're using.
“By looking at some user agents on top of IPs, you could detect attack like this,” Will said.
You should also look at credentials that are exposed in the metadata. You can basically find out all the metadata that AWS tracks about that server that you're on.
But the credentials are short-lived, rotated every one to six hours. If you pull a credential from the metadata service as an attacker, you have a limited window of when that's useful.
Other Attack Vectors
SSRF is probably the most prevalent in the last six months, to Will. “Anytime an attacker can get to your credentials and if they know you're running in the cloud, that's seems to be the avenue they continue to pursue,” he said.
But another thing to think about, connected with the rotating metadata, is the time to weaponization is getting smaller and smaller.
He predicted we’ll continue to see this process evolve.
Supply chain attacks are going to be the next big thing to target a company, figure out what kinds of technologies they're using, and attack them through the back door.
“We're definitely seeing an uptick in the supply chain attacks,” Will said.
If a company’s network stance and application stances are pretty hardened, attackers will try to get through another way. And another way. And another.
Until next time, stay safe out there.