API and microservice security

What is an API?

An application programming interface (API) allows different pieces of software to interact with one another. So if Application A has a piece of information that Application B needs in order to do something, then an API allows Application B to request that information.

Because computer programs can't really think for themselves, an API needs a strict set of instructions. This is where an API definition comes in. An API definition will tell Application B exactly what it can request from Application A - and exactly how to do it.

In an ideal world, every API definition would be publicly available - allowing anyone to interact with an interface within its preset boundaries. But in practice, this is true only in a minority of cases. A definition will exist - somewhere - but may be purely internal (e.g. on a Jira ticket).

What is API security?

The places where systems interact have always been attractive to hackers. These are often places where unexpected behavior can be triggered - so with any form of API, security should be absolutely paramount.

Because APIs are primarily used/consumed by developers and machines, there is often a tendency for their security to be overlooked. This can lead to gaps in API specifications, enabling anyone so inclined to (for example) edit client-submitted parameters in order to access restricted data.

Steps toward good API security

API security is a huge topic. However, there are best practices that organizations should follow when implementing APIs in their systems.

Firstly, it's important to bear in mind that good security is not something that can simply be applied after API design work is complete. In order to avoid the many pitfalls inherent when implementing APIs, security needs to be a consideration from the very outset. This is considered best practice across web application design more generally, and is also core to the DevSecOps philosophy.

Secondly - as with web applications more generally - APIs should be rigorously tested, on a regular basis. This can reveal vulnerabilities that were either missed at the design stage, or which were subsequently introduced by updates to the software. This might involve the use of a web vulnerability scanner, manual penetration testing, or both.

How to test API security

Testing APIs for security flaws has been an important part of web application testing for many years. Bugs that can be found in APIs are not new - they simply represent a subset of all web security issues. Many API endpoints are still vulnerable to SQL injection, directory traversal, or XXE attacks, for instance - so you cannot test them without considering these "classic" bugs. The OWASP API Top Ten list of vulnerabilities is very close to the conventional OWASP Top Ten.

Traditional security scanners which are able to crawl websites and interact with UI components already provide some API testing coverage. At the same time, they are limited to testing API endpoints that are referenced from the client-side HTML or JavaScript code, while servers may silently expose additional endpoints.

To ensure testing coverage, it's important for scanners to understand API definition formats (such as OpenAPI) and to be capable of constructing network requests based on them. Likewise, API specifications should provide as much data as possible to ensure a smooth testing process. Providing specific examples of network requests and responses for your API can significantly help scanners to construct such requests and inject testing values.

As with testing any other part of a web application, while scanners might excel at finding certain vulnerabilities, other bugs will require the lateral thought and adversarial mindset of a manual penetration tester. One key to successful security testing is to strike a balance between the two.

The rise of microservices

Because of the above, many organizations practicing Agile development find that microservice architectures suit them very well. In fact, it's not unheard of for organizations to ensure that every application they produce has an API available, in order to encourage a healthy Agile culture.

And the benefits don't end with easier continuous delivery. Because APIs can enable standardized communication between disparate pieces of software, they also mean there is no requirement for that software to share a common language - or even to be deployed on the same platform. This further increases operational and developmental flexibility.

Increased use of APIs and microservices also means that organizations have more choice when it comes to implementing external services. Best of breed technologies can easily be cherry picked and added to an application. This enables better user experience (UX) in areas like online payments, user authentication, or even tailored advertising.

Of course, monolithic architectures aren't inherently bad. In The DevOps Handbook (while generally promoting the idea of microservices), Kim et al. note that a monolithic approach may in fact be best for an organization early in a product life cycle. The key is that architectural style should always evolve to suit functional needs.

Securing microservices

Microservice architectures, by definition, increase the number of interactions that take place between systems. And, as we have seen, these boundaries are exactly the sort of place where security vulnerabilities can often be found. Without careful oversight, this can take the potential for unexpected dangerous behavior to new levels.

API specifications such as OpenAPI are an attempt to standardize the way APIs are documented, in a machine-readable format. But, while this standardization assists developers in designing more effective applications for the microservice ecosystem, API security problems can still exist.

With APIs, implementation can be tricky - especially for lower-maturity organizations. It's easy to have an API that looks correct, and works as expected - but which will still process dangerous requests. Given the sensitive nature of the data increasingly served through APIs, this can have disastrous consequences, if noticed by a cybercriminal.

A vulnerability scanner that can parse API definitions often gains a real advantage when discovering API functionality to test. In such cases, definitions serve as both identifiers and as a guide to the scanner, which can then perform testing automatically. As with any aspect of web security, this should ideally then be followed up by manual penetration testing.

The challenges of securing microservices

Vulnerability scanning APIs isn't necessarily simple. One problem lies in the proliferation of API development that microservice architecture has encouraged. As above, Agile development tends to prefer small teams - and this mirrors nicely with smaller services.

But segregation of development can mean diminished cross-channel communications. And in less mature organizations, smaller teams are less likely to include security specialists. This can lead to attempts to apply security as an afterthought - which is seldom realistic. Far better to consider security throughout the build process from day one - as with DevSecOps.

In some cases, a complex microservice-driven approach can lead to organizations struggling to manage their APIs. For instance, many organizations are unable to list all the APIs they have on their systems. If a vulnerability scanner can't discover these hidden endpoints (as many can't), then this is bad news from a security assessment perspective.

And despite industry efforts toward standardization of API specifications, this is still far from the norm. Even within a single organization, similar-looking endpoints can have vastly different specifications from one another. This again makes life difficult from a testing perspective. And it of course assumes that a formal API definition is available in the first place - which it often isn't.

How is securing an API different?

APIs have been around for a long time - to the point that they can be considered an integral component of most web applications. So why is scanning an API different to scanning any other part of a web application?

Firstly, with an API call, data will often be presented slightly differently than with conventional traffic. Here JSON and XML will often be used to serialize the data being transferred. It's also worth noting that manually testing an API can be a tedious task. This type of work can often be automated by a vulnerability scanner - which is an easier approach. This also has the benefit of giving testers time to explore other avenues, making better use of their skills.

Then, there's API discovery, as mentioned above. Many vulnerability scanners are unable to parse API definitions - meaning that they are likely to overlook any hidden standalone APIs. These include APIs that are present in an application, but not normally used by the browser. An API allowing a desktop email client to access webmail data is one example. This often leads to scanners missing a huge proportion of an application's attack surface.

API scanning with Burp Suite

Burp Scanner - the scanning engine at the core of both Burp Suite Enterprise Edition and Burp Suite Professional - has the ability to parse API definitions to identify hidden endpoints. This helps you to scan it all - enabling organizations to gain visibility of their security posture like never before, and freeing up more time for testers to do what they do best.