Setting Up SSL Inspection in Developer Environments to Defend Against Advanced Threats

In today's digital landscape, SSL/TLS inspection is crucial for maintaining robust security. Having said that, implementing SSL inspection in developer environments can be a challenge. This is because developers often use a variety of tools and applications that may not trust the default system certificate store or perform certificate pinning, leading to unexpected errors. Dev teams must overcome this hurdle and find a way to implement SSL/TLS inspection in order to  defend against advanced threats and ensure secure communications.

How development teams can utilize the correct certificate store

Developers have options when it comes to integrating their tools with the correct certificate store, a key requirement for enabling encrypted traffic inspection with various development tools:

1. Create an SSL Inspection policy in Zscaler Internet Access that uses the “System & Development”  Cloud Applications category  to bypass the traffic associated with developer tools. Customers can still inspect other traffic developers generate.
2. Inspect traffic by integrating development tools with the appropriate certificate store as per the examples provided in this blog. Zscaler empowers developers with integrations with 20+ tools, components and programming languages beyond those covered in this blog including Rust, Ruby, Microsoft Azure CLI and others.
3. Automate using the correct certificate store so developers can focus on higher-value work: in the future Zscaler will enable customers’ dev teams to utilize the correct certificate store automatically so that software engineers can focus on keeping their development pipelines moving rapidly.

How trusted SSL inspection via proxy works

Before examining developer challenges with encrypted traffic inspection, let’s look at the mechanics of the SSL inspection process.

• The Zscaler Enforcement Node (ZEN in the diagram above) proxies the TLS connection by assuming both the role of a TLS server to the TLS client (the end user application) and of a TLS client facing the destination TLS server.
• The Zscaler service completes a server-side SSL handshake with the server, agreeing on a symmetric session key used to encrypt/decrypt the traffic on the server-side and validating the server’s certificate, similar to what an end user’s browser would do. 
• Further, the service generates a domain certificate that’s similar to the original certificate, but signed using the Zscaler intermediate CA or a customer intermediate CA private key. 
• The service then sends the certificate to the client for validation with the goal of completing  a client-side handshake and agreeing on a different symmetric session key used to encrypt/decrypt the traffic from the client side. At this point (step 6 in the figure above), certificate verification stops working. This prevents a trusted MITM proxy like Zscaler from intercepting, decrypting traffic before re-encrypting the session and sending it to its destination.

Why SSL inspection is challenging in developer environments

SSL inspection involves decrypting, inspecting, and re-encrypting TLS/SSL traffic. This process can be problematic in developer environments because of:

• Non-standard certificate stores: Many development tools and apps do not use the default system certificate store or do not trust Zscaler's certificates. This leads to trust issues with TLS/SSL certificates that, in turn, cause unexpected errors or behaviors that otherwise would not appear.
• Diverse tool sets with divergent certificate handling methods: Developers use a wide range of tools and libraries, each with their own methods of handling TLS/SSL certificates. Some applications use their own application trust store instead of using the default system store. In these instances, the application is unable to validate the TLS interception certificate.

Key components of a developer-friendly enterprise SSL solution

In this blog we’ll review a few examples of how Zscaler can integrate with various developer tools to active encrypted traffic inspection. Our new Software Developer Solution Guide to TLS and Certificate Integration provides multiple solutions to overcome these certificate challenges, enabling effective SSL/TLS inspection in developer environments. Zscaler offers three methods to address certificate challenges and enable SSL/TLS inspection:

1. SSL Inspection Policies: Zscaler allows administrators to create specific SSL inspection policies to bypass certain domains or inspect traffic based on URL categories.
2. Intermediate CA Certificates: Zscaler can dynamically generate and issue certificates on behalf of the server, which are then trusted by the client endpoint.
3. Custom Certificate Bundles: Developers can create custom certificate bundles that include the Zscaler root certificate, ensuring that their tools and applications trust the intercepted traffic.

Building a custom CA-bundle for TLS/SSL inspection

To enable TLS/SSL inspection for various development tools, you need to build a custom CA-bundle that includes the Zscaler root certificate. Here is a step-by-step guide for configuring TLS/SSL inspection for Docker, Git, NPM, Oracle Java, Python, Python PIP/Conda, and the Python urllib3 and requests libraries. These are just a few of the developer tools for which Zscaler can help customers operationalize the inspection of encrypted traffic in developer environments.

1. Docker

Docker uses the system's CA certificates by default. To add the Zscaler root certificate, follow these steps:

• Export the Zscaler root certificate from your system's certificate store.
• Copy the certificate to the Docker certificate directory (e.g., /etc/docker/certs.d on Linux).
• Restart the Docker daemon to apply the changes.

2. Git

To configure Git to trust the Zscaler root certificate, follow these steps:

• Export the Zscaler root certificate and save it as zscaler.crt.
• Configure Git to use the custom certificate bundle: 

git config --global http.sslCAInfo /path/to/zscaler.crt

3. NPM

NPM can be configured to use a custom CA bundle by setting the cafile configuration option:

• Export the Zscaler root certificate and save it as zscaler.crt.
• Configure NPM to use the custom certificate bundle:

npm config set cafile /path/to/zscaler.crt

4. Oracle Java

Oracle Java uses its own certificate store, typically located in the cacerts file. To add the Zscaler root certificate:

• Export the Zscaler root certificate and save it as zscaler.crt.
• Use the keytool command to import the certificate:

keytool -importcert -file /path/to/zscaler.crt -alias zscaler -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit

5. Python

Python's urllib3 and requests libraries, as well as PIP/Conda, can be configured to use a custom CA bundle:

• Export the Zscaler root certificate and save it as zscaler.crt.
• For urllib3, configure the PoolManager to use the custom CA bundle:

import certifi
import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED', ca_certs='/path/to/zscaler.crt')

• For requests, specify the custom CA bundle when making requests:

import requests
response = requests.get('https://example.com', verify='/path/to/zscaler.crt')

• For PIP, use the --cert option or set the PIP_CERT environment variable:

pip install --cert /path/to/zscaler.crt somepackage
export PIP_CERT=/path/to/zscaler.crt

By following these steps, you can ensure that your developer team’s tools and applications trust the Zscaler root certificate, enabling effective TLS/SSL inspection and enhancing your security posture.

Conclusion

Implementing SSL inspection in developer environments can be challenging due to non-standard certificate stores, certificate pinning, and diverse toolsets, but Zscaler provides multiple solutions to overcome these challenges. We can help you build a custom CA-bundle and configure your development tools to trust the Zscaler root certificate, so you can enable TLS/SSL inspection and ensure secure communications in your developer environment.

This blog just scratches the surface of how developers can achieve full inspection of encrypted traffic with Zscaler. Customers can get a free copy of the Software Developer Solution Guide to TLS and Certificate Integration now in the Zscaler Customer Success Center.