How to capture HTTPS traffic from Python requests or httpx library

Cover Image for How to capture HTTPS traffic from Python requests or httpx library

This tutorial will show you how to capture HTTPS traffic from Python requests or httpx library for debugging purpose.

Here is what we achieve in this tutorial:

  • Setup Proxyman
  • ✅ Capture HTTP/HTTPS request response from Python code without changing any code
  • Works with requests and httpx library

Here is the Source code for this tutorial

Prerequisites

  • Python 3.12.0
  • requests or httpx library
  • Proxyman macOS (Download from Proxyman.com)

Let's start

  1. Download Proxyman from Proxyman.com
  2. Open Proxyman and install the certificate to your macOS Devices. You can find the certificate in Proxyman > Certificate Menu -> Install for macOS -> Follow the instruction
  3. In Proxyman -> Setup Menu -> Automatic Setup -> Click on the "Open new Terminal" button
Proxyman Automatic Setup
Proxyman Automatic Setup

TIP #1: You can choose what Terminal should open. Support iTerm, Terminal, Hyper, etc.

TIP #2: If you want to use your own Terminal -> Let's open the Setup Menu -> Manual Setup -> Copy the command -> Open your Terminal and paste the command

Proxyman Manual Setup
Proxyman Manual Setup
  1. On this terminal -> Run your python code.

If you don't have any python code, you can use the example code in the Source code

pip install requests httpx
import requests
import httpx

def make_post_request(): # The URL we're sending the POST request to
url = 'https://httpbin.org/post'

  # Sample data to send in the POST request
  data = {
      'name': 'John Doe',
      'age': 30,
      'message': 'Hello, World!'
  }

  try:
      # Make the POST request
      response = requests.post(url, json=data)

      # Ensure the request was successful
      response.raise_for_status()

      # Print the response
      print("Response Status Code:", response.status_code)
      print("
Response Content:")
      print(response.json())

  except requests.exceptions.RequestException as e:
      print(f"An error occurred: {e}")

def make_post_request_httpx(): # The URL we're sending the POST request to
url = 'https://httpbin.org/post'

  # Sample data to send in the POST request
  data = {
      'name': 'John Doe',
      'age': 30,
      'message': 'Hello, World!'
  }

  try:
      # Create an httpx client
      with httpx.Client() as client:
          # Make the POST request
          response = client.post(url, json=data)

          # Ensure the request was successful
          response.raise_for_status()

          # Print the response
          print("Response Status Code:", response.status_code)
          print("
Response Content:")
          print(response.json())

  except httpx.RequestError as e:
      print(f"An error occurred while making the request: {e}")
  except httpx.HTTPStatusError as e:
      print(f"HTTP Error occurred: {e}")

if **name** == "**main**":
make_post_request()
print("
Now trying with httpx:")
make_post_request_httpx()

Run:

python main.py
Run Python Code on Proxyman Terminal
Run Python Code on Proxyman Terminal

You will see the output in Proxyman.

  1. Proxyman automatically captures the HTTP/HTTPS traffic from your Python code.
Proxyman Capture HTTPS Traffic from Python code
Proxyman Capture HTTPS Traffic from Python code
  1. Done ✅

Summary

Capture HTTP/HTTPS traffic from Python code is very easy with Proxyman. No need to modify any code.

Otherwise, you have to manually change your source code to trust the Proxyman certificate and set a proxy server, which is not a good idea.

What's next?

The automatic or Manual Setup can capture the HTTP/HTTPS traffic from any Python, Ruby or NodeJS code. No need to modify any code.

Noah Tran
Noah Tran