Cloning Yourself on WhatsApp with AI in Python
Integrating OpenAI and Twilio for Chatbot Interactions Which Mimic Your Chatting Style and Understands Image Inputs
Have you ever wished you could clone yourself to keep up with all the WhatsApp messages from your friends, family, and colleagues? Well, in this article, we’ll attempt to do just that in this article.
Consequently, we’re going to show how to create a chatbot that can handle some of those conversations for you.
Specifically, using OpenAI’s GPT models and Twilio’s WhatsApp API, we’ll build a digital buddy that can mimic your chatting style and understand images to converse for you.
This article is structured as follows:
- 1. Setup Environment, Accounts and Credentials
- 2. WhatsApp GPT-4 Chabot that Understands Images
- 3. OpenAI Assistant That Mimics Conversation Style
1. Environment Setup
Step 1. Sign Up for Twilio
- Go to the Twilio website.
- Sign up for an free-trail account. You’ll get $15 free credit.
- Verify your email and phone number as part of the sign-up process.
Step 2. Set Up Your Twilio WhatsApp Sandbox
Firstly, log into the Twilio Console. Then, navigate through ‘All Products & Services’ to ‘Programmable Messaging’ and select ‘Try it Out’ > ‘Try WhatsApp’. Subsequently, follow the instructions to set up your WhatsApp sandbox.
Step 3. Set up Ngrok
Ngrok is a tool that creates a secure tunnel to your local development environment, allowing you to expose your local server to the internet.
This is particularly useful for testing webhook integrations with services like Twilio.
1. Download ngrok from ngrok.com/download.
2. Unzip the Downloaded File: Subsequently, once downloaded, proceed to unzip the ngrok file.
3. Authenticate Ngrok: Before using ngrok, you need to authenticate it using your ngrok auth token. You can find your auth token on your ngrok dashboard. Open a terminal or command prompt and run the following command, replacing YOUR_AUTH_TOKEN with your actual ngrok auth token:
Figure. 1: Ngrok Authentication Setup: Instructions for configuring the Ngrok service to create secure tunnels to local servers, critical for testing webhooks and local development environments.
4. Run Ngrok to Tunnel Port 5000: Since Flask applications typically run on port 5000 by default, you’ll want to tunnel this port. In your terminal or command prompt, navigate to the directory where ngrok is located and run:
ngrok http 5000
If the command does not work for you, try this one instead:
./ngrok http 5000
Figure. 2: Ngrok Tunnel Status: A snapshot of an active Ngrok session, displaying the connection status, version, and the public URL for accessing the local server."
5. Note the Forwarding URL: Once ngrok is running, it will provide you with a forwarding URL, which looks something like https://12345.ngrok.io. This URL is what you’ll use as the webhook URL in your Twilio configuration to receive incoming messages from WhatsApp.
Important Notes:
Continuous Operation: Keep ngrok running while you’re testing your application. If you stop ngrok, the forwarding URL will cease to function, and consequently, you will need to restart ngrok and update the webhook URL in Twilio with the new forwarding URL.
URL Stability: Additionally, the free version of ngrok offers temporary URLs that change with every restart of ngrok. If you require a stable URL for long-term testing or development purposes, you should consider upgrading to a paid ngrok plan.
Step 4. Configure Your Twilio Sandbox
- Go back to your Twilio Console’s WhatsApp sandbox.
- Configure the ‘WHEN A MESSAGE COMES IN’ webhook with your ngrok URL followed by the
/whatsappendpoint. It should look likehttps://12345.ngrok.io/whatsapp. See image below for illustration. - Save your changes.
Figure. 3: Twilio Sandbox Configuration for WhatsApp: This configuration is needed, as it allows developers to test WhatsApp integrations before going live.
Step 5. Obtain Twilio Credentials
In your Twilio Dashboard, go to “Settings” and note down your “Account SID” and “Auth Token”. You will need these to authenticate API requests in Python.
Figure. 4: Twilio Console with Auth Tokens and Credentials: The Twilio dashboard showing authentication tokens and credentials required for API access, emphasizing the importance of secure storage and management.
Step 6. Obtain OpenAI API Credentials
To interact with OpenAI’s models, you’ll need to obtain API credentials, which will authenticate your requests.
Figure. 5: API Keys Management Screen: A secure interface for managing and creating API keys, essential for authenticating and interacting with external services.
2. Python Automation Script
2.1. Install Python Libraries
nitially, before we dive into coding, it’s essential to install several required libraries. Open your terminal or command prompt and enter the following command:
pip install flask twilio openai
2.2. Set Credentials in Python
Subsequently, we proceed to configure our credentials directly in our code editor. We are using a Jupyter Notebook inside VS Code for Simplicity.
Make sure to replace the placeholders with your actual Twilio and OpenAI credentials.
from flask import Flask, request
from twilio.rest import Client
import openai
import requests
from requests.auth import HTTPBasicAuth
import base64
import os
# Twilio credentials
account_sid = ''
auth_token = ''
twilio_number = 'whatsapp:+14155238886'
twilio_client = Client(account_sid, auth_token)
# OpenAI API Key
openai_api_key = ''
2.3. Setting Up the Flask Application
Furthermore, by employing Flask, we establish a web server specifically designed to manage incoming HTTP requests from Twilio.
Additionally, this setup enables us to define routes (URL endpoints) effectively.
Below the credentials in our Python file, initialize the Flask application and set up a directory for uploads.
app = Flask(__name__)
uploads_dir = 'uploads'
os.makedirs(uploads_dir, exist_ok=True)
This snippet initializes the Flask application and prepares a directory for uploads.
2.4. Integrating Twilio with WhatsApp
2.4.1 Using Twilio’s Sandbox for Simplicity
By utilizing Twilio’s sandbox for WhatsApp, we simplify the messaging process significantly. This method allows us to send and receive messages without the complexities of setting up a direct business API.
2.4.2 Message Reception and Handling
Each time our WhatsApp number receives a message, Twilio sends an HTTP POST request to a designated route within our Flask application. This system uses Twilio’s robust API to manage communications across various platforms including SMS, voice, video, email, and messaging apps like WhatsApp.
2.4.3 Workflow Explanation
Here’s a breakdown of the process:
- Listening for Requests: Our Flask application runs on a server, tuned to listen for incoming HTTP POST requests on a specified route.
- Receiving Messages: Upon receiving a message, Twilio forwards an HTTP POST request to this route.
- Processing and Responding: The server then processes this request and takes necessary actions, such as replying to the message, before sending a response back through Twilio.
2.4.4 Handling Media Messages
In addition to text, handling media messages like images involves processing the MediaUrl parameters that Twilio sends.
These URLs direct us to the media content, which can then be retrieved and processed by our server.
For more information on handling incoming media, see the section on receiving media messages with Twilio.
@app.route("/whatsapp", methods=['POST'])
def reply_whatsapp():
# We'll process incoming messages here
pass
2.5. Processing Incoming Messages
Within the reply_whatsapp function, begin by processing the incoming messages and setting up a dedicated space to store them.
To effectively handle incoming messages in Flask, follow these steps:
- Route Definition: Initially, define a route that Twilio can utilize to send notifications when a new message arrives.
- Data Extraction: Subsequently, extract the message content and sender information from the incoming request.
- Response Strategy: Next, determine how to respond or act based on the content of the message.
In our case, as we are developing an automated chatbot, we will further need to take the text or media URL from the incoming message and utilize it to craft an appropriate response.
conversation_histories = {}
@app.route("/whatsapp", methods=['POST'])
def reply_whatsapp():
incoming_msg = request.values.get('Body', '').lower()
from_number = request.values.get('From')
user_history = conversation_histories.get(from_number, [])
# We will continue to build on this function
return 'OK', 200
2.6. Integrating OpenAI’s GPT Models
Furthermore, we’ll use two models from OpenAI for different purposes:
- GPT-4 for Text: These models can understand and generate human-like text based on the input they’re given. In the context of a WhatsApp bot, they can be used to generate replies to text messages.
- GPT-4 Vision for Images: When handling media, like images, we can use the vision capabilities of the model to understand the content of an image and even provide a description.
Additionally, to manage the conversation, we use the conversation_histories dictionary to store the context of each chat.
Moreover, when we get a message from a user, we send their conversation history along with their new message to OpenAI, which allows the model to generate a coherent response.
2.7. Building a Conversation History
Keeping a conversation history allows our AI to maintain context, leading to more coherent and relevant responses over the course of an interaction.
How to Build It:
- Storing Conversations: Firstly, store each conversation in a dictionary using the sender’s number as the key.
- Updating History: Next, append each new message to the conversation history before it is sent to OpenAI’s API.
- Maintaining Continuity: Finally, ensure that the history is consistently passed along with each API request.
2.8. Complete Code
Now let’s piece everything together with the complete code. Here is what the entire script will look like with further elaboration on its components:
Newsletter