Spend Tokens to Make Tokens
8 min readMar 20, 2023
The old adage, you have to spend money to make money can bear fruit if you know what to spend it on. In the case of ChatGPT, you have to spend tokens to carry on the conversation. On other words, you have to provide context for your (able) assistant to give you meaningful answers. That’s because the endpoint (on the server) is purely a prompt driven generative engine (albeit, a very knowledgeable one) and yet does not hold any state. State (in the form of context) is provided by your app.
We demonstrate this with a very simple ChatGPT app that we can run two ways, one a stateless version (which doesn’t move the conversation forward) and the other a stateful version (that does move the conversation forward):
python repl-qa.py and
python repl-qa.py --statefulBelow you see the stateless version where the bot has already lost the plot as it does not remember my name or my dog’s name. It’s unable to answer the question, What’s my dog’s name or What’s my name (this was provided earlier in the conversation). This is simply because the app does not append the prevision conversations (context) along with the current question. And that’s what I mean by ‘spending tokens’, i.e, appending context.
$ python repl-qa.py
stateful state: False
Enter a question for the assistant (or type '--quit' to exit): Hello, my name is Auro and my dog's name is Skarky.
"You entered: ", Hello, my name is Auro and my dog's name is Skarky.
Messages into ChatGPT:
{'role': 'system', 'content': 'You are a helpful assistant.'}
{'role': 'assistant', 'content': ''}
{'role': 'user', 'content': "Hello, my name is Auro and my dog's name is Skarky."}
Total tokens usage: 65
ChatGPT response: Hello Auro, it's nice to meet you and your dog Skarky! How can I assist you today?
Enter a question for the assistant (or type '--quit' to exit): Sparky has parvo. What shall I do?
"You entered: ", Sparky has parvo. What shall I do?
Messages into ChatGPT:
{'role': 'system', 'content': 'You are a helpful assistant.'}
{'role': 'assistant', 'content': "Hello Auro, it's nice to meet you and your dog Skarky! How can I assist you today?"}
{'role': 'user', 'content': 'Sparky has parvo. What shall I do?'}
Total tokens usage: 315
ChatGPT response: I'm sorry to hear that Sparky has parvo. Parvo is a serious and highly contagious viral disease that affects dogs, especially puppies. It is important to seek veterinary care immediately if you suspect your dog has parvo.
Here are some steps you can take to help your dog:
1. Take your dog to the vet: Your vet will be able to diagnose parvo and provide treatment options. Treatment may include hospitalization, intravenous fluids, antibiotics, and anti-nausea medication.
2. Isolate your dog: Parvo is highly contagious, so it's important to isolate your dog from other dogs to prevent the spread of the virus. Keep your dog away from other dogs and clean any areas where your dog has been with a bleach solution.
3. Keep your dog hydrated: Parvo can cause severe dehydration, so it's important to keep your dog hydrated. Offer your dog small amounts of water frequently or use a syringe to give your dog water if they are not drinking on their own.
4. Follow your vet's instructions: Your vet will provide specific instructions for caring for your dog at home. Follow these instructions carefully to ensure your dog recovers as quickly as possible.
I hope this information helps, and I wish Sparky a speedy recovery.
Enter a question for the assistant (or type '--quit' to exit): What is my dog's name?
"You entered: ", What is my dog's name?
Messages into ChatGPT:
{'role': 'system', 'content': 'You are a helpful assistant.'}
{'role': 'assistant', 'content': "I'm sorry to hear that Sparky has parvo. Parvo is a serious and highly contagious viral disease that affects dogs, especially puppies. It is important to seek veterinary care immediately if you suspect your dog has parvo.\n\nHere are some steps you can take to help your dog:\n\n1. Take your dog to the vet: Your vet will be able to diagnose parvo and provide treatment options. Treatment may include hospitalization, intravenous fluids, antibiotics, and anti-nausea medication.\n\n2. Isolate your dog: Parvo is highly contagious, so it's important to isolate your dog from other dogs to prevent the spread of the virus. Keep your dog away from other dogs and clean any areas where your dog has been with a bleach solution.\n\n3. Keep your dog hydrated: Parvo can cause severe dehydration, so it's important to keep your dog hydrated. Offer your dog small amounts of water frequently or use a syringe to give your dog water if they are not drinking on their own.\n\n4. Follow your vet's instructions: Your vet will provide specific instructions for caring for your dog at home. Follow these instructions carefully to ensure your dog recovers as quickly as possible.\n\nI hope this information helps, and I wish Sparky a speedy recovery."}
{'role': 'user', 'content': "What is my dog's name?"}
Total tokens usage: 321
ChatGPT response: I'm sorry, but as an AI language model, I don't have access to information about your dog's name. Could you please tell me your dog's name?
Enter a question for the assistant (or type '--quit' to exit): What is my name?
"You entered: ", What is my name?
Messages into ChatGPT:
{'role': 'system', 'content': 'You are a helpful assistant.'}
{'role': 'assistant', 'content': "I'm sorry, but as an AI language model, I don't have access to information about your dog's name. Could you please tell me your dog's name?"}
{'role': 'user', 'content': 'What is my name?'}
Total tokens usage: 91
ChatGPT response: I'm sorry, but as an AI language model, I don't have access to personal information. Could you please tell me your name?
Enter a question for the assistant (or type '--quit' to exit): Below, you find the stateful version and observe that ChatGPT is able to provide my dog’s name and my name after asking for advice on my dog’s ailment. This is because, at every interaction, the entire conversation is provided as context. Obviously, we would soon run out of tokens after a few more rounds of back and forth in the conversation. There are ways to address this (thru summaries) that we’ll take up in another post.
$ python repl-qa.py --stateful.
stateful state: True
Enter a question for the assistant (or type '--quit' to exit): Hello, my name is Auro and my dog's name is Sparky
"You entered: ", Hello, my name is Auro and my dog's name is Sparky
Messages into ChatGPT:
{'role': 'system', 'content': 'You are a helpful assistant.'}
{'role': 'assistant', 'content': ''}
{'role': 'user', 'content': "Hello, my name is Auro and my dog's name is Sparky"}
Total tokens usage: 60
ChatGPT response: Hello Auro, it's nice to meet you and Sparky! How can I assist you today?
Enter a question for the assistant (or type '--quit' to exit): My dog has parvo. What shall I do?
"You entered: ", My dog has parvo. What shall I do?
Messages into ChatGPT:
{'role': 'system', 'content': 'You are a helpful assistant.'}
{'role': 'assistant', 'content': ''}
{'role': 'user', 'content': "Hello, my name is Auro and my dog's name is Sparky"}
{'role': 'assistant', 'content': "Hello Auro, it's nice to meet you and Sparky! How can I assist you today?"}
{'role': 'user', 'content': 'My dog has parvo. What shall I do?'}
Total tokens usage: 294
ChatGPT response: I'm sorry to hear that your dog has parvo. Parvo is a serious and potentially life-threatening illness, so it's important to seek veterinary care as soon as possible. Here are some steps you can take:
1. Take your dog to the vet immediately. The vet will be able to diagnose parvo and provide treatment options.
2. Keep your dog hydrated. Parvo can cause severe dehydration, so it's important to make sure your dog has access to clean water at all times. You can also give your dog electrolyte solutions to help replace lost fluids.
3. Isolate your dog. Parvo is highly contagious, so it's important to keep your dog away from other dogs to prevent the spread of the virus.
4. Follow your vet's treatment plan. Your vet may recommend hospitalization, IV fluids, antibiotics, and other medications to help your dog recover from parvo. It's important to follow their instructions closely to give your dog the best chance of recovery.
I hope this helps, and I wish your dog a speedy recovery!
Enter a question for the assistant (or type '--quit' to exit): What's my dog's name?
"You entered: ", What's my dog's name?
Messages into ChatGPT:
{'role': 'system', 'content': 'You are a helpful assistant.'}
{'role': 'assistant', 'content': ''}
{'role': 'user', 'content': "Hello, my name is Auro and my dog's name is Sparky"}
{'role': 'assistant', 'content': "Hello Auro, it's nice to meet you and Sparky! How can I assist you today?"}
{'role': 'user', 'content': 'My dog has parvo. What shall I do?'}
{'role': 'assistant', 'content': "I'm sorry to hear that your dog has parvo. Parvo is a serious and potentially life-threatening illness, so it's important to seek veterinary care as soon as possible. Here are some steps you can take:\n\n1. Take your dog to the vet immediately. The vet will be able to diagnose parvo and provide treatment options.\n\n2. Keep your dog hydrated. Parvo can cause severe dehydration, so it's important to make sure your dog has access to clean water at all times. You can also give your dog electrolyte solutions to help replace lost fluids.\n\n3. Isolate your dog. Parvo is highly contagious, so it's important to keep your dog away from other dogs to prevent the spread of the virus.\n\n4. Follow your vet's treatment plan. Your vet may recommend hospitalization, IV fluids, antibiotics, and other medications to help your dog recover from parvo. It's important to follow their instructions closely to give your dog the best chance of recovery.\n\nI hope this helps, and I wish your dog a speedy recovery!"}
{'role': 'user', 'content': "What's my dog's name?"}
Total tokens usage: 319
ChatGPT response: Your dog's name is Sparky.
Enter a question for the assistant (or type '--quit' to exit): What's my name?
"You entered: ", What's my name?
Messages into ChatGPT:
{'role': 'system', 'content': 'You are a helpful assistant.'}
{'role': 'assistant', 'content': ''}
{'role': 'user', 'content': "Hello, my name is Auro and my dog's name is Sparky"}
{'role': 'assistant', 'content': "Hello Auro, it's nice to meet you and Sparky! How can I assist you today?"}
{'role': 'user', 'content': 'My dog has parvo. What shall I do?'}
{'role': 'assistant', 'content': "I'm sorry to hear that your dog has parvo. Parvo is a serious and potentially life-threatening illness, so it's important to seek veterinary care as soon as possible. Here are some steps you can take:\n\n1. Take your dog to the vet immediately. The vet will be able to diagnose parvo and provide treatment options.\n\n2. Keep your dog hydrated. Parvo can cause severe dehydration, so it's important to make sure your dog has access to clean water at all times. You can also give your dog electrolyte solutions to help replace lost fluids.\n\n3. Isolate your dog. Parvo is highly contagious, so it's important to keep your dog away from other dogs to prevent the spread of the virus.\n\n4. Follow your vet's treatment plan. Your vet may recommend hospitalization, IV fluids, antibiotics, and other medications to help your dog recover from parvo. It's important to follow their instructions closely to give your dog the best chance of recovery.\n\nI hope this helps, and I wish your dog a speedy recovery!"}
{'role': 'user', 'content': "What's my dog's name?"}
{'role': 'assistant', 'content': "Your dog's name is Sparky."}
{'role': 'user', 'content': "What's my name?"}
Total tokens usage: 345
ChatGPT response: Your name is Auro, as you mentioned earlier.
Enter a question for the assistant (or type '--quit' to exit): To run this yourself, get an OpenAPI key and run the code provided.
