bobmaster/matrix_chatgpt_bot
1
0
Fork 0
Code Issues 5 Pull requests 2 Projects Releases 10 Packages Wiki Activity

how to have longer output using gpt-3.5-turbo-16k model #28

New issue
Open
opened 2023-06-24 09:35:49 +08:00 by jaodei · 7 comments
jaodei commented 2023-06-24 09:35:49 +08:00 (Migrated from github.com)
Copy link

I manually changed the model in settings.js in node-chatgpt-api container to gpt-3.5-turbo-16k model and also specified max_token to say 12000 tokens. See below. However, when I asked the bot to create a 5000 words story, each time the generated text is quite short, maybe around 700 words. Is there elsewhere limiting the output length?

        payload.update(
            {
                "clientOptions": {
                    "clientToUse": "chatgpt",
                    "openaiApiKey": self.openai_api_key,
                    "modelOptions": {
                        "temperature": self.temperature,
                        "max_tokens": 12000,
                        "model": self.chatgpt_model,
                    },
                    "options": {
                        "maxContextTokens": 16384,
                    },
                }
            }
I manually changed the model in settings.js in node-chatgpt-api container to gpt-3.5-turbo-16k model and also specified max_token to say 12000 tokens. See below. However, when I asked the bot to create a 5000 words story, each time the generated text is quite short, maybe around 700 words. Is there elsewhere limiting the output length? payload.update( { "clientOptions": { "clientToUse": "chatgpt", "openaiApiKey": self.openai_api_key, "modelOptions": { "temperature": self.temperature, "max_tokens": 12000, "model": self.chatgpt_model, }, "options": { "maxContextTokens": 16384, }, } }
hibobmaster commented 2023-06-24 10:10:10 +08:00 (Migrated from github.com)
Copy link

Enable debug mode and get the payload/response from console
217c189cbb/settings.example.js (L37)

Enable debug mode and get the payload/response from console https://github.com/waylaidwanderer/node-chatgpt-api/blob/217c189cbbc7749da0f86631ca0f6dd7f5a43cb0/settings.example.js#L37
jaodei commented 2023-06-24 10:30:41 +08:00 (Migrated from github.com)
Copy link

not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant?

Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite.
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt |
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life.
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt |
|_

not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant? _Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt |_ |_
jaodei commented 2023-06-24 13:47:49 +08:00 (Migrated from github.com)
Copy link

not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant?

Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | |_

Sorry my fault. I was looking at the log info of matrix_chat_bot, not node-chatgpt-bot.

> not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant? > > _Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt |_ |_ Sorry my fault. I was looking at the log info of matrix_chat_bot, not node-chatgpt-bot.
jaodei commented 2023-06-24 14:00:44 +08:00 (Migrated from github.com)
Copy link

Got the log as below. It appears setting max_tokens correctly to 12000. why can't it produce 4000 words as requested? An issue of openai itself?

Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api |
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | https://api.openai.com/v1/chat/completions
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | {
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | model: 'gpt-3.5-turbo-16k-0613',
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | temperature: 0.8,
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | max_tokens: 12000,
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | top_p: 1,
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | presence_penalty: 1,
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | stop: [ '||>', '\nUser:', '<|diff_marker|>' ],
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | messages: [
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | {
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system',
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | name: 'instructions',
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>Instructions:\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'You are ChatGPT, a large language model trained by OpenAI. Respond conversationally.\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'Current date: June 24, 2023\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '\n'
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | },
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | {
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system',
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>User:\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'write a 4000 words story about a boy likes fishing\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '||>ChatGPT:\n'
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | }
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | ]
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | }

Got the log as below. It appears setting max_tokens correctly to 12000. why can't it produce 4000 words as requested? An issue of openai itself? _Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | https://api.openai.com/v1/chat/completions Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | { Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | model: 'gpt-3.5-turbo-16k-0613', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | temperature: 0.8, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | max_tokens: 12000, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | top_p: 1, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | presence_penalty: 1, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | stop: [ '||>', '\nUser:', '<|diff_marker|>' ], Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | messages: [ Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | { Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | name: 'instructions', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>Instructions:\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'You are ChatGPT, a large language model trained by OpenAI. Respond conversationally.\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'Current date: June 24, 2023\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '\n' Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | }, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | { Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>User:\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'write a 4000 words story about a boy likes fishing\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '||>ChatGPT:\n' Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | } Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | ] Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | }_
hibobmaster commented 2023-06-24 18:10:47 +08:00 (Migrated from github.com)
Copy link

I use another tool and got the same result:
prompt: write a 4000 words story about a boy likes fishing
image

Maybe you could try openai playground to verify it.
https://platform.openai.com/playground?mode=chat&model=gpt-3.5-turbo-16k

I use another tool and got the same result: prompt: write a 4000 words story about a boy likes fishing ![image](https://github.com/hibobmaster/matrix_chatgpt_bot/assets/32976627/46ebc01b-8602-4d1b-8fcc-679d813b3d37) Maybe you could try openai playground to verify it. https://platform.openai.com/playground?mode=chat&model=gpt-3.5-turbo-16k
jaodei commented 2023-06-25 07:00:21 +08:00 (Migrated from github.com)
Copy link

wierd. On playground, the maximum token is only 2048 even for gpt-3.5-turbo-16k model.

wierd. On playground, the maximum token is only 2048 even for gpt-3.5-turbo-16k model.
hibobmaster commented 2023-06-25 12:27:54 +08:00 (Migrated from github.com)
Copy link

Some other people have the same question.
https://www.reddit.com/r/ChatGPT/comments/10ddllh/got_tips_for_how_to_get_chatgpt_to_write_a_long/

Some other people have the same question. https://www.reddit.com/r/ChatGPT/comments/10ddllh/got_tips_for_how_to_get_chatgpt_to_write_a_long/
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
dependabot/pip/aiohttp-3.8.5
dependabot/pip/cryptography-41.0.2
dev
v1.2.0
v1.1.1
v1.1.0
v1.0.6
v1.0.5
v1.0.4
v1.0.3
v1.0.2
v1.0.1
v1.0.0
Labels
Clear labels   
bug

Something isn't working

  
dependencies

Pull requests that update a dependency file

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
waiting-reply

   
wontfix

This will not be worked on

No labels
bug
dependencies
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
waiting-reply
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: bobmaster/matrix_chatgpt_bot#28
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?