how to have longer output using gpt-3.5-turbo-16k model #28

Open
opened 2023-06-24 01:35:49 +00:00 by jaodei · 7 comments
jaodei commented 2023-06-24 01:35:49 +00:00 (Migrated from github.com)

I manually changed the model in settings.js in node-chatgpt-api container to gpt-3.5-turbo-16k model and also specified max_token to say 12000 tokens. See below. However, when I asked the bot to create a 5000 words story, each time the generated text is quite short, maybe around 700 words. Is there elsewhere limiting the output length?

        payload.update(
            {
                "clientOptions": {
                    "clientToUse": "chatgpt",
                    "openaiApiKey": self.openai_api_key,
                    "modelOptions": {
                        "temperature": self.temperature,
                        "max_tokens": 12000,
                        "model": self.chatgpt_model,
                    },
                    "options": {
                        "maxContextTokens": 16384,
                    },
                }
            }
I manually changed the model in settings.js in node-chatgpt-api container to gpt-3.5-turbo-16k model and also specified max_token to say 12000 tokens. See below. However, when I asked the bot to create a 5000 words story, each time the generated text is quite short, maybe around 700 words. Is there elsewhere limiting the output length? payload.update( { "clientOptions": { "clientToUse": "chatgpt", "openaiApiKey": self.openai_api_key, "modelOptions": { "temperature": self.temperature, "max_tokens": 12000, "model": self.chatgpt_model, }, "options": { "maxContextTokens": 16384, }, } }
hibobmaster commented 2023-06-24 02:10:10 +00:00 (Migrated from github.com)

Enable debug mode and get the payload/response from console
217c189cbb/settings.example.js (L37)

Enable debug mode and get the payload/response from console https://github.com/waylaidwanderer/node-chatgpt-api/blob/217c189cbbc7749da0f86631ca0f6dd7f5a43cb0/settings.example.js#L37
jaodei commented 2023-06-24 02:30:41 +00:00 (Migrated from github.com)

not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant?

Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite.
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt |
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life.
Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt |

|_

not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant? _Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt |_ |_
jaodei commented 2023-06-24 05:47:49 +00:00 (Migrated from github.com)

not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant?

Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | |_

Sorry my fault. I was looking at the log info of matrix_chat_bot, not node-chatgpt-bot.

> not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant? > > _Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt |_ |_ Sorry my fault. I was looking at the log info of matrix_chat_bot, not node-chatgpt-bot.
jaodei commented 2023-06-24 06:00:44 +00:00 (Migrated from github.com)

Got the log as below. It appears setting max_tokens correctly to 12000. why can't it produce 4000 words as requested? An issue of openai itself?

Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api |
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | https://api.openai.com/v1/chat/completions
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | {
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | model: 'gpt-3.5-turbo-16k-0613',
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | temperature: 0.8,
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | max_tokens: 12000,
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | top_p: 1,
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | presence_penalty: 1,
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | stop: [ '||>', '\nUser:', '<|diff_marker|>' ],
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | messages: [
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | {
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system',
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | name: 'instructions',
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>Instructions:\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'You are ChatGPT, a large language model trained by OpenAI. Respond conversationally.\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'Current date: June 24, 2023\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '\n'
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | },
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | {
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system',
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>User:\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'write a 4000 words story about a boy likes fishing\n' +
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '||>ChatGPT:\n'
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | }
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | ]
Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | }

Got the log as below. It appears setting max_tokens correctly to 12000. why can't it produce 4000 words as requested? An issue of openai itself? _Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | https://api.openai.com/v1/chat/completions Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | { Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | model: 'gpt-3.5-turbo-16k-0613', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | temperature: 0.8, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | max_tokens: 12000, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | top_p: 1, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | presence_penalty: 1, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | stop: [ '||>', '\nUser:', '<|diff_marker|>' ], Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | messages: [ Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | { Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | name: 'instructions', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>Instructions:\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'You are ChatGPT, a large language model trained by OpenAI. Respond conversationally.\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'Current date: June 24, 2023\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '\n' Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | }, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | { Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>User:\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'write a 4000 words story about a boy likes fishing\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '||>ChatGPT:\n' Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | } Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | ] Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | }_
hibobmaster commented 2023-06-24 10:10:47 +00:00 (Migrated from github.com)

I use another tool and got the same result:
prompt: write a 4000 words story about a boy likes fishing
image

Maybe you could try openai playground to verify it.
https://platform.openai.com/playground?mode=chat&model=gpt-3.5-turbo-16k

I use another tool and got the same result: prompt: write a 4000 words story about a boy likes fishing ![image](https://github.com/hibobmaster/matrix_chatgpt_bot/assets/32976627/46ebc01b-8602-4d1b-8fcc-679d813b3d37) Maybe you could try openai playground to verify it. https://platform.openai.com/playground?mode=chat&model=gpt-3.5-turbo-16k
jaodei commented 2023-06-24 23:00:21 +00:00 (Migrated from github.com)

wierd. On playground, the maximum token is only 2048 even for gpt-3.5-turbo-16k model.

wierd. On playground, the maximum token is only 2048 even for gpt-3.5-turbo-16k model.
hibobmaster commented 2023-06-25 04:27:54 +00:00 (Migrated from github.com)
Some other people have the same question. https://www.reddit.com/r/ChatGPT/comments/10ddllh/got_tips_for_how_to_get_chatgpt_to_write_a_long/
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: bobmaster/matrix_chatgpt_bot#28
No description provided.