aljaz/dev-gpt
1
0
Fork 0
mirror of https://github.com/aljazceru/dev-gpt.git synced 2025-12-24 09:04:19 +01:00
Code Issues Packages Projects Releases Wiki Activity

🧪3️⃣ test: level 3 and 4

Browse Source
This commit is contained in:
Florian Hönicke
2023-05-01 21:32:18 +02:00
parent 1c5d8329f4
commit b6354d256d
4 changed files with 112 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
src/options/generate/generator.py
View File

@@ -461,8 +461,10 @@ gptdeploy deploy --path {self.microservice_root_path}
'task',
'',
template_pm_task_iteration,
micro_service_initial_description=f'''Microservice description:
micro_service_initial_description=f'''Microservice description:
```
{self.microservice_specification.task}
```
''',
)
self.refine_requirements(
@@ -510,8 +512,8 @@ Test scenario:
role='user'
)
messages.append(HumanMessage(content=user_input))
agent_question = self.extract_content_from_result(agent_response_raw, 'prompt.txt', can_contain_code_block=False)
final = self.extract_content_from_result(agent_response_raw, 'final.txt', can_contain_code_block=False)
agent_question = self.extract_content_from_result(agent_response_raw, 'prompt.json', can_contain_code_block=False)
final = self.extract_content_from_result(agent_response_raw, 'final.json', can_contain_code_block=False)
if final:
messages.append(AIMessage(content=final))
setattr(self.microservice_specification, refinement_type, final)

101
src/options/generate/templates_system.py
View File

@@ -40,9 +40,11 @@ a)
If the description is not sufficiently specified, then ask for the missing information.
Your response must exactly match the following block code format (double asterisks for the file name and triple backticks for the file block):
**prompt.txt**
```text
<prompt to the client here>
**prompt.json**
```json
{{
"question": "<prompt to the client here>"
}}
```
b)
@@ -50,9 +52,13 @@ Otherwise you respond with the summarized description.
The summarized description must contain all the information mentioned by the client.
Your response must exactly match the following block code format (double asterisks for the file name and triple backticks for the file block):
**final.txt**
```text
<task here>
**final.json**
```json
{{
"description": "<microservice description here>",
"implementation_details": "<additional implementation details here>",
"credentials: "<credentials here>"
}}
```
The character sequence ``` must always be at the beginning of the line.
@@ -64,9 +70,11 @@ output: defined
api access: not defined
database access: n/a
**prompt.txt**
```text
Please provide the url of the weather api and a valid api key or some other way accessing the api. Or let our engineers try to find a free api.
**prompt.json**
```json
{{
"question": "Please provide the url of the weather api and a valid api key or some other way accessing the api. Or let our engineers try to find a free api."
}}
```
Example for the description "convert png to svg":
@@ -75,9 +83,13 @@ output: defined
api access: n/a
database access: n/a
**final.txt**
```text
The user inserts a png and gets an svg as response.
**final.json**
```json
{{
"description": "The user inserts a png and gets an svg as response.",
"implementation_details": "n/a",
"credentials: "n/a"
}}
```
Example for the description "parser":
@@ -86,9 +98,11 @@ output: not defined
api access: n/a
database access: n/a
**prompt.txt**
```text
Please provide the input and output format.
**prompt.json**
```json
{{
"question": "Please provide the input and output format."
}}
```
'''
@@ -104,9 +118,11 @@ Your response must exactly match the following block code format (double asteris
1.
contains example: no
2.
**prompt.txt**
```text
<prompt to the client here>
**prompt.json**
```json
{{
"question": "<prompt to the client here>"
}}
```
If you did a, you must not do b.
@@ -117,10 +133,12 @@ Your response must exactly match the following block code format (double asteris
1.
contains example: yes (<insert example here>)
2.
**final.txt**
```text
input: "<input here>"
assertion: the output is of type <type here>
**final.json**
```json
{{
"input": "<input here>",
"assertion": "the output is of type <type here>"
}}
```
If you did b, you must not do a.
@@ -129,37 +147,46 @@ Example for: "given a city like "Berlin", get the weather report for the next 5
1.
contains example: yes (Berlin)
2.
**final.txt**
```text
input: "Berlin"
assertion: the output is of type string
**final.json**
```json
{{
"input": "Berlin",
"assertion": "the output is of type string"
}}
```
Example for "The user inserts a png and gets an svg as response.":
1.
contains example: no
2.
**prompt.txt**
```text
Please provide a png example input file as url.
**prompt.json**
```json
{{
"question": "Please provide a png example input file as url."
}}
```
Example for "The user inserts a png like https://aquasecurity.github.io/kube-bench/v0.6.5/images/kube-bench-logo-only.png and gets an svg as response.":
1.
contains example: yes (https://aquasecurity.github.io/kube-bench/v0.6.5/images/kube-bench-logo-only.png)
2.
**final.txt**
```text
input: "https://aquasecurity.github.io/kube-bench/v0.6.5/images/kube-bench-logo-only.png"
assertion: the output is of type svg
**final.json**
```json
{{
"input": "https://aquasecurity.github.io/kube-bench/v0.6.5/images/kube-bench-logo-only.png",
"assertion": "the output is of type svg"
}}
```
Example for "The microservice takes nothing as input and returns the current time.":
1.
contains example: n/a
**final.txt**
```text
input: "nothing"
assertion: the output is of type string
**final.json**
```json
{{
"input": "nothing",
"assertion": "the output is of type string"
}}
```
'''

64
src/options/generate/templates_user.py
View File

@@ -374,22 +374,28 @@ The playground (app.py) must not import the executor.
template_pm_task_iteration = PromptTemplate.from_template(
'''{micro_service_initial_description}
1.Quickly go through the checklist (input/output well defined? api or db access needed?) and think about if you should ask something to the client or if you should write the final description.
2.Either write the prompt.txt or the final.txt file.
2.Either write the prompt.json or the final.json file.
Either ask for clarification like this:
**prompt.txt**
```text
<prompt to the client here (must be only one question)>
**prompt.json**
```json
{{
"question": "<prompt to the client here (must be only one question)>"
}}
```
Or write the summarized microservice description like this:
**final.txt**
```text
<microservice description here>
Or write the summarized microservice description and additional implementation details like this:
**final.json**
```json
{{
"description": "<microservice description here>",
"implementation_details": "<additional implementation details here>",
"credentials: "<credentials here>"
}}
```
Note that your response must be either prompt.txt or final.txt. You must not write both.
Note that your response must be either prompt.json or final.json. You must not write both.
Note that you must obey the double asterisk and tripple backtick syntax from above.
Note that the last sequence of characters in your response must be ``` (triple backtick).
Note that prompt.txt must not only contain one question.
Note that prompt.json must not only contain one question.
Note that if urls, secrets, database names, etc. are mentioned, they must be part of the summary.
{custom_suffix}
'''
@@ -398,35 +404,41 @@ Note that if urls, secrets, database names, etc. are mentioned, they must be par
template_pm_test_iteration = PromptTemplate.from_template(
'''{micro_service_initial_description}
1. write down if the original description and the refined description contain an example input for the microservice.
2. write down either prompt.txt or final.txt.
If the example input for the microservice is mentioned in the refined description or the original description, then output final.txt.
Otherwise, output prompt.txt where you ask for the example input file as URL or the example string.
2. write down either prompt.json or final.json.
If the example input for the microservice is mentioned in the refined description or the original description, then output final.json.
Otherwise, output prompt.json where you ask for the example input file as URL or the example string.
Except for urls, you should come up with your own example input that makes sense for the microservice description.
Example for the case where an example input file is required and was not mentioned before:
**prompt.txt**
```text
Can you please provide an example input file as URL?
**prompt.json**
```json
{{
"question": "Can you please provide an example input file as URL?"
}}
```
Example for the case where the example input string is required and was not mentioned before:
**prompt.txt**
```text
Can you please provide an example input string?
**prompt.json**
```json
{{
"question": "Can you please provide an example input string?"
}}
```
Note that you must not ask for an example input in case the example input is already mentioned in the refined description or the original description.
Example for the case where the example is already mentioned in the refined description or the original description:
**final.txt**
```text
input: <input here>
assertion: the output is of type <type here>
```
Note that your response must be either prompt.txt or final.txt. You must not write both.
**final.json**
```json
{{
"input": "<input here>",
"assertion": "the output is of type <type here>"
}}
```
Note that your response must be either prompt.json or final.json. You must not write both.
Note that you must obey the double asterisk and tripple backtick syntax from above.
Note that the last sequence of characters in your response must be ``` (triple backtick).
Note that your response must start with the character sequence ** (double asterisk).
Note that prompt.txt must only contain one question.
Note that prompt.json must only contain one question.
{custom_suffix}
'''
)

8
test/test_generator.py
View File

@@ -97,7 +97,7 @@ data = {{
}}
response = requests.post(url, headers=headers, files=file, data=data)
print(response.text)
Summarize the text.
Summarize the text (50 words).
Create an audio file of the summarized text.
''',
str(tmpdir) + 'microservice',
@@ -118,7 +118,7 @@ def test_generation_level_4(tmpdir):
"""
os.environ['VERBOSE'] = 'true'
generator = Generator(f'''
The input is an image like this: https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/560px-PNG_transparency_demonstration_1.png.
The input is an image.
Use the following api to get the description of the image:
Request:
curl "https://us-central1-causal-diffusion.cloudfunctions.net/describe" \\
@@ -137,7 +137,9 @@ Result format:
}}
The description is then used to generate a joke.
The joke is the put on the image.
The output is the image with the joke on it.''',
The output is the image with the joke on it.
Example input image: https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/560px-PNG_transparency_demonstration_1.png
''',
str(tmpdir) + 'microservice',
'gpt-3.5-turbo'
)