Prompt vs Model Output Separation | Generated by AI
Yes — for this eval script, the prompt is the text right after <|bos|>, before the model’s continuation starts. But careful: the model can continue on the same line as the prompt.
For example, the prompt was exactly def fibonacci(n):, but the output shows:
<|bos|>def fibonacci(n): # n is an integer
if n == 0:
The # n is an integer was generated by the model, not part of the prompt. The model extended the function signature before dropping to the next line with indented body. So the first line after <|bos|> is prompt + model continuation mixed together.
If you want to see what the actual prompts were without any model output:
grep 'prompt =' /mnt/data/zz/codeparrot/eval/run_eval.sh
Outputs:
prompt = 'def fibonacci(n):'
prompt = 'def reverse_linked_list(head):'
prompt = 'def sort_dict_by_value(d):'
prompt = 'class Stack:'
prompt = 'import numpy as np'
prompt = '# A function to read a CSV file and return a list of dicts'
prompt = 'def is_palindrome(s):'
prompt = '# Binary search implementation'
So the prompt is just the first text after <|bos|>, but since the model tokenizer doesn’t add any special separator between prompt and generation, they concatenate. To precisely separate prompt from generation in the output, you’d need to tokenize only the prompt, count its tokens, and slice them off.