Part 1
The challenge begins. The first puzzle was quite easy to solve, but posed some challenges to write an efficient prompt.
Reminder: my challenge is to attempt to solve the problem without writing a single line of code. Ideally, just writing an initial prompt and get the whole code in one go. You can see the full source code (and the respective prompts) in my repository at https://github.com/nunodonato/advent-of-code-2021
Although the problem was fairly simple, Copilot struggled with a very specific part of it.
First, I was impressed how easy it was to get it to read from a file which name I didn’t specify directly. I wanted each day to have it’s own input in an “inputs” folder, and have the filenames match. This was easy to accomplish and the solution presented was spot on!
$input = file_get_contents(__DIR__ . '/inputs/' . basename(__FILE__, '.php') . '.txt');
The next part involved a lot of trial and error with the right prompt phrasing. Interestingly the problems with the generated code was always EITHER:
- looping the array in such a way the it would go out of bounds (trying to read the next position when already in the last, for example) – although this was ok as PHP doesn’t crash, I didn’t accept the solution
- not incrementing the counter for every find, but instead adding the value of the line to the counter
I had to tinker with this particular line of the prompt until I managed to get it right. And even that was not perfect as the first solution proposed was incorrect. The second, however, was flawless
for ($i = 0; $i < count($input) - 1; $i++) {
if ($input[$i] < $input[$i + 1]) {
$counter++;
}
}
Codex vs Copilot
After I got the problem solved and the solution accepted, I copy&pasted my prompt into OpenAI’s playground using the davinci-codex model.
The solution presented was quite different from what I got, and it was wrong.
I tried a few more, while changing the temperature value, and got a very similar solution(but still not correct) at 0.7
It’s clear that Copilot has some clear advantages over using Codex, as it can present us several solutions at once (and it’s free). I do wonder what are the parameters that it uses! Hopefully will get more insights into it during the challenge.
Part 2
Part two brought a more complex problem. My first attempt was over complicated and copilot was throwing a bunch of useless code.
I re-framed the problem in my mind and decided to approach it in a different way. (important reminder: AI will struggle to solve complex coding problems if you don’t know the steps to solve it yourself)
After simplifying and attacking it from a different angle, I was very close to a solution. Copilot was struggling again with the iteration aspect of it, almost always going out of bounds. I tried different wording but they either made it worse or seemed to be ignored completely.
Finally I tried to write something more verbose and explicit:
// iterate the array until reaching the last-3 element
// store in counter A the sum of three values from the current position
// store in counter B the sum of three values from the next position
// if counter B is greater than counter A, increment the total by one
That gave me a clean and simple solution which I accepted and passed the second part of today’s challenge.
Codex vs Copilot
Using temperature at 0.7 and the exact same prompt, Codex gave me exactly the same solution as copilot did.