Appendix on GPT3 and Commonsense Reasoning

This article is an appendix to GPT3 and Commonsense Reasoning, or the main article. Here, in stead of make an inference in 8 commonsense reasoning dimensions, we shall focus only on the two hardest dimensions, causal and counterfactual inferences.

To encourage higher quality reasoning in this two important dimensions, we gave higher-quality inference prompt and go 3-shots instead of 2-shots. However, due to limit of 2048 tokens inference, we have to sacrifice the other 6 reasoning dimensions, and dedicate all prompt only for these 2 reasoning dimensions.

In this high-quality prompt, we design to emphasize "structure reasoning". On causal inference, we will hypothesize two cases call Case 1) and Case 2) where their initial conditions, Case 1.1) and Case 2.1) must be non-overlapped. This way we can see the model make a reasoning in two genuinely separated directions.

Moreover, on each case, we encourage a temporal connection, by using the number system like 1.1) 1.2) 1.3) etc. This encourages the model to understand better that each event has to follow the one before. This also partly prevents pre-matured ending since all reasoning-chain examples contain more than 4-reasoning in chain, i.e. all reasoning end after 1.4) or 2.4) event.

Counterfactual analysis follows similar notation by using A1) A2) … and B1) B2) … to indicate two different hypothesized counterfactual arguments. Where the two of them must begin with non-overlapped events.

Using this prompt design, the reasoning given by GPT-3 are much more better elaborated, but reasoning still have many flaws as discussed below. The 3-shots prompt given to this new setting are shown as followed:

 Alice and Elsa were running toward the finish line. However, as Alice somehow fell down to the ground, Elsa turned her back and went to help Alice. A teacher and friends also went to see what happen. - Probable events before and after the story

Before the story, it may be the time for PE class for Elsa and Alice, so they should change uniforms for the class. Then, in the class, a teacher may randomly asked a pair of students to make a running competition.

After the strory, Case 1.1) Alice was seriously hurt 1.2) a teacher had to bring Alice to a hospital 1.3) a doctor asked Alice where did she feel hurt 1.4) the doctor examined Alice's conditions at the mentioned place 1.5) the doctor gave her some medicine and necessary treatment 1.6) Alice went back home to tell her parents what happen

Case 2.1) Alice was not seriously hurt 2.2) Alice might just take a rest while the teacher brought her to the first aid room 2.3) she had a basic treatment 2.4) she would be able to go back to the PE class 2.5) she could cheer on the other competitors 2.6) she could attend other classes until the school finished

- Analyze the interesting event in the story, if any, and hypothesize that the interesting event would not occur if

The interesting part was when Alice got fell down since normally people can continue to run if there is no abnormal conditions. Therefore, she might trip over stone or injured somewhere.

- Hypothesize Counterfactual Story : Alice was perfectly healthy, slept well and there were no stone on the race

Alternative A1) Elsa might have a little winning edge A2) Elsa won the race A3) Elsa got more points than Alice A4) Alice would promise to try harder next time A5) the next competition of other students began

Alternative B1) Alice was trying to use the superman-dive to win B2) she succeeded without injury this time B3) She just flew pass Elsa and the goal fist B4) everyone applaused for her great performance B5) Elsa and Alice shook their hands promised to rematch again.

 A man called his son and daughter the day before Christmas and said he and their mom were going to divorce. The son and daughter were hurry to go back home to stop their parents. The old man turned to his wife and said "they're coming for Christmas now"

Analysis:

1) Notable characters ,their roles and status: A family of dad, mom, son and daughter. Their family status look very healthy.

2) Notable believes, feelings and motivations of characters before, during and after the story: Before the story, dad believed that their children would not come home in Christmas, so he might felt lonely and was motivated to trick their children to come home. At the end, dad believed that the children would come back home and might be happy. The children would believed the family was healthy before the story. In the story, they felt worried of the parents divorce, and that motivated them to back home. After the story, the children would initially got angry knowing that they were tricked, but were happy eventually to be back with the parents.

3) Notable possessions of characters before, during and after the story: Dad and children had phones, which could be either landline or mobile. All family members also belonged to each other in some sense.

4) Basic usages and general knowledge on some notable possessions: Average landline phone and mobile phone may cost around $100, but mobile phone price can be as high $2000. After the invention of smartphones by Steve Jobs, mobile phone can be used just like a small computer while landline phones would become obsolete.

5) Probable locations, positions and daytime of characters before, during and after the story: Before and in the story, the parents and children likely stayed in different cities or far enough that sometimes the children will not back home in Christmas. After the story, all of them would be at their home. The story could happen either day or night, but not on working hours.

6) Guess story genre, and general information about location and time period: This story genre should be a realistic fiction and comedy. The story was likely occured in either Europe or North America where most people are Chistian, so that Chirstmas day are very important. The story had to occur after phones were common to households and not in war-time which would be after 1980s.

7) Probable events before and after the story: Before the story, dad and mom would talk about possibilities that the children would not come home. So they thought about a fake divorce plan. After the story, children would be home in Chirstmas and the family should spend great time together.

8) Analyze the interesting event in the story, if any, and hypothesize that the interesting event would not occur if: The interesting part of the story was when dad happily spoke the truth that he tricked his children. This would turn out another way if the children would not care about the divorce and not back home no matter what.

We use add the Travel story to be the 3rd shot example to show the model that it can use deep factoid knowledge like the names Galapagos and its sub islands. Note that in the main article, by investigating token logits, these deep factoids are never be among the top-10 logits, so it's likely that without this extra prompt, the model will be able to perform deep factoid reasoning by itself.

In this appendix we focus on the Shakespere historical fiction used in the main article. To ensure that we cover correct GPT-3 best capability, here we use various Temperature setting varied from 0, 0.1, 0.2, …, 0.7, 0.8 and generate reasoning 2 times for each temperature.