Skip to main content

Append vs Prepend Mode

Choose Best uses the model's likelihood to classify a query given a number of options. When using Choose Best, you need to specify a mode: PREPEND_OPTION or APPEND_OPTION. As the names imply, the mode specifies where the options will be concatenated to the query. This distinction needs to be made because our model predicts from left to right so the ordering of the query/option changes how we compute the likelihoods.

Example:

  • Query: I like this vacuum. It works great!
  • Option 1: This is a positive review.
  • Option 2: This is not a positive review.

Append Mode#

In APPEND_OPTION mode, the model will see:

  • Option 1: I like this vacuum. It works great! This is a positive review.
  • Option 2: I like this vacuum. It works great! This is not a positive review.

and the score of the options will be based on the log-likelihood of the query given the each option: logโกp(queryโˆฃoption)\log p(\textrm{query}|\textrm{option}). The token log-likelihoods on the option part of the string will be used to compute the score:

" This": -4.693203,
" is": -1.0524566,
" a": -1.651489,
" positive": -7.2853556,
" review": -3.1058905,
".": -1.3287007

Prepend Mode#

In PREPEND_OPTION mode, the model will see:

  • Option 1: This is a positive review. I like this vacuum. It works great!
  • Option 2: This is not a positive review. I like this vacuum. It works great!

and the score of the options will be based on the log-likelihood of each option given the query: logโกp(optionโˆฃquery)\log p(\textrm{option}|\textrm{query}). The token log-likelihoods on query part of the string will be used to compute the score:

"I": -1.5097405,
" like": -3.9063432,
" this": -2.666364,
" vac": -15.177841,
"uum": -1.2795182,
".": -1.5351871,
" It": -1.795983,
" works": -3.9086056,
" great": -2.7300918,
"!": -2.6350093

Recommendations#

Often the decision between append and prepend mode will be determined by the type of task you're tackling. However, in some situtations, you may be able to reformulate the inputs so that both append and prepend mode are possible. In such cases, which one should you use?

A general rule of thumb is to predict the shorter sequence given the longer sequence. So if your query is shorter on average than your options, then PREPEND_OPTION mode may be the right choice. If the query is on average longer than the options, APPEND_OPTION might be the right choice. Remember that these are only rules of thumb, and the best setup will depend on your specific task. The only way to find out which is best is try them out and see what works best!

Comparing Options#

Once the log-likelihoods of the tokens have been computed, how do we select the most likely option? For PREPEND_OPTION mode, we compare โˆ‘ilogโกp(queryiโˆฃoption)\sum_i \log p(\textrm{query}_i|\textrm{option}) for each option, where the sum is over the tokens of the query. In contrast, for APPEND_OPTION mode, it's less straightforward because the lengths of the options may differ, resulting in prediction biases based on option length. For instance, in the example given above, "This is a positive review." is 6 tokens whereas "This is not a positive review." is 7 tokens.

In general, there are three ways of comparing options in APPEND_OPTION mode: sum of token log-likelihoods, average of token log-likelihoods, and dividing the sum by the unconditional likelihood.

Sum#

โˆ‘ilogโกp(optioniโˆฃquery)\sum_i \log p(\textrm{option}_i|\textrm{query})
where the sum is over the tokens of the option. This score will be biased towards shorter options.

Average#

โˆ‘ilogโกp(optioniโˆฃquery)N\sum_i \log \frac{p(\textrm{option}_i|\textrm{query})}{N}
where N is the number of tokens in the option sequence. This score will be biased towards longer options.

Dividing by the unconditional likelihood#

โˆ‘ilogโกp(optioniโˆฃquery)p(optioni)\sum_i \log \frac{p(\textrm{option}_i|\textrm{query})}{p(\textrm{option}_i)}

where the unconditional likelihood p(optioni)p(\textrm{option}_i) can be computed using the Likelihood endpoint.

By default, the score provided by Choose Best is the average. If the length of the options are all equal, then the ranking given by the sum and average will be the same. However, when the lengths are different, we recommend trying out different scores to see what works best for you!