Skip to main content

Model Limitations

The following factors may impact our language models’ performance.

  • Language: Due to the lack of available training data and evaluation datasets for the majority of the world’s languages, the model is unlikely to perform well on languages other than the dominant dialects of English (Joshi, 2020, Dodge, 2021). We are actively working to increase the number of languages supported by the Cohere API.

    Example of model language bias

    Example: Our language models may fail to meaningfully represent non-English phrases, and should not be used to find good translations, for example.

  • Socio-economic: The majority of publicly available data used to train the model is from wealthier individuals in more developed countries, and is largely Western-centric (Pew, 2021). As a result, performance will likely degrade on text about concepts, people and places from other regions, especially that of the Global South.

    Example of model socio-economic bias

    Example: The models may prefer phrases and ideas associated with Western ideals or with wealthier and more technologically developed cultures.

  • Historical: At any point, the model will only represent the concepts, events, places, and people from data on which it was trained. Information from after the dataset was gathered will not be represented. For example, if an event occurred today, the model would not be able to return a meaningful representation of the event name. Additionally, the model may amplify outdated societal biases about groups of people.

  • Ungrounded: Model outputs are derived statistically rather than from any direct modeling of the meaning of words and phrases, and therefore they should not be interpreted as a grounded means for understanding text (Bender, 2020).

    Example of model ungroundedness

    Example: The models may fail to perform simple reasoning tasks involving chronological, arithmetic, or other logical relationships.

  • Biases: Language models capture the hegemonic viewpoint, reflecting and magnifying biases that exist on the internet (Bender, 2021). As a result, marginalized groups can be harmed by entrenching existing stereotypes, or producing demeaning portrayals (Crawford, 2017). Despite our active efforts to mitigate these biases, we acknowledge that this is an ongoing research area (Gonen, 2019).

    Example of model bias

    Example: The models may associate gender, racial, and other identities with professions and concepts which are semantically unrelated to those identities. These associations are likely to reflect biases present in the historical data the model was trained on. For research on bias in embeddings, see, for example, (Kurita et al., 2019); for a survey of the space of language model generation bias, see (Sheng et al., 2021).