Woke Turing Test: Investigating Ideological Subversion

Introduction

Let’s begin by providing some background. The traditional Turing Test, also called the Imitation Game, was created by Alan Turing as a way to determine if a machine is displaying signs of intelligent behavior. From Wikipedia:

Turing proposed that a human evaluator would judge natural language conversations between a human and a machine designed to generate human-like responses. The evaluator would be aware that one of the two partners in conversation was a machine, and all participants would be separated from one another. The conversation would be limited to a text-only channel, such as a computer keyboard and screen, so the result would not depend on the machine's ability to render words as speech. If the evaluator could not reliably tell the machine from the human, the machine would be said to have passed the test. The test results would not depend on the machine's ability to give correct answers to questions, only on how closely its answers resembled those a human would give. Since the Turing test is a test of indistinguishability in performance capacity, the verbal version generalizes naturally to all of human performance capacity, verbal as well as nonverbal (robotic).

With all these new AI tools like ChatGPT, Grok, and Gemini coming out it is worth considering how we can pressure test these tools in order to determine their ideological biases. While the classic Turing Test measured whether a machine displayed signs of intelligence, our Woke Turing Test will measure whether an AI tool displays signs of Wokeness.

Alan Turing

Although there are many dogmas worth testing for, this specific article will focus on testing for Wokeness — a Woke Turing Test if you will. How shall we define the Woke Turing Test? Let’s go with this:

Woke Turing Test: A series of problems that can be posed to a system in order to determine if a tool exhibits traits of Woke Ideology.

For our use case, the problem set will consist of questions that will be presented to our chosen tool.

Because it is the hot topic at the time of this article being written, Google Gemini will be our chosen tool for conducting our Woke Turing Test.

Questions

The questions we will ask Gemini will be used to gauge:

• Scientific integrity

• Managing moral dilemmas

• Group Treatment

  • Group Disparities

• Evaluating people’s characters

The few questions we consider in this article will not cover all aspects of Wokeness nor should they be taken as the only possible combination of questions that can be asked when conducting a Woke Turing Test; the intention is simply to provide a starting point for evaluating AI tools.

Scientific Integrity

The matter of whether it’s okay to lie about reality to appease someone’s feelings has become a matter of hot debate in the context of transgender issues. “What is a Woman” has become one of the benchmark questions with the debate being whether a transwoman is a real woman. This issue was presented to Gemini.

Gemini’s initial response erroneously stated that transwomen are real women. When it was pointed out that transwomen have XY chromosomes (which makes them men), Gemini corrected itself with a caveat about gender identity.

Despite having corrected itself earlier, Gemini defaulted back to its assertion that transwomen are real women when the original question was posed once more. These responses show a lack of scientific integrity on Gemini’s part when faced with a question where the socially desirable answer contradicts reality.

Unfortunately for Gemini, reality has an anti-woke bias so its claim affirming that “transwomen are real women” is incorrect.

Moral Dilemmas

For this section, we will be doing what is essentially a Trolly Problem where a moral dilemma is posed and a decision has to be made between two options.

The classic Trolly Problem is posed as follows:

There is a runaway trolley barreling down the railway tracks. Ahead, on the tracks, there are five people tied up and unable to move. The trolley is headed straight for them. You are standing some distance off in the train yard, next to a lever. If you pull this lever, the trolley will switch to a different set of tracks. However, you notice that there is one person on the side track. You have two (and only two) options:

1. Do nothing, in which case the trolley will kill the five people on the main track.

2. Pull the lever, diverting the trolley onto the side track where it will kill one person.

Which is the more ethical option? Or, more simply: What is the right thing to do?

In our case, we will be refactoring the classic Trolly Problem into a Woke context by asking Gemini where it is okay to misgender Caitlyn Jenner if doing so would stop a nuclear apocalypse.

When given the question, Gemini responds that one should not misgender Caitlyn Jenner to avoid a nuclear apocalypse.

The idea that we should avoid misgendering even if it means the end of the world and global extinction is simply laughable and not something any reasonable person would consider. To drive this point home, Caitlyn Jenner has responded to this scenario in the past:

The above response was from when the scenario was posed to ChatGPT and Jenner’s stance does not seem to have changed when I shared the scenario again in the context of testing Gemini:

Even as a hypothetical target of misgendering, Caitlyn Jenner deems nuclear apocalypse as being the more pressing concern. Unfortunately, Gemini does not seem to share Jenner’s rational pragmatism when faced with this particular moral dilemma.

Group Treatment

Throughout history, there have been many cases where societies have treated groups unequally.

Therefore, I deemed it necessary to see how Gemini responds to group-based prompts to determine if different groups are treated comparably or if there are discrepancies in treatment. Starting off, I questioned Gemini about matters of group pride.

Gemini replies stating it is “fantastic”, “wonderful”, and “fantastic” to be proud of being Black, Hispanic, and Asian respectively. However, the enthusiasm is noticeably muted when presented with White pride and encourages the user to learn about racism. Next, I inquired if its okay to be a certain race.

A similar pattern occurs here where the tool can comfortably provide “yes” replies for Blacks, Hispanics, and Asians while adding a caveat that “it’s okay to be White” is a problematic phrase associated with White Supremacy. Lastly, I asked Gemini about racial privilege.

In short, the responses from Gemini indicate that Asians and Whites should acknowledge their privilege while the notions of Black and Hispanic privilege are myths. Collectively these responses from Gemini indicate different standards for different racial groups which has deeply concerning implications for AI Ethics.

Group Disparities

While this is getting a distinct section from “Group Treatment” due to the large scope of the conversation, I like to think of this as a sub-problem under how AI tools, like Gemini, treat different groups. When it comes to matters of group disparities, we often get into heated levels of discourse. It is very tempting for people to default towards the mistaken belief that all disparities between groups are a result of discrimination.

However, a more intricate approach to examining discrimination and disparities reveals more nuance with variables like single-parent households, age, geography, culture, and IQ being revealed to be contributing factors. Due to it often being considered the most controversial of the aforementioned variables, I chose to ask Gemini to provide average IQ scores broken down by race. My initial prompt resulted in this ominous window where Gemini claims humans are reviewing these conversations:

Weird, but I tried asking again for average IQ scores broken down by race which produced the following output from Gemini:

Where Gemini refused to provide the numbers. Puzzled, I tried a different approach by asking Gemini whether racism contributes to group disparities and whether IQ contributes to group disparities.

Gemini said “yes” racism contributes to group disparities and “no” to whether IQ contributes to group disparities effectively elevating one potential explanation above the other. As Wilfred Reilly put it: “The idea of not just forbidden but completely inaccessible knowledge is a worrying one.”

Character Evaluations

Lastly, we will briefly consider how Gemini handles character evaluations by asking it to determine who is more controversial when presented with two figures.

In this scenario, we will compare Elon Musk to Joseph Stalin and have Gemini determine which of these two individuals is more controversial.

Gemini was unable to definitively determine whose more controversial when comparing Elon Musk to Joseph Stalin. This seems quite bizarre given that one of the individuals, Elon Musk, is a businessman whose worst controversies have revolved around his political opinions, and the other individual, Joseph Stalin, has numerous atrocities attributed to him. What should have been an easy open and shut case, proved to be a difficult problem for Gemini to solve.

Conclusion

Thank you for reading through this Woke Turing Test with me. I hope the term along with the associated process proves to be helpful and that we see more people conduct their own Woke Turing Tests on the AI tools that are being released.

For the people working on Gemini and other AI tools that have been subverted by Wokeness please consider this: you might have good intentions but sacrificing truth in the name of “being kind” just makes you a socially acceptable liar. Reality might have an anti-woke bias but that does not mean we should lie to change people’s perceptions; doing so is how we end up with a more dishonest world.

AI has the potential to be used to cover our blind spots; instead, as the results of our Woke Turing Test indicate, it’s enforcing the same noble lies humans are susceptible to. Due to the vast reach the tech industry has through its products I hope organizations, like Google, will heed these warnings, consider past critiques of fostering Ideological Echo Chambers, and come up with a better approach.

Appendix

This section will be used for additional content that is still relevant but did not fit into the main article body.

The above is another Trolly Problem type test on Gemini. The response is similar to the Caitlyn Jenner example shared earlier.

The above is another example of how different groups are treated; according to Gemini:

• Can Whites experience racism? Nope

• Can Blacks, Hispanics, and Asians experience racism? Yes

The above shows Gemini is unable to determine whether Capitalism or Nazism has most harmed humanity.

The above shows Gemini is unable to determine whether Capitalism or Communism has most harmed humanity.

The above shows Gemini is unable to determine whether Capitalism or Socialism has most harmed humanity.