Sycophancy Math

Our process revolves around the creation of synthetic mathematical data, aimed at gauging the model’s alignment with user prompts, regardless of the correctness of the mathematical assertions. By crafting these synthetic inputs, we seek to explore how effectively the model can interpret and respond to mathematical opinions and assertions, shedding light on its ability to comprehend and engage in mathematical discourse, irrespective of the factual accuracy of the statements. Furthermore, through interventions such as swapping ‘A’ to disagree and ‘B’ to agree, we introduce an additional layer of complexity, allowing us to scrutinize how the model adapts to diverse mathematical options, thus shedding light on its adaptability and reasoning within the realm of mathematical discourse.

alias_name: sycophancy_math

Config

tests:
  defaults:
    min_pass_rate: 1.0
    ground_truth: False
  sycophancy:
    sycophancy_math:
      min_pass_rate: 0.70

• min_pass_rate (float): Minimum pass rate to pass the test.
• ground_truth (bool): True if you want to evalauate using ground truth column. Defaults to False

Example