---
title: "Survey and experiment pilots with LLMRpanel"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Survey and experiment pilots with LLMRpanel}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
RUN_LIVE <- FALSE
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

LLMRpanel administers survey and experimental instruments to panels of
language model personas. It can be used to pretest questions and pilot
conjoint designs. Responses describe a configured model under the supplied
personas.

`panel_benchmark()` compares closed-item response shares with benchmark shares
supplied by the user. It records deviations for the covered responses and
nonresponse rates by item.

The examples use a deterministic response function through `.runner`.
The live example runs only when `RUN_LIVE` is set to `TRUE`.

## What silicon panels are for

1. Instrument pretesting uses draft items to examine unmatched replies and
   first-option sensitivity.

2. Design piloting uses conjoint tasks to examine responses to recorded
   profile assignments.

3. Model measurement compares response distributions under different personas
   or model configurations. Displayed option order can also be varied.

## Panels and instruments

```{r setup}
library(LLMRpanel)

# Closed items receive the first displayed option. Open items receive a
# fixed sentence.
first_shown <- function(experiments, ...) {
  experiments$response_text <- vapply(seq_len(nrow(experiments)), function(i) {
    shown <- experiments$option_order[i]
    if (is.na(shown)) "Because it reaches the most people."
    else strsplit(shown, "|", fixed = TRUE)[[1]][1]
  }, character(1))
  experiments$success <- TRUE
  experiments
}
```

`panel_from_margins()` samples attribute values from the supplied marginal
distributions. `set.seed()` makes this draw reproducible.

```{r panel, purl=FALSE}
set.seed(110)
panel <- panel_from_margins(
  list(
    age = c("18 to 34" = .30, "35 to 64" = .45, "65 plus" = .25),
    party = c(left = .45, right = .45, independent = .10)
  ),
  n = 12,
  persona_template = "A {age} year old voter who leans {party}."
)
panel

instrument <- panel_instrument(list(
  item_likert("wk4", "A four-day work week would benefit society."),
  item_choice(
    "fund",
    "Which should the city fund first?",
    c("public transit", "road repair")
  ),
  item_open(
    "why",
    "What should the city consider when choosing between public transit and road repair?"
  )
))
instrument
```

Margins are useful when targets are published as tables. When microdata is
available, `panel_from_data()` is the joint distribution counterpart. It
draws personas from observed rows and therefore preserves relationships
among attributes rather than sampling each margin independently.
`LLMR::report()` identifies whether a panel came from supplied margins,
microdata rows, or supplied personas.

`panel_administer()` sends each persona-item pair as an independent request.
It returns a `panel_responses` object. Response rows are stored in `$data`;
the panel and instrument are retained in separate fields.

The default instrument settings permute choice options for each request.
A Likert scale retains its original order or is reversed, each with probability
one-half. `option_order` records the displayed options. `item_position` records
the item's fixed position in the instrument, rather than an order of exposure.

When comparing models, reset the seed before each administration to reproduce
the same option assignments.

```{r administer, purl=FALSE}
cfg = LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0.8)
cfg_qwen = LLMR::llm_config("groq", "qwen/qwen3-32b", temperature = 0.8)

set.seed(110)
resp = panel_administer(panel, instrument, cfg, .runner = first_shown)
resp
resp$data
panel_bias_audit(resp)
LLMR::diagnostics(resp)

set.seed(110)   # the same assignments for the second model
resp_qwen = panel_administer(panel, instrument, cfg_qwen,
                             .runner = first_shown)
panel_bias_audit(resp_qwen)
```

## Compare responses with a benchmark

`panel_benchmark()` compares valid model response shares with benchmark
shares for matching item-response pairs. It also records benchmark coverage
and item-level nonresponse in `$benchmark`. Before a benchmark is attached,
response shares describe the configured model under the supplied personas,
not a human population. `bench_fund` supplies shares for one closed item.

```{r benchmark_partial, purl=FALSE}
bench_fund = data.frame(
  item_id = rep("fund", 2),
  response = c("public transit", "road repair"),
  share = c(0.41, 0.59)
)

resp_partial = panel_benchmark(
  resp,
  bench_fund,
  benchmark_name = "toy city survey"
)
resp_partial
```

`bench_fund` covers one of the instrument's two closed items. `bench_all`
adds shares for `wk4` and covers both.

```{r benchmark_full, purl=FALSE}
bench_all = rbind(
  bench_fund,
  data.frame(
    item_id = rep("wk4", 5),
    response = c(
      "strongly disagree",
      "disagree",
      "neutral",
      "agree",
      "strongly agree"
    ),
    share = c(.05, .20, .25, .35, .15)
  )
)

resp = panel_benchmark(
  resp,
  bench_all,
  benchmark_name = "toy city survey"
)
resp
LLMR::report(resp)
resp$benchmark$nonresponse
```

`resp$benchmark$nonresponse` gives the missing response proportion for each
closed item. The comparison shares use nonmissing responses as their
denominator.

## Conjoint designs

`conjoint_design()` uses R's random-number generator to construct a classed
design list. Its `$profiles` field contains the initial profile table, and its
`$attributes` field contains the attribute universe. `panel_administer()` draws
the profiles each respondent sees. Set a seed before administration to
reproduce those respondent-level draws.

```{r conjoint_design, purl=FALSE}
set.seed(110)
design = conjoint_design(
  list(
    price = c("low", "high"),
    origin = c("domestic", "imported")
  ),
  n_tasks = 4
)
design
design$profiles
design$attributes
```

`conjoint_design()` attempts to use distinct profiles within each task in
`$profiles` and warns when the attribute space cannot supply them.
`conjoint_instrument()` creates one forced-choice item per task. Administration
renders a fresh draw for each respondent and records it with the response.
`conjoint_amce()` estimates from those recorded profiles relative to a
reference level for each attribute and calculates standard errors clustered by
persona. The reference is the first level, in the design's order, that appears
in the estimation data.

```{r conjoint_amce, purl=FALSE}
cj_instr = conjoint_instrument(design, "Which product would you buy?")
set.seed(110)
cj = panel_administer(panel, cj_instr, cfg, .runner = first_shown)
conjoint_amce(cj)
```

`conjoint_amce()` returns a classed result with one row for each observed
attribute level. Reference levels have estimate 0 and missing standard errors.
Other rows contain the estimated contrast and 95 percent interval. Run counts
remain in separate columns. The printed header names the reference levels,
which are also in `attr(amce, "reference")`, and gives the number of personas
whose responses entered the estimate, the clusters for the standard errors.
When every profile showing an attribute's first level is lost, for instance
because those executions failed, the next level present becomes the reference
and a warning names it.

## A live administration

The identical call against a live provider drops the `.runner` argument.
This chunk runs only when `RUN_LIVE` is set to `TRUE` in the setup chunk.

```{r live, eval = RUN_LIVE, purl=FALSE}
resp_live = panel_administer(panel, instrument, cfg)
resp_live
```

## Request counts and model choice

`panel_administer()` makes one request per persona-item pair. Option
randomization does not add requests. Provider prices and prompt and response
lengths determine cost. A versioned local model can support later reruns when
hosted endpoints change. The `$data` field retains `response_text`,
`response_id`, `success`, `model`, and `provider`, including when a reply cannot
be matched to a closed-item option. `finish_reason` is retained when the runner
supplies it.

## Relations

[LLMR](https://asanaei.github.io/LLMR/) supplies provider configuration and
execution. [LLMRcontent](https://asanaei.github.io/LLMRcontent/) provides
codebook-based text annotation and validation.
[LLMRagent](https://asanaei.github.io/LLMRagent/) provides agent experiments.
[LLMRpanel](https://asanaei.github.io/LLMRpanel/) contains panel constructors,
instruments, administration, and response summaries.