How we report fieldwork and response rates
A response rate on its own can hide who we never reached. For each study we show the whole path from the people we sampled to the interviews we kept.
Why we publish this
We are a young company, and we would rather you check our work than take it on trust. Every study we run comes with a fieldwork report that lets a client see how many people we sampled, how many we managed to reach, how many answered, and what happened to everyone who did not. The report is built from our own records as fieldwork runs, so it changes day by day until the study closes.
What each report contains
The report starts with a funnel: sampled, invited, delivered, opened or answered, started, completed and valid. It then breaks the same figures down by channel, by region and by day. After that it gives the final outcome of every sampled case, the standard response, cooperation, refusal and contact rates, the rules we followed when trying to reach people (how many attempts, how far apart, within which window), and a short note on how to read it all. Clients see totals and groups. Admins on our side see the same report with every group shown.
Each report also links to a contact log, a CSV file with one row for every sampled case. The row holds an anonymous case number, the channel, the time of each attempt, the delivery status and the final disposition. It has no names, phone numbers or email addresses, and the case numbers are made with a secret key we keep, so they cannot be traced back to a person. We record every download of the log.
How we know a message arrived
Most of what separates a weak response rate from a strong one is whether the invitation reached the person. We keep a status history for every invitation: queued, sent, delivered, opened or read, or failed with the reason. What we can know depends on the channel.
- WhatsApp. Meta tells us when a message is sent, delivered and read, and when it fails. We record each of these.
- Email. Our email provider tells us when a message is delivered, bounces, is marked as spam, or is opened. Open tracking depends on the reader's mail program, so an unopened email is not proof that nobody read it.
- SMS. The mobile network sends a delivery report through our SMS provider. Some networks are slow to send these, and a few never do.
- Telegram. Telegram sends no delivery reports. If Telegram accepts the message, we count it as delivered and mark the basis as acceptance by Telegram. If the person has blocked the bot or closed their account, Telegram refuses it and we record that.
- Phone. The dialer logs every call attempt and the interviewer records how each call ended. We import those records as they are and change nothing in them.
A person opening the survey link also counts as opened, whichever channel the invitation came by. Automatic link previews made by messaging apps and security scanners are ignored, because they are not the person.
AAPOR disposition codes
Every sampled case ends in exactly one final disposition from the American Association for Public Opinion Research (AAPOR) Standard Definitions. When a case has several outcomes, for example three calls to the same number, the most informative one is kept: an interview first, then a known ineligible, then what an eligible person did, then the unknowns. We use the codes below.
| Code | Outcome | Group | What it means |
|---|---|---|---|
| 1.1 | Complete interview | Interview | A valid, complete interview. |
| 1.2 | Partial interview | Partial | Enough of the questionnaire was answered to be usable, but not all of it. |
| 2.11 | Refusal | Refusal and break-off | The person said no, or asked us to stop contacting them. |
| 2.12 | Break-off | Refusal and break-off | The person started and stopped before the end. |
| 2.20 | Non-contact | Non-contact | An eligible person we could not reach, or who never opened the invitation. |
| 2.33 | Language problem | Other eligible non-interview | We reached the person but could not interview them in a language they speak. |
| 2.35 | Other eligible non-interview | Other eligible non-interview | We reached the person, but no usable interview came out of it. |
| 3.11 | Not attempted or worked | Unknown if reached | Selected, but never tried (no working channel, or not yet reached in the field period). |
| 3.12 | Always busy | Unknown if reached | Every call attempt found the line busy. |
| 3.13 | No answer | Unknown if reached | Every call attempt rang out. |
| 3.14 | Answering machine, unknown contact | Unknown if reached | Only a voicemail was reached, so we do not know who holds the number. |
| 3.17 | No screener completed | Unknown if reached | Someone answered but we never established whether they were eligible. |
| 3.90 | Unknown, other | Unknown, other | The invitation went out, but nothing confirms it arrived and nothing came back. |
| 4.31 | Non-working number | Not eligible | The number is not in service. |
| 4.70 | No eligible respondent | Not eligible | The person reached was not eligible (for example a minor). |
| 4.80 | Quota filled | Not eligible | The person was eligible, but their quota cell was already full. |
| 4.90 | Not eligible, other | Not eligible | A wrong number, or another reason the case was never in the target population. |
How our records map to those codes
Phone interviews
The dialer records one outcome for each call. We map them like this.
| What the interviewer recorded | AAPOR code | Outcome |
|---|---|---|
| complete | 1.1 | Complete interview |
| refusal | 2.11 | Refusal |
| not interested | 2.11 | Refusal |
| do not call | 2.11 | Refusal |
| dnd request | 2.11 | Refusal |
| partial breakoff | 2.12 | Break-off |
| callback scheduled | 2.20 | Non-contact |
| language barrier | 2.33 | Language problem |
| connected decision maker | 2.35 | Other eligible non-interview |
| busy | 3.12 | Always busy |
| no answer | 3.13 | No answer |
| voicemail left | 3.14 | Answering machine, unknown contact |
| connected gatekeeper | 3.17 | No screener completed |
| not in service | 4.31 | Non-working number |
| ineligible under 18 | 4.70 | No eligible respondent |
| quota full | 4.80 | Quota filled |
| wrong number | 4.90 | Not eligible, other |
A number that was dialled but has no recorded outcome counts as no answer. A number that was released to an interviewer but never dialled counts as not attempted. The numbers are generated by random digit dialling, so most of them never lead to a person and many stay in the unknown groups. The rates below are built to handle that.
Online invitations to the panel
People we invite online are existing panel members, verified adults who agreed to take part, so we know they are eligible. What we do not always know is whether the invitation reached them.
- A valid, complete interview is code 1.1. A completed interview removed by our quality checks is code 2.35.
- Someone who started and did not finish is a break-off, code 2.12. If we kept their partial answers and they covered enough of the questionnaire, it is a partial interview, code 1.2.
- Someone who replied STOP is a refusal, code 2.11.
- Someone who opened the invitation but did not take part is code 2.35, because we reached them and got no usable interview.
- An invitation that was delivered but never opened, or that failed to deliver, is a non-contact, code 2.20.
- An invitation that was sent but has no delivery report and no reply is unknown, code 3.90. We cannot tell whether it arrived.
- A panel member selected for the study but who had no working channel is code 3.11, not attempted.
Because this is a panel and not a random sample of the country, a response rate here describes how much of the invited panel took part. It does not describe how well the panel represents the population. We say so on every report.
The rates
We calculate each rate with the formulas in the AAPOR Standard Definitions. I is complete interviews, P partial interviews, R refusals and break-offs, NC non-contacts, O other eligible non-interviews, UH cases of unknown contact, and UO other unknowns.
| Rate | Formula | In plain words |
|---|---|---|
| RR1 | I / ((I+P) + (R+NC+O) + (UH+UO)) | Complete interviews out of every case that could have been eligible. The strictest response rate. |
| RR2 | (I+P) / ((I+P) + (R+NC+O) + (UH+UO)) | Like RR1, but partial interviews count too. |
| RR3 | I / ((I+P) + (R+NC+O) + e(UH+UO)) | Like RR1, but only the estimated eligible share of the unknown cases is counted. |
| RR4 | (I+P) / ((I+P) + (R+NC+O) + e(UH+UO)) | Like RR3, with partial interviews counted. |
| RR5 | I / ((I+P) + (R+NC+O)) | Complete interviews out of cases known to be eligible. Assumes every unknown case was not eligible, so it is the most generous. |
| RR6 | (I+P) / ((I+P) + (R+NC+O)) | Like RR5, with partial interviews counted. |
| COOP1 | I / ((I+P) + R + O) | Of the eligible people we actually reached, how many gave a complete interview. |
| COOP2 | (I+P) / ((I+P) + R + O) | Like COOP1, with partial interviews counted. |
| COOP3 | I / ((I+P) + R) | Of the people who were asked, how many gave a complete interview. |
| COOP4 | (I+P) / ((I+P) + R) | Like COOP3, with partial interviews counted. |
| REF1 | R / ((I+P) + (R+NC+O) + (UH+UO)) | Refusals and break-offs out of every case that could have been eligible. |
| REF2 | R / ((I+P) + (R+NC+O) + e(UH+UO)) | Like REF1, using the estimated eligible share of the unknown cases. |
| REF3 | R / ((I+P) + (R+NC+O)) | Refusals and break-offs out of cases known to be eligible. |
| CON1 | ((I+P) + R + O) / ((I+P) + (R+NC+O) + (UH+UO)) | How often we reached a person, out of every case that could have been eligible. |
| CON2 | ((I+P) + R + O) / ((I+P) + (R+NC+O) + e(UH+UO)) | Like CON1, using the estimated eligible share of the unknown cases. |
| CON3 | ((I+P) + R + O) / ((I+P) + (R+NC+O)) | How often we reached a person, out of cases known to be eligible. |
The estimate for unknown eligibility, e
When we do not know whether a case was eligible, such as a number that always rang out, a response rate has to make an assumption. RR1, RR2, RR5 and RR6 make a fixed one. RR1 and RR2 treat every unknown case as eligible. RR5 and RR6 treat every unknown case as not eligible. The rates marked with e sit between those extremes. They count only a share of the unknown cases, and that share is e.
We work out e from the cases we do know about: the number of eligible cases divided by the number of eligible and not eligible cases together. The report shows the value it used. If we have no case with a known eligibility yet, e cannot be estimated and the rates that need it show as not available.
What the numbers cannot tell you
A high response rate does not make a survey accurate, and a low one does not make it wrong. It tells you how much of the sample the results rest on and how much we had to assume about the rest. Our sampling, weighting and margin of error are described on the main methods page. While fieldwork is open, the report shows the position so far, and cases still in play carry the outcome they have at that moment. The outcomes are final once the study closes.
Source
The American Association for Public Opinion Research. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 10th edition, 2023. Available at aapor.org/standards-and-ethics/standard-definitions.
If you spot a mistake in how we have applied it, write to corrections@pollsafrica.com and we will correct it.