Module 13: Data ethics, quality, and coverage bias

View study sheet (PDF) View SQL cheat sheet (PDF)

Teaching this module? A facilitator guide adds a preparation checklist, timing cues, discussion guidance, slides, and a printable PDF.

This session teaches students to read a real dataset critically rather than taking any figure it produces at face value. The central case study is a coverage bias built into how this dataset, like most transaction datasets assembled from many independent sources, actually gets built: the newest quarters are still being ingested at the time any query is run, so a naive comparison of recent-quarter counts against older ones will show a decline that looks like falling activity but is actually an artifact of incomplete ingestion, not a market signal. Students confirm this pattern themselves by counting rows per quarter and per vendor in the sandbox, then use what they find to develop a broader set of habits for responsible data use: disclosing what a dataset does not see, treating unsold lots correctly rather than discarding them, respecting point-in-time correctness, and stating a claim's limitations alongside the claim itself. The session's goal is not to teach students to distrust data, but to teach them to read it the way a careful analyst does.

Target course(s) and level

Data ethics, research methods, information literacy, or a data science course with an ethics or quality module. Suitable for undergraduates and graduate students across data science, business, and social science programs. No finance or programming background is assumed; the entire session is sandbox-based with no code.

Learning objectives

By the end of this session, students will be able to:

  1. Define coverage bias and explain, using the recency under-ingestion pattern in this dataset, how an ingestion artifact can be mistaken for a real trend if a dataset's construction process is not understood.
  2. Query row counts by quarter and by vendor in the sandbox to confirm the recency pattern directly, rather than accepting the caveat on faith.
  3. Explain why unsold lots must be retained and correctly labeled rather than dropped, and describe what would be distorted, such as sell-through, if they were excluded.
  4. Describe point-in-time correctness and explain why a query that ignores sale_date can produce a claim that misrepresents when a transaction actually happened.
  5. Distinguish a claim a dataset can support from one it cannot, and draft a disclosure statement appropriate to a specific claim.
  6. Identify at least three sources of coverage bias in this dataset beyond recency, such as uneven vendor participation or category-specific gaps, and explain how each could mislead an unqualified reader.

Prerequisites

No prior coursework in data science, statistics, or finance is required. Familiarity with the general idea of a database table (rows and columns) is helpful but not assumed; the sandbox orientation segment covers this from first principles.

Materials and access needed

  • Sandbox access at sandbox.altfndata.com, self-registered with a work or school email, auto-approved.
  • Projector or screen share for the instructor demo.
  • The coverage browser tab, used to look at how vendors and categories are represented before the demo.
  • A short handout defining coverage bias, point-in-time correctness, and disclosure, for reference during the small-group exercise, prepared by the instructor.

Session outline (90 minutes)

  • 0 to 15 min: Introduce the session's central question, how would you know if a pattern in the data is real or an artifact of how the data was collected, and preview the recency case study.
  • 15 to 25 min: Sandbox orientation. Confirm students can open a data table and locate sale_date, status, vendor, and usd_price_decimal in the data dictionary.
  • 25 to 45 min: Guided demo, count rows by quarter for a single category and observe the recency drop-off directly, then discuss why it reflects ingestion timing rather than a market decline.
  • 45 to 60 min: Guided demo, count rows by vendor for the same category and discuss uneven vendor coverage as a second, independent source of coverage bias.
  • 60 to 75 min: Small-group exercise, each group picks a different category, reproduces both counts, and drafts a one-paragraph disclosure statement they would attach to a claim built on that category's data.
  • 75 to 85 min: Class discussion, groups share their disclosure statements and the class critiques whether each one is specific enough to be useful to a reader.
  • 85 to 90 min: Wrap-up and homework assignment.

In-class demo (sandbox-first, no code)

  1. Open sandbox.altfndata.com, sign in, and select a data table, for example the fine art data, from the SQL editor dropdown.
  2. Open the coverage browser tab and look at how several vendors are represented in the vendor field, noting that participation is not evenly distributed across houses.
  3. Return to the SQL editor and run the first guided query, sold-lot counts by quarter, and have students read the last two or three bars or rows aloud. Ask what they notice about the most recent quarters compared to the ones before them.
  4. Explain the recency under-ingestion pattern directly: the newest quarters are still being ingested at any given point in time, so the drop-off in the most recent quarters is a data collection artifact, not evidence that fewer transactions occurred or that a market cooled. Emphasize that this is exactly the kind of pattern a careless reader could mistake for a real trend, and exactly why any time-bucketed claim in this course has been built on stable, longer-run figures rather than the newest quarters.
  5. Run the second guided query, sold-lot counts by vendor, and discuss how unevenly represented vendors are, and what that means for any claim based on "the market" as a whole rather than on the specific houses actually captured.
  6. Introduce point-in-time correctness using the status field: explain that an unsold lot is retained in the data and marked unsold, and ask what would happen to a sell-through calculation, or to any claim about how much of a category found a buyer, if unsold lots were silently dropped instead.
  7. Run the third guided query, comparing sold-lot counts against total offered-lot counts by quarter, and ask students whether the same recency caution applies to sell-through as it does to raw counts, and why.
  8. Close by asking students to draft, in one sentence, the disclosure they would attach if asked to report "how the fine art market performed last quarter" using only this dataset.

Datasets and queries used

Dataset: any production category table (documented fields: designer, model, item_title, sale_date, usd_price_decimal, sale_estimates_high_usd_price, status, vendor, stock_ticker). Examples below use all_fine_art_data; the same queries run unchanged against all_watches_data, all_handbags_data, all_jewels_gems_data, and the other all_*_data tables.

Query 1, sold-lot counts by quarter, used to observe the recency drop-off directly:

SELECT date_trunc('quarter', CAST(sale_date AS date)) AS quarter,
       COUNT(*) AS sold_lots
FROM all_fine_art_data
WHERE status = 'sold'
GROUP BY date_trunc('quarter', CAST(sale_date AS date))
ORDER BY quarter DESC
LIMIT 12;

Query 2, sold-lot counts by vendor, used to observe uneven vendor coverage:

SELECT vendor,
       COUNT(*) AS sold_lots
FROM all_fine_art_data
WHERE status = 'sold'
GROUP BY vendor
ORDER BY sold_lots DESC
LIMIT 25;

Query 3, sold versus offered lots by quarter, used to test whether the recency caution also applies to sell-through:

SELECT date_trunc('quarter', CAST(sale_date AS date)) AS quarter,
       COUNT(*) FILTER (WHERE status = 'sold') AS sold_lots,
       COUNT(*) AS offered_lots,
       COUNT(*) FILTER (WHERE status = 'sold') * 1.0 / COUNT(*) AS sell_through
FROM all_fine_art_data
GROUP BY date_trunc('quarter', CAST(sale_date AS date))
ORDER BY quarter DESC
LIMIT 12;

Discussion questions

  1. If the most recent two quarters in a chart show a steep drop-off, what evidence would distinguish an ingestion artifact from a genuine decline in market activity, and how would you gather that evidence using only this dataset?
  2. Why is it not enough to simply state "the newest data may be incomplete" in a footnote? What would a more specific and useful disclosure look like?
  3. If a house's lots are underrepresented in this dataset relative to its actual share of the market, what kinds of claims about "the market" would be most distorted by that gap, and which claims would be relatively unaffected?
  4. Why does treating an unsold lot as if it never existed change a sell-through calculation, and what other calculations in this dataset would be similarly distorted by dropping unsold rows?
  5. What does point-in-time correctness mean in the context of sale_date, and what claim would be wrong if it used a transaction's retrieval date instead of its actual sale date?
  6. A dataset can be technically accurate in every row and still support a misleading claim. How does that happen, and what is the analyst's responsibility once they understand it?
  7. How would you explain the recency caveat to a non-technical audience, such as a journalist or an investor, without either overstating the problem or hiding it?
  8. What is the difference between a dataset having a limitation and a dataset being untrustworthy, and why does that distinction matter for how you communicate findings?

Homework assignment

Each student selects one category table and one claim they might want to make about it, for example "which auction house sells the most in this category" or "how has demand for this category changed over time." The student runs the three guided queries (or their equivalents) for their chosen category, writes a one-page memo stating the claim, the evidence for it, and a disclosure section that specifically addresses recency under-ingestion, vendor coverage, and correct treatment of unsold lots as they apply to that claim. The memo must include the SQL queries used as an appendix and the exported result sets. Grading criteria: correct construction and reading of the three queries (25 percent), accuracy and specificity of the coverage bias analysis (35 percent), quality and honesty of the disclosure section (30 percent), and clarity of the memo (10 percent).

Going deeper

Key terms

  • Coverage bias: a systematic gap between what a dataset captures and what it claims or is assumed to represent, arising from how the data was collected rather than from the underlying reality.
  • Recency under-ingestion: the specific coverage bias in this dataset in which the newest quarters have fewer records than they eventually will, because ingestion of new transactions is ongoing.
  • Point-in-time correctness: the property that a record's date fields accurately reflect when the underlying event, here a sale, actually occurred, as distinct from when it was added to the dataset.
  • Unsold lot: a lot that was offered at auction but did not meet its reserve price and therefore did not sell, retained in the data and marked with a status of unsold rather than removed.
  • Sell-through: the share of offered lots that find a buyer, a figure that requires both sold and unsold lots to be present and correctly labeled to compute honestly.
  • Disclosure: an explicit statement, attached to a claim, of the limitations in the underlying data that a reader should weigh before accepting the claim.
  • Survivorship-style gap: a general category of coverage bias in which the data available for analysis systematically omits certain cases, distinct from but related to the recency gap studied in this session.
  • Vendor coverage: the degree to which different auction houses or marketplaces are represented in a dataset, which need not match their actual share of real-world transaction volume.

Common pitfalls

  • Reading a drop in the most recent one or two quarters as a market signal instead of checking whether it matches the known recency under-ingestion pattern.
  • Excluding unsold lots from a query in order to "clean up" the data, which silently changes the meaning of any sell-through or demand figure computed afterward.
  • Making a claim about "the market" when the data actually reflects a specific, unevenly weighted set of vendors.
  • Writing a disclosure that is vague enough to be true of any dataset, rather than one specific to the claim being made and the gaps that actually affect it.

Additional queries to explore

  • Row counts by category and by quarter together, to see whether the recency under-ingestion pattern is uniform across categories or more pronounced in some than others.
  • A comparison of vendor counts across two different categories for the same house, to see whether a house's coverage is consistent or uneven across the categories it participates in.

Extension activities

  • Have students find a published claim, in a news article or report, that uses auction or resale data, and write a short critique identifying what coverage biases the claim may not have accounted for.
  • Ask students to design a disclosure template that could be attached to any claim made from this dataset, then test it against two or three of the claims discussed in class to see if it holds up.

Connections to other modules

  • Module 8, Data visualization and storytelling, for how a chart can either surface or obscure a coverage bias depending on how it is built.
  • Module 10, Time series and market indices, for the deeper mechanics of why time-bucketed figures are especially sensitive to the recency pattern studied here.
  • Module 11, Auction theory and behavioral economics, for a case where the same recency caution applies to sell-through and pricing power figures used to test economic predictions.