DuckDB backend

Build and estimate on larger-than-memory data. The stack is created and (in Stata) never leaves DuckDB; regressions run from an exact cell compression, so point estimates and standard errors match the in-memory path exactly.

R duck_src(), duck_collect(), duck_write_parquet(), duck_disconnect(), duck = TRUE   ↔︎   Stata stacked duckconnect, stacked build, parquet()

Usage

# wrap larger-than-memory input (parquet/csv glob, a DBI connection, or a df)
src <- duck_src("panel/*.parquet")

# build / screen inside DuckDB
stack <- build_stack(src, "year", "state", "adopt_year",
                     kappa_pre = 3, kappa_post = 2, duck = TRUE)
kt    <- kappa_trade_offs(src, "year", "state", "adopt_year", duck = TRUE)

# estimate as usual; the stack_duck handle flows straight through
model <- stackreg(stack, "outcome", cluster_var = "state")

# materialize / persist / close
dt   <- duck_collect(stack)                 # -> data.table
path <- duck_write_parquet(stack, "out.parquet")
duck_disconnect(stack); duck_disconnect(src)

Helpers

Function Description
duck_src(x, table, duck_options) Wrap input: path(s)/glob, a DuckDB DBIConnection (+ table), or a data.frame. duck_options (e.g. list(memory_limit="4GB", threads=4)) apply to package-owned connections.
duck_collect(x, n) Materialize a stack_duck as a data.table (n caps rows).
duck_write_parquet(x, path) Write a stack_duck to parquet.
duck_disconnect(x) Close a package-owned connection (user connections left open; also closed on GC).
duck = TRUE Argument on build_stack() / kappa_trade_offs().

Needs the DBI and duckdb R packages; the duckdb package bundles the engine (no Java, no external install).

* one-time driver download (~75 MB, into PERSONAL), then open a session
stacked duckconnect, download
stacked duckconnect [, memory(str) threads(#) db(str)]

* build / screen against parquet or csv, never entering Stata memory
stacked build, parquet("panel/*.parquet") time(year) unit(state) ///
    adopt(adopt_year) kpre(3) kpost(2) [table(name)]
stacked kappa, parquet("panel/*.parquet") time(year) unit(state) adopt(adopt_year)

* estimate from the in-database stack
stacked reg  depvar, duck [cluster(name) ref(#)]
stacked levels depvar, duck

stacked duckdisconnect

Options

Option Description
download Fetch the self-contained DuckDB JDBC driver (one time).
memory(str), threads(#), db(str) DuckDB session settings.
parquet(str) Parquet/csv path or glob for build/kappa.
table(name) Name for the in-database stack table.
duck Run reg/levels against the in-database stack.

R ↔︎ Stata mapping

R Stata
duck_src(path) parquet() path/glob on build/kappa
duck_src(x, duck_options=list(...)) stacked duckconnect, memory() threads() db()
build_stack(..., duck = TRUE) stacked build, parquet(...)
kappa_trade_offs(..., duck = TRUE) stacked kappa, parquet(...)
stackreg(stack_duck, ...) stacked reg depvar, duck
stack_levels(stack_duck, ...) stacked levels depvar, duck
duck_write_parquet(x, path) jdbc exec "COPY (...) TO 'x.parquet' (FORMAT parquet)"
duck_disconnect(x) stacked duckdisconnect
duck_collect(x) (none; Stata keeps the stack in DuckDB)

Why it is exact (not approximate)

Once the regression is Q-weighted, fixed effects are redundant (paper Proposition 2), so the saturated specification involves only treatment status and event time and compresses to one cell per (cluster ×) treat × event_time. The compressed regression reproduces the full-data estimates and SEs exactly — sub-second even for stacks of 100M+ rows — and the on-disk stack duplicating controls across sub-experiments costs essentially nothing, because its compressed size does not grow with the duplication.

Both backends support model(att)/model(eventstudy) and bygroup. The Stata duck path also supports fe(interacted) via in-database FWL demeaning; current duck limits (clear errors): panel data only, and no covariates()/ absorb() in stacked reg, duck.

Example

library(stacked)
library(data.table)
data(signflip)

# write a tiny parquet file to stand in for a large one
pq <- tempfile(fileext = ".parquet")
con <- DBI::dbConnect(duckdb::duckdb())
duckdb::duckdb_register(con, "df", signflip)
DBI::dbExecute(con, sprintf("COPY (SELECT * FROM df) TO %s (FORMAT parquet)",
                            DBI::dbQuoteString(con, pq)))
[1] 72
DBI::dbDisconnect(con, shutdown = TRUE)

src   <- duck_src(pq)
stack <- build_stack(src, "year", "id", "adopt_year",
                     kappa_pre = 2, kappa_post = 1, duck = TRUE)
model <- stackreg(stack, "outcome", cluster_var = "id", model = "att")
attr(model, "avg_post_att")$estimate   # exact +2.4, as in memory
[1] 2.4
duck_disconnect(stack); duck_disconnect(src)
stacked duckconnect, download
stacked build, parquet("data/signflip.parquet") ///
    time(year) unit(id) adopt(adopt_year) kpre(2) kpost(1)
stacked reg outcome, duck cluster(id)
stacked duckdisconnect

See also