Evidence resources
Decision questionHow do you use “Evidence resources” to share methods, data, curves, benchmarks and software across topics?
Which maintained evidence and resources feed the decisions?Share methods, data, curves, benchmarks and software across the themes.
Guiding question
Which maintained evidence and resources feed the decisions?
Share methods, data, curves, benchmarks and software across the themes. The expected outcome is a quantified decision, with its assumptions, its limits and a review condition.
In depth
Evidence resources : mechanisms, measurements and decision thresholds.
A reference resource is useful only if its scope, its version, its assumptions, its provenance and its conditions of use are maintained.Methods, data, curves and benchmarks must be traceable to a specific decision. The reader has to tell an observed value from an estimate, a simulation and an editorial judgement.
Observation or resource identified by its source, its population, its unit, its period, its version and its usage rights.
State the collection date, the freshness that matters for the decision and the review cadence; a recent date does not fix an inadequate definition.
Compare distributions and methods on a consistent perimeter; never treat an external benchmark as an automatic local target.
Publish sample size, dispersion, exclusions, transformations, selection bias and the limits on reproducibility.
A resource enters a decision only when its provenance, its applicability and its transformations can be audited.
Points of vigilance
The distinctions that prevent a misreading.
Who produced the data, how, on which population and on what date.
In which sectors, scales, horizons and units the resource can be used.
What changed, why, and which earlier result it may affect.
Protocol
From framing to validation.
Document source, method, period, grain, unit, licence and limits.
Check consistency, extreme values, reproducibility and sensitivity.
Provide a usage example, a counter-example and the decision it supports.
Set an owner, a cadence, a change log and an archiving rule.
A benchmark without its distribution creates false precision
Situation. An average return of 1.8 is published from 40 campaigns, with a median of 1.1 and an interquartile range of 0.6–2.0.
Calculation. The mean is pulled up by a few extreme campaigns; it does not represent the most frequent outcome.
Decision. The resource must publish distribution, population, unit and method before any comparative use.
Adversarial review
Five questions to settle before presenting a recommendation.
- Is the unit actionable?
Check that it matches the level at which a decision can be executed and tracked. A broader average can reverse the reading by hiding loss-making segments, channels or periods.
- Does the horizon cover the mechanism?
Recompute the conclusion over a short window and a long one. Immediate costs, retention, saturation and delayed effects do not materialise at the same time.
- Does the comparator really represent the absence of action?
Explain what would have happened without the decision. A historical trend or an attribution of contacts is not enough if selection, seasonality or simultaneous changes remain active.
- Which assumption tips the choice?
Identify the parameter that changes the ranking, publish its plausible range and show the prudent scenario. A recommendation without sensitivity hides its main risk.
- Who stops or revises the decision?
Name the owner, the observed signal, the threshold and the review date. Without a stopping rule, a favourable pilot easily becomes a permanent, unexamined commitment.
Acceptance conditions
What must be true before acting.
- No value without source, period, unit and population.
- No method without identification assumptions and contraindications.
- No silent update of a resource used to decide.
Limits
What the analysis does not prove.
- A sector benchmark is not a normative target.
- Public data can be well documented and still be unfit for a local decision.
- Freshness does not compensate for a poor definition.
References
Works used to frame the method.
- Wilkinson, M. D. et al. (2016), “The FAIR Guiding Principles for Scientific Data Management”, Scientific Data.
- Peng, R. D. (2011), “Reproducible Research in Computational Science”, Science.
Questions to settle
Choose your
next decision.
Start with the uncertainty most likely to change the choice: demand, margin, channel, measurement, capacity or risk.
Downloadable marketing mix modeling dataset with dictionary, quality profile, validation period and authorised uses.
Compare experiment, quasi-experiment, MMM, attribution, survey and choice model by question, estimand, bias, granularity and horizon.
Compare linear, exponential, Hill and monotone spline by support, saturation, marginal slope, uncertainty and decision use.
A teaching example on eleven simulated profiles, to turn a benchmark into a comparable distribution, a weighted prior and a decision interval.
A decision protocol for assessing marketing-science software on reproducible cases rather than vendor claims or unverified rankings.
Continue the analysis