Freight data · Methodology

Freight Data Methodology and Data Quality

The rules behind every figure in the Freight Data Center and the research reports. This is the one methodology page for both.

By RapidShips Dock OperationsLast reviewed 2026-09-301 source

Data Quality & Methodology

Where the data comes from

Only from the organization that collects it: the U.S. Bureau of Labor Statistics, the Bureau of Transportation Statistics, the Census Bureau and the Port Authority of New York and New Jersey. News sites and data resellers are never the source of a figure.

How it is processed

Values are copied exactly from the publisher's API, file or table into a saved snapshot, then into the dataset. The build refuses to publish a value that differs from the snapshot by a single digit.

How often it is updated

Each dataset shows its update frequency and the publisher's next release date where one is announced. Updates are made by hand after review, so there can be a gap of days after a release. Nothing on these pages is live.

What RapidShips changes

Nothing in the figures. RapidShips adds charts, tables, year-over-year changes and short explanations. Every calculation is labelled RapidShips analysis and shows its formula.

What we do not know

These datasets do not show freight rates on a lane, terminal congestion, dwell times, container availability or what any shipment will cost. Where a figure cannot be verified, it is not published.

Estimates and forecasts

None are published. If one is ever added it will carry an Estimate or Forecast label and will never be presented as a fact.

How to verify the originals

Every dataset links to the original source and names the series codes, files or tables used. Open the source, find the same period, and the numbers will match.

From source to page

Eight steps. Only the fetch step talks to the internet, and only a person can publish.

  1. SourceChoose an official publisher that collects the data itself: BLS, BTS, the Census Bureau, the Port Authority. Record its licence.
  2. Fetchscripts/fetch_data.py downloads the figures from the publisher's API or files and saves a raw snapshot with the URL and time.
  3. ValidateThe build checks source, period, unit, geography, dates, duplicates and labels, and stops on any failure.
  4. NormalizeEvery stored value is compared with the raw snapshot. Nothing is rounded, filled in or estimated.
  5. StoreThe reviewed values live in content/data/datasets.json with their metadata.
  6. ReviewA person reads the fetch report and sets the status: published, needs-review or draft. Nothing publishes itself.
  7. PublishThe static build writes the pages, charts, tables, downloads and schema.
  8. UpdateWhen the source releases new figures, fetch again. The build flags data that is stale or past its expected update.

Validation rules

The build stops if any of these fail.

  • Missing source: every published dataset names its source organization, an https source URL and source records that exist in sources.json.
  • Missing period: every dataset states a start and end period, and every value has its own period.
  • Invalid unit: units must be in the unit register (TEUs, index bases, jobs in thousands, establishments, employees).
  • Unknown geography: geographies must be in the geography register.
  • Unexplained future dates: no value, retrieval, publication or update date after the build date unless the dataset is a labelled forecast with a stated reason (there are none).
  • Duplicates: no repeated dataset id, series id or period within a series.
  • Unsupported or fabricated values: every published value must match the raw snapshot saved from the source by the fetch step, to the digit. A value that is not in the snapshot stops the build.
  • Labels: datasets hold only FACT values. Any calculation (a percentage change, a share) is computed at build time from published values and labelled RapidShips analysis with its formula. Estimates and forecasts are refused.
  • Stale data: the build warns when a monthly dataset's latest period is more than 150 days old or an annual one more than 1,100 days, and when a source's announced release date has passed. The page then says an update may be available.
  • Needs review: a dataset marked needs-review stays visible with a warning but is removed from search, the sitemap, Latest data and compare.

Fact, analysis, estimate, forecast

Every figure carries one of these labels.

Fact
A figure exactly as the source publishes it.
RapidShips analysis
A calculation by RapidShips from published figures (for example a percentage change). The formula and inputs are shown.
Estimate
A modelled or approximate figure. Never shown as a fact. None are published today.
Forecast
A projection of a future value. None are published today.

Source classes

Each source is classed so you can see how close it is to the original data.

Primary official statistics PRIMARY_OFFICIAL
The federal statistical agency that collects and publishes the figure (for example BLS, BTS or the Census Bureau).
Port authority PORT_AUTHORITY
The port authority's own published cargo figures.
Government GOVERNMENT
A government page or document that is not the statistical producer.
Regulatory REGULATORY
A statute, regulation or regulator's rule text.
Industry INDUSTRY
A trade association, carrier or terminal operator. Used for context, not for statistics on this site.
Secondary SECONDARY
A republisher of someone else's data (for example a news site or FRED). Never used as the source of a figure here.
RapidShips internal INTERNAL
RapidShips' own operating data. None is published in the Data Center.
Estimate ESTIMATE
A modelled figure. None are published in the Data Center.

Licences, downloads and embeds

U.S. federal statistics are generally in the public domain. BLS states that its material is public domain and asks to be cited as the source.[1] BTS labels its Transportation Services Index dataset Public Domain U.S. Government. Those datasets offer CSV and JSON downloads and an embeddable image.

The Port Authority of New York and New Jersey does not state an open-data licence for its cargo tables, so RapidShips shows those figures with attribution and a link, and offers no file download or embed.

Embeds are static SVG images with the source and a link back, not iframes. They carry the data period in the file name and do not change after publication.

Sources

  1. U.S. Bureau of Labor Statistics, “Copyright Information,” Terms of use. Accessed September 30, 2026.