Who Keeps the Internet’s Memory?

The internet gives the impression that everything is being saved.

Search engines seem to remember where things are. Cloud services keep copies. Social platforms hold years of posts. Government websites publish reports that look as permanent as a printed document on a library shelf.

Then a link stops working.

A project ends. A newsroom moves to a new publishing system. A ministry restructures its website. A company shuts down and its domain expires. The information may have existed for years, been cited many times and informed real decisions, and still disappear without notice.

The web is one of the largest records people have ever produced. It is also more fragile than it looks, and whether any of it survives depends on deliberate work by a small number of institutions.

The web was not built as an archive

A web address is a pointer, not a container. It tells a browser where to ask for something; it does not guarantee that the thing will still be there, or that it will be the same thing next time. Each page lives on a server that someone has to keep paying for, maintaining and organising.

That makes the web mutable by default. Pages can be edited in place with no visible trace of what they used to say. Sites are rebuilt and their address structures change. Content management systems are replaced, and old URLs are redirected to a homepage or simply dropped. Domains lapse and are bought by someone else. Organisations close and their servers are switched off.

None of this is unusual or sinister. It is how the web normally works. The assumption of permanence comes from habit: a printed report, once distributed, exists in many copies that are hard to change. A web page usually exists in one place, under the control of one owner.

The Library of Congress describes websites as ephemeral and “at-risk born-digital content”: new sites appear constantly, URLs change, content changes, and sites sometimes disappear entirely. UNESCO made a similar point in its 2003 Charter on the Preservation of Digital Heritage, noting that digital materials are frequently ephemeral and “require purposeful production, maintenance and management to be retained.”^[8],[9]

What link rot actually means

Researchers distinguish two problems. Link rot is when a URL stops returning anything useful: an error page, a dead server, a domain for sale. Content drift is subtler: the link still works, but what it shows has changed, sometimes so much that it no longer supports the claim it was cited for. Together they are often called reference rot.^[3]

The scale is measurable. In a 2024 analysis, Pew Research Center sampled nearly a million pages captured by the Common Crawl between 2013 and 2023 and checked whether they still existed. A quarter of them were no longer accessible by October 2023. For pages from 2013, the figure was 38%. Pew counted a page as gone only if it returned one of a set of error codes that clearly indicate the page or its server no longer exists, so its estimate is deliberately conservative and does not measure content that has changed.^[1]

The same study found that 23% of sampled news pages and 21% of sampled government pages contained at least one broken link, and that 54% of sampled English Wikipedia articles had at least one reference link pointing to a page that no longer existed. On X, then Twitter, nearly one in five sampled posts was no longer publicly visible within a few months.^[1]

Research and law are affected too. A 2014 study by Jonathan Zittrain, Kendra Albert and Lawrence Lessig found that more than 70% of URLs cited in three Harvard law journals, and about half of the URLs cited in United States Supreme Court opinions, no longer led to the material originally cited. The same year, a team at Los Alamos National Laboratory and the University of Edinburgh examined over a million web references in scientific, technical and medical papers published between 1997 and 2012. They estimated that around one in five such articles suffered from reference rot, rising to about seven in ten among articles that cited web resources at all.^[2],[3]

These are not the latest possible figures, and link rot rates vary by field, publisher and age of the link. But the direction across studies is consistent: the longer a link has existed, the less likely it is to lead where it once did.

The Internet Archive

The Internet Archive is a nonprofit digital library based in San Francisco. Since 1996 it has been capturing websites and making the captures available to the public. Its best-known service, the Wayback Machine, lets anyone enter a web address and browse earlier versions of it.^[4],[5]

In October 2025 the organisation marked a milestone of one trillion web pages preserved and available through the Wayback Machine. Its collections also extend well beyond the web, to digitised books, audio, video, television and software.^[5],[4]

The number is less important than what it represents. Much of what now counts as the history of the early and middle web — personal pages, small organisations, defunct services, earlier versions of major sites — survives in a usable form largely because it was captured at the time. Many national libraries and other institutions run their own web archives, often selective and focused on a single country or subject; the Internet Archive’s distinguishing feature is the breadth of what it attempts to capture and its open public access.

That breadth also concentrates risk. In their 2014 paper, Zittrain, Albert and Lessig noted that the Internet Archive is a single organisation, “privately funded and voluntarily supported,” and argued that distributed, redundant preservation is a better long-term answer than relying on any one archive. The point is not a criticism of the Archive. It is an observation about how much of the public record now rests on a small number of preservation efforts.^[2]

How the Wayback Machine works

Most archived pages are collected by automated crawlers that follow links across the public web and save what they find. Each capture is a snapshot: a record of what a particular address returned at a particular moment, identified by a date and time embedded in the archived URL. A popular page may have thousands of snapshots across many years. An obscure page may have none.^[6]

Coverage is uneven by design and by necessity. The Internet Archive notes that crawlers tend to find sites that are well linked from elsewhere, and that some pages are missing because the crawler never learned they existed, because they were password-protected or blocked by a site’s robots.txt file, or because the site owner asked for them to be excluded.^[6]

Even captured pages may be incomplete. Images and other files may not have been saved alongside the page. Interactive features that need to contact the original server will not work in an archived copy. Navigating an incomplete capture can silently pull in parts from a different date. Anyone can also ask for a single page to be captured through a feature called Save Page Now, but that saves one page once; it does not add the site to future crawls.^[6]

A snapshot, in other words, is not the current website, and it is not necessarily the whole of a past website either. It is a dated record of what one address returned when someone, or something, happened to ask.

A snapshot is evidence of a moment

That limitation is also what makes snapshots useful. A dated capture can show what a page said on a specific day. It can help reconstruct a project that has since shut down, recover a document whose link broke, or show how an organisation described itself before it changed its message. Researchers use archived pages to study how online information spreads and changes. Journalists use them to check whether a statement was edited after publication.

Archived pages are also used in more formal settings. The Internet Archive says that although the Wayback Machine was not designed with legal use in mind, it regularly receives requests for certified records for legal proceedings, and it has a procedure for providing affidavits.^[6]

There is an important distinction here. An archived page is good evidence of what was published at an address at a given time. It is not evidence that what the page said was true. A preserved press release still contains the claims of whoever issued it; a preserved forum post is still one person’s opinion. Archives preserve the record of publication, not a verdict on its accuracy.

Preservation is not neutral or complete

It is tempting to think of web archives as a neutral mirror of the web. They are not, and no archive could be.

Every archive makes choices, whether by explicit selection or by the behaviour of its crawlers. Well-linked, publicly reachable, technically simple pages are far more likely to be captured than pages behind logins, inside apps, in less-linked parts of the web or in languages and regions that crawlers visit less often. Content behind paywalls or authentication is often out of reach. Media files are expensive to store and easy to miss.

There are also deliberate limits. Archives must handle copyright, privacy and removal requests. UNESCO’s charter states both that preserved digital heritage should be accessible “free of unreasonable restrictions” and that “sensitive and personal information should be protected from any form of intrusion.” Those principles can pull in opposite directions, and different archives resolve them differently.^[9]

Preservation also costs money, storage, energy and staff time, indefinitely. UNESCO identified “uncertainties about resources, responsibility and methods” as one of the main threats to digital heritage more than twenty years ago, and that has not gone away.^[9]

The honest description is that web archives preserve a large, valuable and uneven part of the record. Knowing what they tend to miss is part of using them well.

Software is culture too

Some digital culture cannot be preserved as a page at all. A game, a piece of creative software or an early interactive work depends on executable code, an operating system, particular hardware, file formats and sometimes a server that no longer exists. UNESCO lists the rapid obsolescence of the hardware and software that bring digital materials to life as a central risk.^[9]

The Internet Archive’s software collections take one approach: preserving the programs themselves and, for some historic software, making them usable in a browser through emulation, so that a visitor can run a program rather than just look at a picture of it.^[7]

A screenshot of software shows what it looked like. It does not show how it behaved, what it asked of the person using it or how it felt to use. For interactive culture, keeping the experience available is a different and harder problem than keeping the image.

The problem becomes harder as the internet grows

The early web was mostly documents linked to other documents. That is the kind of material crawlers capture best. Much of today’s internet is different.

Pages are increasingly assembled in the browser by JavaScript, pulling content from separate services after the page loads. The Internet Archive notes that JavaScript-driven elements are often hard to archive, and that features which need to contact the original server will fail in an archived copy. As a general rule, it says, simple HTML is the easiest thing to preserve.^[6]

Beyond that are whole categories that sit largely outside public crawling: personalised feeds that differ for every user, content inside apps, private groups, streaming media, cloud tools whose documents live on a company’s servers, and platform data available only through interfaces that can be changed or withdrawn. When a service closes, its users’ contributions can vanish with it.

A future historian trying to understand the 2020s may find it easier to recover a static personal homepage from 2002 than to reconstruct what a typical person actually saw on a typical day in a modern app. That is not inevitable, but changing it would take deliberate effort from platforms, archives and publishers.

What should a publication preserve?

This question applies to us as well. A publication that cites the web is making a promise to its readers: here is where this came from, and you can check it. If the cited page later changes or disappears, the promise quietly breaks.

A useful principle is that a citation should remain useful even if the original page later changes or disappears. For journal articles with persistent identifiers such as DOIs, or for formal datasets maintained by stable institutions, the original link is usually the most durable option. For ordinary web pages — organisation pages, announcements, explainers, news articles — it often is not.

From this piece onward, some sources on It’s Important include an archived copy alongside the original where one exists. The original remains the primary citation; the archived copy is a dated record that a reader can fall back on. On this page, several of the sources below carry one. The links are chosen editorially and add no third-party requests when a story loads.

This is a small step, not a solution. It does nothing for sources that were never captured, and it still depends on archives continuing to exist. But it treats preservation as part of citing rather than as an afterthought.

The web remembers only when someone makes it

Paying attention to something only matters while the evidence remains available to examine.

The internet has made publishing easier than at any point in history. It has not made preservation automatic. Pages go on changing and disappearing at the ordinary pace of the web, mostly without anyone noticing until they need something that is no longer there.

Archives cannot save everything, and a snapshot cannot recreate every part of the web. But keeping enough of the record lets future readers ask better questions about what existed, what changed and what disappeared. That depends on people deciding, again and again, that it is worth doing.