The U.S. Census: Driving Decisions With Data for 230 Years

After Article I of the United States Constitution establishes the legislative power of Congress, it briefly describes the process for determining the number of representatives for each state. That process, called “Apportionment,” is based on the decennial (every 10 years) census — a survey of the population of the nation that gives lawmakers an understanding of the size and composition of the population. That information is then used to inform a wide variety of decisions at all levels of government, from distributing funding for federal programs to redrawing voting districts and planning transportation systems until the next census takes place.

Fundamentally, a census provides the government with essential data, without which effective governing would be all but impossible. They’re hardly a modern invention; ancient civilizations used censuses to better understand the makeup of their populations and their vocations, determine taxes and estimate resources.

But today’s U.S. census expands on these historical censuses in many ways. In addition to surveying the population, the U.S. Census Bureau conducts more than 130 surveys of the nation, covering topics including housing, consumer expenditure, manufacturing, health and hospitalization, and many others — even including a National Crime Victimization Survey.

Each of these surveys is part of a massive effort to drive decisions with data on an enormous scale. Population growth and explosive changes in technology offer new opportunities for the Census Bureau, but they also create challenges.

The first U.S. Census was conducted in 1790, under the direction of Thomas Jefferson. For the first hundred years of the census, the primary tool for ingesting and processing census data was a pencil. (Or a pen.) This low-tech approach was sufficient for nearly a hundred years.

By 1880, the population of the United States had grown to more than 50 million. Census takers had to travel everywhere in the growing nation to collect data, then manually process it. And this was happening while the topics the census aimed to cover were expanding. The system couldn’t last — processing the data by hand was no longer possible, and new methods for processing the data had to be invented. For the 1880 census, use of a “tabulating machine” sped up the process significantly, but it still took nearly the entire decade to complete and publish the results.

The Census Bureau held a competition in 1888 to find a more efficient means of processing the recorded data, and the winning design — Herman Hollerith’s Electric Tabulator — marked the beginning of a new era. Hollerith’s machine completed the sample tasks exponentially faster than before. The machine used punch cards — a system which was still in use until the mid-1950s, when the Bureau developed Film Optical Sensing Device for Input to Computers (FOSDIC), which processed microfilm photographs of questionnaires and stored the data on magnetic tape. FOSDIC was first used in the 1960 census, and remained integral to the Census Bureau’s data processing system until the mid-1990s.

While elements of the census have changed since punch cards and magnetic tape were phased out, the 2020 census represents a monumental change.

For the first time, people living in the U.S. will be able to respond to the census online, by phone or by mail. All households will receive information about how to respond in 12 non-English languages online and by phone, and about 9.3% of households (or 13 million people) will receive English/Spanish bilingual invitations.

The Census Bureau will have additional layers of complex data to work with. There’s the census data itself, plus all the data generated by the census takers and their devices, all of which needs to be monitored for performance and security. Census takers will use secure devices to manage every aspect of their work — from getting their daily assignments and communicating with supervisors to submitting timesheets and flagging issues.

To achieve their goal, the Census Bureau is working with Splunk. Splunk’s dashboards, reports and data visualizations deliver real-time insights to help teams all over the bureau bring data to every question, decision and action.

Check out the full story, to learn more about how Splunk is powering success for the Census Bureau and the 2020 census.