Sleeplessness has a long and tortured history. A 15th-century Italian lawyer named Hippolytus de Marsiliis is said to have first documented sleep deprivation as a way to punish prisoners. (If you’re unconvinced by his cred, note that he is the same fellow credited with confirming the effectiveness of slow-drip water torture.) And he was only making formal what humans had known for centuries: Not getting enough sleep is painful.

Today, the problem of too little sleep, and the quest for more of it, is as acute as ever: 27 percent of people in a new Consumer Reports survey of 4,023 U.S. adults said they had trouble falling asleep or staying asleep most nights, and 68 percent—or an estimated 164 million Americans—struggled with sleep at least once a week. 

A good night’s sleep can require everything from the practical (a cool, comfortable pillow) to the ethereal (a deep sense of calm and peace of mind). The modern marketplace has exploded with proffered solutions for people who can't sleep, from mattresses to white-noise machines, sleeping pills to sleep coaches.

Americans spent an estimated $41 billion on sleep aids and remedies in 2015, and that’s expected to grow to $52 billion by 2020, according to Natana Raj, an analyst with BCC Research in Wellesley, Mass. The rub is that certain solutions don’t work as well as claimed—if they work at all.

Sleep drugs are arguably the most significant concern. About one-third of the people we surveyed had tried either a sleep drug (such as Ambien, which requires a prescription, or Sominex, which does not) or a dietary supplement (especially melatonin) at least once in the previous year.

Forty-one percent of people who use over-the-counter sleep aids reported taking them for a year or longer, and 48 percent use the drugs several times per week. Most concerning: The way people misuse sleep drugs can be dangerous, our survey found.

Why Sleep Eludes So Many

Sleep experts today aren’t as prescriptive as they once were about how much shut-eye humans require each night.

“You need as much sleep as it takes for you to stay awake and alert the next day, without caffeine,” says Nathaniel Watson, M.D., co-director of the University of Washington Sleep Center in Seattle and president of the American Academy of Sleep Medicine, which last summer issued new sleep guidelines. It’s also okay if you wake up in the middle of the night, as long as you fall back asleep, Watson says.

Still, most people don’t function well with less than 7 hours of sleep, Watson says, and regularly getting less than that amount can, over time, harm your health. Continued sleep shortages contribute to depression, heart disease, lowered immunity, obesity, and type 2 diabetes, among other ills, he says.

Why are so many of us sleeping so little? One reason is simple math: Americans are working longer—an average of 44 hours per week, according to an August 2015 Gallup Poll. That’s the longest workweek in the history of Gallup’s annual work survey. And a record 17 percent of U.S. adults now log 60 or more hours per week in the office, leaving less time for sleep.

Working longer hours, plus having longer commutes, leaves less time for domestic chores—paying bills, doing repairs, dealing with paperwork for taxes or kids’ college—which get stuffed into twilight hours.

In today’s global economy, working late into the night or first thing in the morning is often a necessity. That kind of shift work—once mainly confined to nurses, emergency room doctors, factory workers, and police officers—can wreak havoc on the body’s sleep-wake cycle.

And then there are modern habits: Many people are staying up late to entertain themselves on electronic devices—even as research suggests that blue light from those devices may interfere with the body’s production of melatonin, the sleep hormone.

What Works—What Doesn't

The Consumer Reports Best Buy Drugs team, which evaluates the safety and effectiveness of medications, reviewed the research on sleep drugs and found them to be limited in their effectiveness and prone to potentially dangerous misuse.

Our medical experts looked at the research on melatonin as well as the lifestyle changes that improve sleep. We also have Ratings from our labs on dozens of mattresses, as well as advice on how to find a pillow that matches your sleep style.

Overall, our review of the products, drugs, and services from the sleep industry found a whole lot of marketing spin and misinformation but also some positive news about therapies and products that can do some good in the age-old quest for better sleep.

Editor's Note: This article also appeared in the February 2016 issue of Consumer Reports magazine.

These materials were made possible by a grant from the state Attorney General Consumer and Prescriber Education Grant Program, which is funded by a multistate settlement of consumer fraud claims regarding the marketing of the prescription drug Neurontin (gabapentin).