Categories
Circadian science Technology

Scientific Tests

Algorithms are really easy to mess up. Take your pick for how: overfitting to training data, having bad training data, having too little training data, encoding human bias from your training data in the model and calling it “objective”. Feeding in new data that’s in the wrong format. Typos, subtle typos, nightmarishly subtle typos. Your cat stepping on the keyboard when you’re out of the room.

Having done this for a while, my first impulse any time I get amazing performance with an algorithm is to be deeply suspicious. This isn’t because algorithms can’t be incredibly powerful; you really can get amazing performance if you do them right. But you can also get seemingly amazing performance if you do them wrong, and there are a lot of ways to be wrong.

The core issue, I think, is that there are so many choices involved in the making and maintaining of an algorithm, and if the algorithm is trying to do something complicated, those choices can have complicated downstream effects. You can’t readily anticipate what these effects might be (this problem is hard enough that you’re building an algorithm for it, after all), but your brain tells you that there can’t be that much of a difference between a threshold being 0.3 and that same threshold being set to 0.4. So you blithely make the change to 0.4, expecting minimal effects, and then the whole thing just collapses underneath you.

I’m saying this because, while developers on the whole have gotten pretty on board with the concept of unit tests and test coverage for code, I’m not as sure about what currently exists around tests for data science algorithms in biology, medicine, and health. I’m not talking about tests that confirm a function gives the number we expect it to based on an old run of it (e.g. asserting that f(3.4) = 6.83 because we ran it once with 3.4 and got 6.83 as the answer). Those kinds of tests basically act like a flag that something’s changed, and if you change your function f, you can just paste in the new output from f to make the test pass.

I’m talking about using a lot of data — a representative sample of what you’ve collected— in your code’s tests, to assert that some macro-property of the algorithm’s output is preserved. If you make a change to the algorithm, and that macro-property changes, your code should let you know about it. If this sounds like one flavor of functional testing, that’s because it is. But the key thing I’m arguing for is that the performance of data science-components of a product— code two steps removed from sklearn or torch or R—be tested functionally as well.

Let’s talk through an example. Imagine you’ve got an algorithm for distinguishing sleep from wake over the course of the night. In my own experience, algorithms of this kind can have an unfortunate tendency to flip quickly, in response to a threshold being changed, from thinking a person slept a lot over the night, to thinking that person barely slept at all. There are ways to address this, and it’s not always an issue, but it’s still something to look out for.

So how could you look out for it? Take a representative sample of data from a group of sleeping people, and add a test that runs your sleep algorithm on all of them and asserts that everyone gets detected to have at least some baseline amount of sleep. This way you don’t have to worry that changes to the algorithm made some people better but other people unexpectedly, dramatically worse. You can rest easy because assurances of this kind are built into your development pipeline.

At Arcascope, a lot of our scientific tests center around the mean absolute error of our prediction of melatonin onset. We want to predict when somebody’s melatonin onset is happening so we can map it to other quantities of interest: minimum core body temperature, peak athletic performance, peak fatigue, you name it. But we also want to do continual development work on these algorithms, to keep bringing the mean absolute error down over time. How can we make sure that our changes actually make things better? How can we make sure that a change that improves performance for some people isn’t sabotaging others? Tests that confirm the properties we care about are preserved, no matter what we do in the backend.

I mentioned above that I don’t know what’s out there elsewhere in digital health, and I don’t. Maybe a lot of people are writing tests of this kind in their code! All I know is that when we realized we could do this— add fundamental performance checks for the algorithms that make up our backend, embedding deidentified human data from our studies directly into our testing suite— it was a very cool moment. It helps me sleep a lot better at night knowing those tests are there.

Which is good, because sleeping more means I’m introducing fewer nightmarishly subtle typos to our code. Wins all around.

Categories
Circadian science Technology

Biophysics for Better Living 2.0

In my last blog post, I talked about the power and potential for biophysics engines to contribute to clinical care and ragged on my own ability to play video games. The short version of it, if you don’t have time to circle back: If you’re going to make a digital twin of somebody, base it on a system of differential equations that captures how the physics of the human body work. Also, I’m terrible at video games.

I talked in that blog post about how you can use a biophysics engine—a.k.a. known properties of how the body works, codified into equations— to estimate things that are hard to measure because they’re hard to reach. Things like the firing rates in your ventral and dorsal suprachiasmatic nucleus (SCN), which are tucked pretty deeply inside your brain. You’re not going to be able to easily peek and see what the SCN is up to (at least not right now, in 2022), but you can figure out what your SCN is likely up to passing the same inputs your real SCN got into a model of the SCN.

Woke up really early and got a blast of light? That probably sped up your clock. Stayed up late and kept the lights on overhead? That probably slowed your clock down. Where did your clock wind up at the end of it all? That’s a question that hinges on the details of when, how long, and how bright your light exposures were. In other words, it’s a question for the mathematical model. 

But there are other questions you can ask a model, beyond “where am I now?” You can ask “what if?” What if I were to stay up the entire night in bright light? What if I dimmed the lights at 3:15 pm? How would these changes affect the state of my internal clock? And how would that changed state affect downstream outputs, like the timing of my peak fatigue or my peak performance?

When we talk about what we do at Arcascope, we say we do circadian tracking and circadian recommendations. The tracking is using a biophysics model to answer the question “where is your circadian clock right now?” The recommendations come from repeatedly asking the model “what would happen if you did (insert series of behaviors here)?” Asking a lot of questions gets you a lot of answers, which you can then pick from based on which ones best meet your goals, whether that’s sleeping more, adjusting faster, or being maximally alert at a specific time. 

It’s all still biophysics models, but in one use-case, we’re putting them to work to capture reality, while in the other, we’re trying to capture a whole swath of possible futures. A list of potential realities to choose from as you scope out the rest of your week. Better living by picking the best answers to “what if?”— I think that’s pretty neat. And it keeps me up at night thinking—what if we could do this for other systems, besides sleep and wake, too?

Interested in beta testing our app? Send us an email!

Categories
Circadian science Technology

Biophysics for Better Living

One of the highest stakes moments of my life was the time in graduate school when I was (badly) playing Donkey Kong at Pinball Pete’s Arcade, and a very, very good Donkey Kong player came over to watch me play in silence. I’d just spent six dollars worth in quarters dying repeatedly on the first floor of the first level, but something about his judging, appraising eyes summoned the fireball-hopper within. I made it past the first level, then the second, then the third—an absolute record for me—and I had almost cleared the fourth when the pressure finally got the best of me. Still, it felt like a tremendous achievement. I like to think the good player and I shared a nod of mutual respect as he elbowed me out of the way and proceeded to spend the next forty-five minutes on an uninterrupted victory run. 

Besides letting everyone know that I need to get out more, this story is an opportunity to talk about a kind of math that should be more integrated into clinical care, and isn’t— yet. There was a model of physics deciding what happened in that game, a system of differential equations for converting my button mashing into movement across a screen. And just like the inputs I gave the system could be transformed into outputs that captured the physics of motion, so too can the inputs we give our bodies be transformed by a realistic biophysics model to give us outputs that capture the ways our bodies work without the need for invasive tests. 

Let’s back up: At a very high level, what’s going on in a video game is that you’ve got prescribed rules for how to update all the positions of all the moving pieces from one frame to the next. In the next blink of an eye, where should Mario go? If I’m pushing the joystick to the left in that instant, nudge him a little to the left. If I’ve just hit the jump button, nudge him a little bit upwards. If I’m mid-jump, make the force of gravity and the speed he took off the ground fight it out to see if he should be nudged a little up or a little down. 

The whole set of rules for making these decisions on how to nudge can be called a physics engine. Passing the history of my joystick spins and button presses into this physics engine gets you an output that describes how Mario moves, how he jumps, and how far I make it in the game before an untimely demise. This type of math is all over the place in games, CGI, and user interaction elements (like views that scroll and bounce). But there’s no reason a physics engine has to be restricted to visuals, or video games, or the physics of movement. You can have a physics engine that captures ion channels open and close, for instance, or how neurons talk to each other in the brain. 

Which brings us to your body’s internal clock. Circadian rhythms are a massive pain to measure experimentally; it’s one of their defining properties. It’s hard to know what’s going on in the brain without looking at it, and it’s hard to get a glimpse of it without getting pretty invasive. But if you know the rules for how the clock part of your brain works— “the differential equations describing the suprachiasmatic nucleus”— you can pass through a history of inputs to the system, and get out an output that describes how the neurons are communicating with each other at a specific point in time. Pass in what my lighting history has looked like over the last four weeks, and I can tell you what time my brain would cue melatonin to start rising in my body tonight. 

So here’s why I think we should be using differential equations in medicine more: they can help you see things you otherwise wouldn’t be able to see. You can use a system of differential equations to track how the brain would react to the inputs you give it, even if you can’t readily experimentally measure brain state, just how a person with access to Donkey Kong’s code could track how far I got on my amazing run, even if someone blocked off the console screen with a tarp. Knowing the rules means you can measure things in the dark. 

And there’s another reason, too—personalization. By tweaking the rules of your video game physics engine, you can change the way the gameplay works. Cut gravity in half, and suddenly Mario’s soaring with every jump. Double it, and he’s a sitting duck for everything Donkey Kong throws at him. As the importance of individual differences in light sensitivity and fatigue management become better understood, biophysics engines are ready-made for customization. Want to better fit the data of someone who’s hyper-sensitive to light? Crank up the light sensitivity parameters. Got genetic data on them? Plug that into a molecular model of their rhythms

Biophysics engines can be the key to a true digital twin, helping us look inside a simulation of ourselves in ways we can’t look inside our real selves. There’s potential for these quantitative methods in medicine beyond sleep and circadian rhythms, too. Who knows? Maybe someday a model of this kind could be useful for understanding how the (fun) stress of my epic Donkey Kong run affected my mood and memory. Or at the very least, how my body processed the enormous bubble tea I drank in triumph right after. 

Want to learn why we don’t think the answer to circadian estimation in the real world is to just throw an artificial neural net at it? Check out this blog post by CTO Kevin Hannay. 

Categories
Circadian science Interviews Sleeping troubles Technology

Interview with Dr. Cathy Goldstein

We sat down to talk with Dr. Goldstein on what she sees as the future of wearables in the sleep clinic. Enjoy!

Thank you for taking the time to meet with me today! Could you take a minute to introduce yourself and say a little about what your job is?

Cathy Goldstein, MD. I’m a neurologist, Associate Professor of Neurology here at Michigan Medicine, and my clinical practice is entirely in sleep medicine.

You’ve done work with [Arcascope CEO] Olivia on wearables, and you’re heavily involved in the AASM and consumer sleep technologies. Where do things stand today for consumer sleep technology and wearables, and where do they need to go?

These devices really started becoming prominent about 8 years ago. I remember I was right out of my fellowship at that time, and I thought, wow, these will be so awesome for when a patient sees me in the clinic: I have all these questions about their sleep and sleep timing, and how they sleep on the weekdays vs weekends vs vacation, and they’re in the position of having to try and remember what their problems have been over the long time they’ve been waiting to see me. 

And the problem with a sleep log [that someone fills out manually] is that it’s hard to keep up with every day. In this busy world, you need more of a passive degree of tracking, particularly over time. So when wearables came out I was like, wow, you know, wouldn’t it be cool if when the patient shows up I can just take a look at their data in the last six months and see their sleep patterns. 

The problem as far as adoption has gone, is that we don’t know how accurate they are at tracking sleep. So, there’s a medical grade version of consumer trackers called “actigraphy”, although they’re not completely accurate, we KNOW how accurate they are. And so, particularly in science and medicine, what is known is better and less scary—even if the performance isn’t great— than the unknown. 

There are a lot of unknowns in regard to consumer-targeted sleep tracking because there’s not a lot of peer-reviewed literature about the performance of these. It’s growing, but even when the accuracy does get reported in a paper, oftentimes the algorithms change with updates, or the hardware changes a little bit with each iteration. The big thing is—we just don’t know. And that’s really prevented adoption in research and clinical practice, unfortunately. 

What are the biggest misconceptions about sleep wearables that you see out in the world?

The biggest misconception is that they’re totally inaccurate, that they’re inferior to actigraphy for some reason, because the data that we have so far doesn’t necessarily suggest that. For many of these devices, we just don’t know, but the ones that we do know about appear to perform similarly. 

I’d say the other misconception is people say, “My watch tracks my sleep.” Your watch doesn’t track sleep: your watch tracks your heart rate or your pulse, and your watch tracks movement. And then, math estimates your sleep from that device. So, that’s a big misconception of people telling me in the clinic, “My device is telling me I don’t have REM, so my sleep is really bad.” Sleep stages like REM, NREM  are polysomnography EEG constructs. So, to use a consumer wearable sleep tracker’s components of dream sleep, light sleep, deep sleep, interchangeably with PSG-defined states— we’re just not there yet in my opinion. 

I do think the estimates are getting better, and they do use properties of heart rate variability that we know change in the different sleep stages, but still, those [stages] are defined by brainwaves and eye movements. So, it’s very hard to recapitulate that. The question is, “Do we even need to track that on a daily basis?” What we’re really looking for, and what the clinical and research world wants, is to track on a longitudinal basis— objectively— just sleep-wake patterns day to day.

How do sleep trackers fit in with sleep clinics right now?

People sometimes think that doctors just don’t want to look at sleep wearable data because we don’t like it—that’s not true. A lot of us love seeing this data and love seeing sleep patterns over time, particularly if a patient can tell me “Hey you know, you treated my sleep apnea with this CPAP machine, and look at the change in my wearable tracked sleep.” One of the massive problems, though, is that clinical practices are really, really busy. We have a lot of patients that need help, and we need to give everybody high-quality care. 

This means we have a high volume of work and most of our work takes place through something called the Electronic Health Record, and at this point, there is no real way to interface the data that comes from consumer wearable devices with that Electronic Health Record. So, we really don’t have a good way of integrating any of this into our clinical practice. It’s not that we are absolutely opposed to using this as an adjunct, particularly the ones that have some idea about performance, but we don’t really have a way of getting it in there to make it an easy part of our workflow. Patients will send me screenshots, or they will show me their app in the clinic. 

I think medicine moves slower than consumer-geared technologies for a multitude of reasons, including security issues, but I do think that one day we will get there. And again, we’re not using these as diagnostic tools; we’re using them as an adjunct to our clinical decision-making. So, as long as they are reasonably accurate, and we overall know how they work, I think a lot of doctors would like to adapt them into their practice as long as we have a way of seamlessly integrating that data into our workflow.

Do you think we’re going to see circadian rhythms enter the clinic more in the near future?

Yeah, and what I would hope is that they don’t just enter the sleep clinic, but they enter the wellness and general health area as well. I think a lot of us are doing all of our body systems a big disservice by living in desynchrony with both our central clock and peripheral clocks. 

When my friends and family ask, “What are some of the best things I can do for my sleep and circadian health?” I say “Wake up at the same time every single day.” That’s going to entrain your circadian phase. And hopefully, you’re eating in line with that (and you’re not eating when you’re supposed to be asleep), and when you’re getting light when you wake up. We are all undergoing mild degrees of circadian disruption by varying our wake-up time and getting as much light at night as we are.

Definitely. Especially with screens. I was one of the people who thought my phone screen couldn’t possibly affect my quality of sleep.

Exactly! And I think people are becoming more aware. I mean when you talk about intermittent fasting, that’s kind of a chronotherapeutic measure. It’s so simple, but people get so excited about it. It’s like yeah, don’t eat when your body is biologically prepared to be asleep.

You’re right that it’s incredibly simple, and I think people really respond to small changes that make a big effect on their health and wellness.

And it’s not magical, it’s timing. It’s literally all about timing.

What do you believe the future holds for sleep technology? What are you most excited about?

I’m just hopeful for a day where we change the way clinical evaluation works now, where a  patient might be waiting for me to see them for months and months, and then I see them, give them instructions, send them home with sleep logs so I can see how they’re doing, and tell them to come back to the clinic in 6 months because I’m booked out I can’t see them sooner. 

I want to get to a point where I see them, and at that initial point of care I know what their last 6 months of sleep looked like, and then I can come up with an intervention that’s precise for them and that’s also adaptive based on how their sleep looks in response to intervention. 

Then possibly, when they do go home, we can change that intervention, maybe in an automated way, maybe with me being able to interact with an app, whatever it may be – but instead of writing down instructions and giving people medications, we’re using the mobile application as a prescription to really make patient-specific interventions that are based on wearable data.

So, it’s important to make it less labor-intensive for the patient because then it’s less likely to be done correctly or be done at all.

Exactly, we’re talking about behavior change here. That’s one of the cornerstones of so many diseases we take care of in medicine: they’re due to things that with behavior change could be different, and behavior change is hard. One of the behaviors that we’re really, really good at in current society is working with our devices and our apps. So, I think it’s just a no-brainer as far as delivery goes, I think it’s really time to make this stuff an adjunct and a helper in healthcare.

What have you seen in the clinic in the age of COVID? 

I’ve seen a dichotomy. I mean I don’t think we’ve ever collectively (people my age, middle age, most of my patients) have gone through a stressor like this. So, there are quite a few people who had significant insomnia; there are people that had COVID that had major health disruptions during and following that, including fatigue during the day. Then there are people who actually had marked improvements in their sleep because they can sleep according to their clock, and they have more time due to less commute and so they can extend their sleep a bit. 

There have actually been patients who go off alertness-promoting medications, and are a bit happier with their sleep. There was a great article about a gentleman who had always felt confined to the service industry because he was a night owl. He was always kind of stuck, like a bartender/server, and during COVID (because hours were more flexible with work from home), he was able to go into other industries and have regular hours at the time he selected, and have a more stable work path. Which is what he wanted. You know, we shouldn’t chronotype shame people. Just because you’re biologically late you shouldn’t be at a disadvantage in life, and that’s helped a lot of people’s sleep.

It’s nice to hear some good news come out of the year 2020. That’s all the questions I had lined up, but do you have anything you would like to add?

I think there’s also, what can wearables NOT do. So, we don’t know how accurate the oximetry is yet, we don’t use this as a way to diagnose sleep apnea, I don’t think that sleep staging is something we should rely on. It’s that sleep disordered breathing, is not really something we have not been able to pick up yet, so when people have some episodes of sleep snoring or gasping, feeling sleepy during the day – even if your wearable says you sleep great, that is a limitation and you should still see your doctor.

So, if you feel something isn’t right, it’s best to go in and talk to your doctor.

Exactly, trust your body. The manifestation of diseases is how you feel, not just how your numbers are. 

Categories
Circadian science Technology

Yesterday’s Weather

I love my Apple Watch.

The ability to track my exercise, heart rate, activity levels, and sleep has enabled a real awareness of how my physical and mental health changes over time. The ability to track personal health data over long time periods outside of laboratories is one of the most exciting developments of the last decade. I believe this data will usher in a new era of personalized, precision health which just wasn’t possible in the past. At Arcascope, we are at the forefront of developing algorithms to turn the data collected by wearable devices into insights that improve people’s lives. 

With that being said, the current state of things just isn’t all that satisfying when you think about what’s being left on the table. So much of the data being collected is uninterpretable. Knowing my current heart rate is cool, but what can I do with that information? The part of this that bothers me the most is that so much of this data is focused on the past. 

Here is a screenshot from Apple Health showing my sleep over the last month. You can see that I had some wake periods at 3 am at the beginning of the month. But how does this information really help me? 

Sleep tracking in particular reminds me of a weather app that can only tell you yesterday’s weather. Clearly, a weather prediction service that could only tell you the weather from 24 hours ago wouldn’t do well against the Doppler radar. It is useful to be able to say exactly how hot it was yesterday, and interesting to know how that compares to years past, but I really want to know if I should bring an umbrella with me when I leave the house. 

I can tell that I didn’t sleep well last night from the fact that I am feeling tired. Having a device to quantify exactly how poorly I slept can be useful for tracking long-term trends, but it isn’t all that useful on a day-to-day basis. 

Another snapshot from my Apple Health data. Doesn’t this remind you of a weather app pointing out how this weather’s month compares to historical trends? What about today? Or how about tomorrow?

Okay, enough of the weather prediction analogy. I’ve already pushed that analogy further than I should. First, unlike the weather, we actually have control over our behavior, and what we are doing now will change the forecast for our physiology tomorrow. Also, these variables are much more predictable than weather. 

The technology we have developed at Arcascope can answer questions like: 

  • What separates the days where I am at my best, from the ones where I am struggling? 
  • How can I alter my behavior now so that I will sleep better tonight?
  • When is it best for me to stop drinking coffee for the day? 
  • When will it be best for me to study, exercise, eat and relax? 
  • When is the best time to take my medication to minimize side effects? 
  • When should I avoid high stakes activities because my chances of making a mistake are highest? 

We believe this is the future of personalized health tracking. We also think it’s a heck of a lot more exciting than looking back at yesterday. 

Categories
Technology

Circadian Phase Estimation and Deep Learning

One of the most common questions we get at Arcascope is…

“Can’t you just do circadian phase estimation using machine learning?”

Living in the data age, we have become used to thinking that big data and machine learning can do just about anything. In this post, I will break down some of the unique challenges for circadian phase estimation with an eye towards machine learning techniques.  I’ll also do a brief review of the previous attempts to apply machine learning to this task. 

Categories
Technology

Wearable Headaches (and how to fix them)

So you want to get somebody’s internal time from a wearable…

Let’s talk about wearable data. On the one hand, wearables are an incredible innovation, allowing self-quantification and anomaly detection with unprecedented ease, at unprecedented scales. 

On the other hand, they’re a data science nightmare. Or three nightmares, really. 

Nightmare #1: All the devices are different, and you have to use different ways to get raw data off them. 

Sure, apps like Apple Health that act as clearinghouses make this easier for you. But you can’t use Apple Health for everything. Sometimes, wearables require permission to be granted for you to access their full data. Sometimes, wearable companies go out of business after you’ve built an infrastructure to work with them. 

Can you process heart rate signals from two wearables using the same algorithm? What if they decide what counts as a “step” in different ways? What if the firmware changes? People have certainly thought about these questions, and that’s the whole point: you have to think about them. The effort of keeping track of everything adds up.