Libellule (Dragonflies), my first poetry collection, has finally been released!
Where? On GitHub (of course) and as a PDF (link)!

I included the reasons for this publishing choice in the collection.

Libellule speaks about my view of life, my story and the path to transcendence.

If you’re curious, you can read it here,

Un abbraccio,
Adriano

#poesia #poetry

I have a strong opinion that there will not be any new scale-up without using at least a low code/no code/vibe code tools.

These allow everyone to build an MVP and validate business ideas quickly, and there is a good chunk of discussions that will go well beyond this by eating up the payroll of traditional software development.

While it is hard to neglect the impact AI will have on the industry, the way a company operates in practical terms is far from an AI agent or Make automation.

Without a shadow of a doubt, mundane tasks will be automated; however, beyond a certain company scale and for core business functions, humans will still be needed (I will explain below what I mean by that).

This is my take as a B2B product manager turned product builder in a fintech scale-up.

Key considerations I have learned on what it means to use low-code tools in a fast-growing company:

i. Low code/no code/vibe code tool is real software, and I repeat it: Low code/no code/vibe code tool is real software.

Hence, it MUST follow the same rules of traditional software development.

By saying so, I mean a clear distinction between a development and a production environment (maybe also staging?), good logging tools to debug, and security conditions beyond a checkbox in a random settings page (not to mention DB considerations and beyond).

When your organization is generating $M, you cannot just move fast and break things. That is good for a pre-seed venture, but not when your headcount crosses 50+ people, have several department,s and a workflow fails, the logging tools are limited and not meant for real debugging, you have an issue, my friend.

So cool to low/no/vibe code, so good luck to manage it afterwards...

ii. It seems easy to drag, drop, and add API Keys, AHA Moment, and it works!

...And then you may want to pass an array in a parameter and you spend an afternoon with no documentation and localtunnel to pray it will work...

That is the key point: the more complexity you are going to have in your low/no/vibe code the more hard it will be to surgically add just another piece of complexity on top of it. It is like playing shangai while your business is running.

iii. It is not about the tools but the logic… and SWEs have superpowers!

Software is software, and the way it is built always follows the same logic: frontend, backend, and database. Then you can have workflows or scenarios, but the field of logic is the same, and a software engineer will quickly pick up how to use one tool or another.

A logical mind is a logical mind at the end of days (as my colleagues once suggested)

How to embrace these tools in the right way (opinionated)

Here is what I would do if I had to start a venture from scratch and needed to use low-code/no-code/vibe code:

• A) define the fit for purpose (aka when to kill them): you can always assemble a motorbike in your garage, but if you want to play MotoGP, that's whole another story. Code is still ruling the world, and you will likely dismiss these tools unless for minimal use cases (e.g., an internal MVP). These are the expiration date upfront.

• B) Check security and user role privacy settings (also have a look at only discussions on data leaks): make sure the company's data is safe.

• C) Think like a software engineer: create a DEV and a PROD environment even in the early days of the venture. And if you like them, and your tools allow use feature flags.

• D) Manage complexity: your scenarios or workflows should be well segregated so that it is easier to debug (e.g, avoid having 60 steps with 7 API connections but define a single point of failure per workflow). You are working in a distributed system, so manage dependencies smartly (limit the number of tools).

Opinions expressed are solely my own and do not express the views or opinions of my employer.

Two years ago, I became interested in measuring aspects related to my well-being. After several iterations, I built YourOS, a system that enables users to measure 25+ variables of their well-being via a quick daily questionnaire.

I have filled out the survey consistently since 1/17/2023 - 593 days so far. The considerations I found apply not only to those who measure their well-being but also to everyone working on product analytics.

Source: YourOS Looker Dashboards (illustrative). The well-being variables are clustered into six dimensions: health, mindset, identity, relationships, financials, and growth.

Here is what I learned:

• Simplicity is key: from data entry to data visualization, the king is providing a frictionless experience and minimal lag from data entry to user value. In YourOS, it takes only 30 seconds from starting to fill in the questionnaires to getting the data plotted.

• Unexpected: when you improve on a metric, you are motivated to be more consistent. This works especially well for habits (with streaks). Conversely, when a metric is underperforming, it does not necessarily translate into action. Monitoring ≠ Action. I discussed this point in a previous article. Hence, It is not enough to provide data; there should be systems in place that drive and nudge the user to take action.

• Concrete: what you provide to the user should be of interest to them. It may sound trivial, but it will always take a few iterations to get it right.

• Credible: the user should believe in the value of the data collected. In other words, the data reported should be as close to reality as possible, even when we are trying to measure qualitative aspects of one's life. For example, the question "How satisfied are you with your current relationships?" can have a numeric rating scale from 1-5 as answer options, where each value should be clearly defined (e.g., 1 is "I feel alone," 5 is "I have a fulfilling social life"). Note: psychometrics is a complex matter.

• Emotional: what data shows should be something the user is already aware of. When things are not going well, the user may be reluctant to see the data. This ties back to having systems to nudge the user to improve. For instance, in my system, I automatically get food recommendations from my task manager for the following day if in the past three days, the perceived food quality has been below 3/5.

• Stories: Numbers are just numbers. Saying that the health value is 73% or 68% does not change much. What is relevant is to have a way to let users reflect on the data. In my example, well-being is not a function to maximize but to keep stable beyond what we perceive as feeling good. Unforeseeable events can happen in one's life, but happiness stabilizes back after a certain time. Here is an interesting reading on happiness.

Source: Hottakes

• S(tart with the end in mind): those familiar with product management books may recall the SUCCESs framework from the book "Made to Stick" by Chip Heath. I added an "S" to the model to pinpoint an important takeaway. I believe that data is a product per se. This is also why I believe data product managers play an integral role in a product team. My insight is that whenever we plan to collect and provide data to the user, we should be careful and not just consider it as an addition to the solution.

How we move from the act of collecting data to getting personalized insights and actions is where we achieve real value for the user.

About the System: YourOS

Last March, I launched YourOS, a public version of the system I use to track my well-being. Since then, YourOS has been downloaded organically ~800 times. YourOS is available for free on YourOS.me.

In May, I presented YourOS in Milan during the first Italian Product(Un)Conference.

Source: Product (Un)Conference, Milan (thanks to @RosarioMarrazzo for the invitation)

I enclose here the slides for those interested in knowing more about the story and principles behind YourOS.

Youros A Structured Approach To Manage One's Life Adryhealth

4.28MB ∙ PDF file

Download

Download

What is Next for Digital Health Product Analytics?

Nowadays, everyone uses technology to measure aspects related to their health, either via self-reported data or wearables. I see an uptick of companies active in the longevity and prevention space (e.g., Superpower).

As the amount of health data about ourselves increases, we are getting more resolution to (fully) defining a person from a quantitative standpoint (aka Digital Twin).

Digital Twins, combined with Large Language Models, can democratize predictive access to health care by providing daily support to individuals and making predictive and preventive medicine a reality. However, the traditional challenges in healthcare remain (e.g., privacy, and inclusion). I discussed this topic in a previous article.

Nonetheless, I am a strong believer that digital twins will change how we manage our health. I also think mental health is a core component of each longevity and prevention solution, which should not only consider Electronic Health Records but let the user interact with it (AI companion).

Given the type of data provided by the user (which can include very personal thoughts), I am enthusiastic about a digital-only, locally-operated, and privacy-focused system.

Who knows, maybe one day YourOS will have a cocktail with a Llama...

Source: AI-generated (DALL-E)

Deploying LLM models in healthcare requires regulatory compliance, strategic planning, and productive partnerships. How is this achieved in practical terms?

Today, I had the privilege to sit down and discuss these topics with Neel Das, LLM Tech Lead @Roche.

Neel is an AI leader who has taken an unconventional route in the healthcare industry. With over a decade of experience in machine learning, he now plays a key role at Roche in developing Gen AI solutions for their digital healthcare software offerings. 

Neel's position is pivotal in generating evidence for the safe deployment of AI models and fostering effective collaboration among data, engineering, and product teams. 

The views expressed by Neel in this article are his own opinions and observations.

Hello Neel, we appreciate your participation in the Digital Medicine Beats Newsletter. Could you tell us more about your professional path at the intersection of AI and Biomedical Sciences?

My background is quite diverse. I initially pursued a degree in ocean engineering during my undergraduate studies, followed by three years as a software developer. Afterward, I got a master's degree in a related field.

During my time pursuing my master's degree, I became increasingly intrigued by machine learning, particularly deep learning, which was just beginning to gain momentum. During that period I was also seeking a career transition to a field where I could make a meaningful impact and find fulfillment in my work.

Fortunately, I stumbled upon a PhD position for students with strong mathematical and ML expertise. The topic revolved around diagnostic AI models for healthcare. I researched diagnostic AI models and had the chance to collaborate on a startup initiated by my promoters, which eventually produced software products utilized by pharmaceutical companies.

After nearly seven years in academia, I transitioned to Roche in 2022 as a senior data scientist, where I now collaborate with various product teams.

Currently, I lead a team of NLP engineers focusing on developing LLM-based features for our external-facing products used by healthcare professionals and patients. 

How do you shape the product strategy of a new potential LLM-based solution?

While I am not directly involved in defining the product strategy, which is typically the responsibility of the product manager and business team. However, with the advent of Gen AI, it has become more important than before for the product teams to work closely with data scientists like us.

The first step typically involves product managers identifying pain points faced by users that could be addressed through LLM features. 

Subsequently, product managers collaborate with data scientists to develop features that effectively address these pain points, prioritizing initiatives based on business value and potential ROI. Product teams must work closely with data scientists to manage expectations and ensure the feasibility of proposed features, considering limitations imposed by current technological capabilities. 

So, before the PoC phase, the product and data teams build business cases to help this process.

In your experience, what are the most common evidence-generation flaws made by data scientists when deploying an LLM into production? 

Although there is no playbook, innovators should consider all evidence that supports safe deployment.

Indeed, the level of evidence required depends on how the AI model is classified by regulatory bodies. For instance, if the model is considered a medical device, extensive scientific evidence, such as clinical trials, is necessary to demonstrate safety and efficacy. This process cannot be expedited, as different classes of medical devices have varying evidence requirements. 

On the other hand, for AI products not classified as medical devices or class I devices, we need to demonstrate that these technologies are still safe and accurate. In the case of LLMs, internal validation studies are essential to demonstrate limited to no hallucinations. These studies are particularly relevant, especially in scenarios where the model's output is inputted into another system.

Specific challenges related to LLM evaluation include also the complexity of validating text outputs. Unlike traditional machine learning predictions, which can be validated quantitatively, text outputs require more nuanced validation methods. 

• Common approaches to building a hallucination framework involve assessing text eligibility using various criteria, though this can be challenging. Strategies like quantifying similarity in summarization tasks offer partial solutions but may not capture the full context of the text.

• To address these challenges, a two-pronged validation approach combining automated machine learning techniques with human validation has been effective in ensuring accuracy at scale. Continuous improvement in validation methods is sought to enhance the validation process further.

Furthermore, the abundance of intellectual property surrounding LLMs has led to varied expectations from executives, requiring effective communication and expectation management between Product Managers, Data Scientists, and stakeholders.

Do you have any best practices for collaboration between the data, engineering, and product teams? Is there something that can be scaled across other organizations?

From my experience, establishing best practices for effective teamwork can be challenging due to varying levels of understanding and capabilities across teams. There is no one-size-fits-all approach to collaboration in this dynamic environment.

There is a role called Data lead who often find themselves navigating between different responsibilities, including data science, strategy development, and stakeholder engagement. Flexibility and adaptability are key attributes for individuals in these positions as they must be able to switch tasks and responsibilities seamlessly to meet the demands of the project.

While there isn't a singular solution that can be universally applied to all organizations, the ability to be versatile, pivot between different roles, and adapt to changing circumstances is crucial for fostering collaboration and driving innovation within cross-functional teams.

Indeed, a data leader is a multifaceted role that involves not only managing datasets but also engaging with various stakeholders to address data privacy concerns and leverage advanced data strategies to gain a competitive edge in the market.

What counterintuitive advice would you give to yourself, looking back at your journey?

Reflecting on my journey, one counterintuitive piece of advice I would offer myself is the importance of balancing technical skills with soft skills.

Initially, I prioritized enhancing my technical expertise in areas like data science and machine learning. 

While this focus was valuable, in hindsight, I recognize the significance of cultivating strong people skills and communication abilities.

There were instances where I had to forego certain opportunities that required adept interpersonal and communication skills, areas I had not emphasized due to my singular focus on technical proficiency. However, as my career progressed, I found myself gravitating towards roles that necessitated strong communication and relationship-building capabilities.

Is there any point that you think would be interesting for data scientists, product managers, or strategic leaders to know?

I would like to recommend a book that many of you may have already read, "The Lean Startup." This book resonated with me due to its approach to product development, which emphasizes starting with a minimally viable product and iterating based on customer feedback. 

Coming from an academic background, I appreciated the author's insights on the iterative nature of product development and the value of quick feedback loops in improving products efficiently.

The concept of treating product development as a series of experiments, as highlighted in "The Lean Startup," is applicable not only in building products but also in research.

Before we wrap up, is there anything specific you would like to ask the community?

I'm eager to hear about the readers' experiences with how leaders at their companies perceive innovation from frontline workers and how they have successfully scaled up projects based on these experiences.

Key Takeaways:

• On Shaping a Product Strategy for LLM-Based Solutions: Product Data Manager should identify user pain points as a first step, and early involve data scientists to ensure the proposed solutions are double and impactful.

• Evidence level needed to Deploy Medical LLMs: Varying levels of evidence are required based on regulatory classification. When it comes to LLMs a two-pronged validation approach combining automated machine learning techniques with human validation has been effective in ensuring accuracy at scale. Intellectual property rights considerations cannot be overlooked as well.

• The Job of a Data leader: requires flexibility, adaptability, and the ability to pivot between roles, and establish best practices for effective teamwork.

• On iterative testing (The Lean Startup): Starting with a minimal viable product and quickly iterating based on customer feedback, is a best practice that can be applied both in product development and academic research.

• On Career Advice: soft skills are as important as hard skills. Strong communication and relationship-building can propel professional growth.

• A question for the community: What are the readers' experiences with how leaders at their companies perceive innovation from frontline workers? How do they have successfully scaled up projects based on these experiences? 

Copyright © 2024 - Adriano Fontanari

The promise of personalized medicine lies in the ability to generate robust evidence. How is evidence generated and what are the best practices of collaboration between biostatisticians, data scientists, and the engineering team?

Today, I had the opportunity to interview Eric J. Daza, a renowned biostatistician and an advocate of personalized medicine through n-of-1 trials and single-case designs.

Eric Jay's diverse experiences in academia, industry, and digital health data science provide a unique perspective on the evolving landscape of data science and evidence-based medicine.

Hello Eric Jay, we appreciate your participation in the Digital Medicine Beats Newsletter. You possess extensive expertise and achievements in the field of biostatistics. Could you tell us more about your professional path?

I started as a biology major with a focus on neurobiology in school. Towards the end of my undergraduate studies, I developed an interest in statistics. My college introduced a one-year master's program in Applied Statistics, which I successfully pursued. This program laid the foundation for my career in biostatistics, where I worked as a biostatistician (SAS programmer) at a pharmaceutical company for five years. Wanting to delve deeper into statistical concepts, I decided to pursue a doctoral program in biostatistics, which lasted for eight years. 

During this time, I got interested in the field of data science and completed a three-year postdoc, focusing on digital health data science. This journey eventually led me to my recent role in the digital health data science industry (at Evidation Health), where I found great fulfillment.

Could you provide more information about n-of-1 trials? And what motivated you to advocate for their use?

I became interested in n-of-1 trials for two main reasons. Firstly, during the completion of my doctorate, I was exploring potential applications for my background in causal inference. A prominent figure in the field, Professor Susan Murphy, mentioned n-of-1 trials during a keynote lecture, sparking my interest. 

Additionally, a close family member struggles with irritable bowel syndrome, a condition known for its idiosyncratic triggers and symptoms, making it an ideal case for n-of-1 trials. The variability in triggers and responses among individuals with conditions like irritable bowel syndrome or migraines underscores the need for personalized treatment approaches. This personalized medicine concept resonated with me, especially in cases where the average effect of a treatment may not be reflective of individual responses. 

For instance, while a specific trigger like coffee may worsen migraine severity for one person, it could alleviate symptoms for another. This variability highlights the importance of tailored interventions through n-of-1 trials. 

Inspired by these insights, I transformed my research interests into a newsletter (Stats-of-1) and Podcast to raise awareness about the significance of n-of-1 trials and single-case designs in advancing personalized medicine approaches.

Many organizations lack the internal resources (early-stage startups) or established rigorous evidence-generation practices (growth-stage startups). Which practical advice would you give to the data leaders working in these organizations?

I emphasize the importance of adhering to good statistical practices in analysis and analytics. Good statistical practice helps you prevent overfitting. This advice is particularly relevant in environments like early-stage startups or growth-stage startups where data scientists may outnumber statisticians. 

It is essential to highlight that rigor in statistical analysis goes beyond just modeling; it also involves diligent tracking of data changes and decision-making processes to guard against overfitting.

From a business standpoint, what this means is you will give your clients much more careful advice or conclusions and you won't make them overconfident and misinterpret the results.

Furthermore, it is crucial to address the false notion that statistical analysis can be treated less rigorously than engineering processes. While engineering workflows prioritize meticulous checks and testing due to the unforgiving nature of logic and computer code, statistical analysis should be approached with a similar level of diligence. The fallacy is that it's okay if the statistical analysis is more lax. And that's true, too. That's true to a certain degree. 

But just as engineering processes prioritize reproducibility, it is essential to conduct statistical analysis in a structured and monitored manner to ensure the replicability and consistency of scientific conclusions. 

Approaching statistical analysis akin to an engineering pipeline involves maintaining a structured framework while allowing for exploration, similar to the principles of the Scrum or Kanban methodologies for software development. By closely monitoring the analytics process with the same rigor as engineering workflows, organizations can ensure that their statistical analyses are replicable, leading to reliable scientific conclusions that ultimately support the business objectives.

Do you have any best practices for collaboration between the data, engineering, and product teams? Is there something that can be scaled across other organizations?

One effective best practice for improving collaboration among data, engineering, and product teams is to integrate processes into the workflow that involves the analysts, analytics engineers, data engineers, and product teams as closely as possible. It is essential to establish linkages and processes that all teams will utilize, rather than imposing a specific mindset on teams that may not align with their roles. 

For instance, incorporating tools like dbt into the analytics engineering pipeline can greatly benefit the data analysis pipeline as well. This integration enables a better understanding of the data origins of the variables and pathways to the final dashboards used by the product team. 

It also helps statistical analysis, because it helps you to monitor when you run each analysis, the data used, and the versions.

Tools such as Airflow and dbt should not be limited to data and analytics engineering but should also be implemented in data science, data analysis, and product development to enhance collaboration and efficiency across the organization. 

What counterintuitive advice would you give to yourself, looking back at your journey?

The counterintuitive advice I would give myself is to embrace the idea that career paths are often non-linear and that it's okay not to have everything figured out early on.

In today's work environment, it's common to have a more agile approach to career progression, where moving between different roles and organizations is a normal part of the process. I like the concept presented in the book: "The Startup of You", written by Reid Hoffman: adopting an entrepreneurial mindset, even without starting your own business, can be beneficial in navigating this dynamic landscape.

I think that's the counterintuitive advice I would give to younger professionals.

Before we wrap up, is there anything you would like to share with the community?

When I began my career, the concept of data science did not exist in the way we know it today. Approximately 20 years ago, I ventured into the realm of quantitative fields, particularly biostatistics. Data science was still in its infancy back then. 

As I reflect on my journey, one question that often arises in my discussions with colleagues and friends is: How should a comprehensive training program for data scientists be structured? 

Personally, I believe such a program should encompass elements of consulting, business acumen, and managerial skills, alongside a strong foundation in computer science and statistics as key pillars. I am curious to learn what perspectives the community, especially those who operate outside my statistical background, might bring to this discussion.

Key Takeaways:

• On how to generate rigorous evidence: it is paramount to adopt good statistical practices in analysis and analytics to prevent overfitting, especially in environments where statisticians may be outnumbered by data scientists. Carry out statistical analysis with the same rigor and diligence as engineering processes to ensure the replicability and reliability of scientific conclusions.

• On the Collaboration between the Data and Product Team: Improve collaboration among data, engineering, and product teams by integrating processes. Furthermore, incorporating tools like dbt and Airflow into the analytics engineering pipeline can greatly benefit the data analysis pipeline as well, as it helps you to monitor when you run each analysis, the data used, and the versions.

• On n-of-1 Trials: n-of-1 trials and single-case designs can be powerful tools to address the variability in triggers and responses among individuals with conditions like irritable bowel syndrome and migraines

• On career advice: Embrace the non-linear nature of career paths and be open to evolving roles and organizations. Adopt an entrepreneurial mindset, as suggested in "The Startup of You" by Reid Hoffman, to navigate the dynamic landscape of career progression.

• A question for the community: How should a comprehensive training program for data scientists be structured? Eric Jay suggests a comprehensive training program for data scientists should include elements of consulting, business acumen, and managerial skills in addition to a strong foundation in computer science and statistics, inviting perspectives from the community on structuring such programs.

Data practitioners interested in learning more about n-of-1 trials can read more at Eric Jay’s newsletter Stat-of-1.

Eric Jay is actively seeking his next opportunity in roles such as Associate Director of Biostatistics, Data Science Manager, and Lead Data Scientist. It is a unique chance for companies to have a renowned biostatistician in their team. More info on LikedIn. 

Healthcare has been notably slower in embracing digital innovations. What are the main obstacles and which lessons can we learn from faster-paced industries?

Today, I had the privilege of interviewing Christine Doig-Cardet, Chief Product Officer at IOMED, a startup that has created a Natural Language Processing system capable of standardizing structured and unstructured healthcare data.

Before joining IOMED, Christine had a successful career leading data science and product teams at various large consumer tech companies, including Netflix and Google.

Hello Christine, thanks for joining the Digital Medicine Beats Newsletter. You have quite a unique experience with product and data science leadership roles for Anaconda, Netflix, and Google. Could you tell us more about your professional path?

I have always had a passion for mathematics. My career began in the field of manufacturing and operations research, where I utilized data and modeling techniques to enhance processes. Over time, I transitioned across industries such as energy, entertainment, productivity, and healthcare.

I have found immense value in the cross-pollination of knowledge gained from working in diverse sectors, as it has highlighted commonalities in utilizing data and AI effectively. I have transitioned throughout my career from being a data scientist-practitioner to assuming more strategic responsibilities.

This shift has empowered me to not only implement solutions but also innovate by developing new software, integrating advanced AI models, and exploring novel applications in digital products.

Additionally, personal circumstances, such as becoming a mother, have influenced my decisions, sparking introspection on what matters most to me outside of work and shaping my evolving interests and industry preferences.

You have expertise in both developing technical and consumer-oriented products across several industries. Which aspects of your previous work experiences can be applied to your current position at IOMED, and what do you consider specific to the healthcare industry?

Healthcare data is notably sensitive and necessitates a dedicated emphasis on data privacy, governance, and ethics. While these considerations are relevant across industries, their impact on healthcare is distinct due to the critical nature of the sector.

Unlike industries that may prioritize rapid innovation at all costs, healthcare demands a more deliberate and cautious approach to safeguard sensitive information and ensure ethical practices. This slower pace also presents opportunities to learn from past mistakes in other industries, facilitating more efficient progress in healthcare.

The potential impact of AI in healthcare is significantly profound compared to other sectors. We focus a lot on the potential harms of AI and not enough on its benefits, but there are also ethical implications in being too cautious when it comes to bridging AI-based solutions that can change patient outcomes. Time is of the essence for many patients.

I propose a balanced approach that prioritizes caution without allowing fear to impede innovation, recognizing that there is an opportunity cost involved, particularly in terms of saving lives.

There is a lot of buzz around AI in healthcare. In your view, what are the current challenges limiting the adoption of AI-based solutions for healthcare providers and life science organizations? How can these challenges be mitigated?

Healthcare is probably the most complex industry, given the variety and nature of the different stakeholders. To highlight the main stakeholders, we have patients, doctors, research groups, patient advocacy groups, pharmaceutical companies, regulators, hospitals, and policymakers.

Progressing towards alignment among these diverse stakeholders, each with their own interests and viewpoints is a complex task that requires time to achieve the desired industry transformation.

Not to mention the data access challenges. While the industry generates vast amounts of data, much of it is either stored in incompatible, siloed systems or unstructured formats like free text or handwritten notes. Improving data accessibility while adhering to data privacy and governance regulations is crucial for enabling data-informed decision-making at all levels of healthcare.

It has been quite unexpected for me that there isn't as strong of a data-driven culture in the healthcare industry compared to other sectors, considering the scientific nature of the field. While people are well-acquainted with hypothesis testing and conducting studies to validate treatment efficacy are common practices, there seems to be a reluctance to move beyond the established research methodologies and use data for making a wider range of decisions. I found this phenomenon particularly fascinating.

Conversely, in consumer digital products, we have adopted their scientific approach to hypothesis testing. We have incorporated a mindset of experimentation, borrowed from clinical trials, into digital product innovation. Interestingly, the field that originated clinical trials has not extended this mindset further in the industry.

To mitigate these challenges and facilitate the adoption of AI in healthcare, efforts should be focused on improving data accessibility, fostering a data-driven culture within the industry, and promoting the use of data for informed decision-making across various healthcare functions.

What strategies can be employed to measure the effectiveness of AI-driven digital health products to enhance their acceptance rates?

It's crucial to establish our objective and the problem we aim to solve. I've noticed a tendency to misuse metrics without grasping their underlying purpose. Metrics serve as approximations, not always perfectly aligning with our true objectives. It's essential to differentiate between our goals and the metrics we use to measure them accurately.

When making data-informed decisions, there is always a degree of uncertainty involved. The level of precision required varies based on the significance and impact of the decision being made. It's crucial to consider the consequences of the decision, the context in which it will be implemented, and the objectives you aim to achieve. By taking these factors into account, you can determine the appropriate processes, metrics, and approaches to use.

Factors like company size and available resources play a significant role in determining the most appropriate approach. Another aspect to take into account is the potential ambiguity present in measurements, particularly when conveying metrics to external stakeholders. Embracing uncertainty is crucial in the process of making data-informed decisions.

For instance, when I was working in product at Google or Netflix we had highly skilled product analytics teams and robust infrastructure for experimenting and computing various metrics. Conversely, the situation is vastly different at a startup, where there is no product data analyst on the team or in-house tailored build data platform. It is important to consider the context of the company and determine the most suitable approach, but ultimately, you can adopt data culture and tailor your approach to whatever your context is.

How can organizations foster open communication and collaboration to bridge knowledge gaps and ensure effective data-driven decision-making?

We tend to be confined within our familiar roles, routines, and language, forming tight-knit groups where assumptions are easily shared and understood. However, when collaborating with individuals from different parts of the organization or external partners, our limited understanding and incorrect assumptions can hinder effective communication and hinder our ability to achieve shared goals.

And since we are hesitant to engage in open and honest discussions regarding the information gaps, our lack of knowledge, or the assumptions we bring to the conversation, we fail to address these important aspects.

We face miscommunication scenarios where the meaning I assign to a data point may differ from the interpretation you derive from it. This disparity often results from discussing these data points using varying terminology within our respective fields.

I believe the critical piece lies not in technology or processes, but in collectively defining concepts with clarity and explicitness. Often, what we assume to be a shared understanding may not be as clear-cut as we think.

And suddenly, I realize why this isn't feasible. I now comprehend the reason behind this limitation, as I had been basing my understanding on assumptions about what you were seeking from this data point or what was accessible in your database. It all boils down to clear, open, and honest communication, being unafraid to inquire about the unknown, or challenging the assumptions we hold.

What are the potential challenges and best practices for validating and testing ML models in the real world?

I observed that while there is a wealth of resources on testing algorithmic outcomes offline, there is limited guidance on validating results online, in production.

I was surprised to see that many startups or AI teams tend to deploy a single model in production without thorough validation. They make improvements offline and promptly deploy them without double-checking.

Based on my experience at Netflix and Google, where models are tested in a live environment, it's not as simple as blindly trusting your offline experiments to deploy a new or improved model.

Real-world implementation can lead to unexpected results, highlighting the importance of comprehensive testing and validation of models in an online environment. I suggest readers delve deeper into this area to improve their understanding of ensuring model effectiveness in real-world applications.

Is there any point that you think would be interesting for data scientists, product managers, or strategic leaders to know?

When it comes to giving advice, I believe it's crucial to emphasize the importance of industry expertise alongside technical skills. Understanding the context of an industry is just as vital as mastering technical knowledge.

I encourage everyone to engage with experts and users, delving deep into their perspectives to broaden understanding.

Follow your curiosity and explore niche intersections, such as AI in healthcare, where opportunities abound due to the limited presence of experts.

I suggest everyone maintain a realistic perspective during this AI hype period, manage expectations, and advocate for education on the technology's capabilities and limitations.

By approaching AI with humility, we can foster a more realistic and accepting environment for its adoption.

Personally, as a newcomer to the healthcare sector, I am eager to witness the impact that AI and innovative products can bring to the industry.

Key Takeaways

• On the Barriers to AI Adoption in Healthcare: Challenges such as limited data access, multistakeholder environment, interoperability issues, and a lack of data-driven culture in healthcare despite its inherently scientific foundation, are the key factors that hinder the widespread implementation of AI.

• On building Effective Product Measurement Strategies: Understanding the purpose behind metrics, embracing ambiguity, and aligning data analysis with organizational context is crucial for measuring the effectiveness of AI-driven products accurately.

• On Communicating Data Effectively: clear, open, and honest communication is essential for bridging knowledge gaps and ensuring effective data-driven decision-making across diverse stakeholders and roles. Mutual understanding is enhanced by confirming assumptions, as individuals may rely on various professional lexicons.

• On Real-world Validation of ML Models: comprehensive testing and validation in online settings is paramount to ensure model effectiveness in practical applications.

• On Career Advice: industry expertise is as important as technical skills. Furthermore, continuous engagement with experts and users is key to building empathy. Finally, product data managers and data scientists should manage expectations during the AI hype period, and approach AI adoption with humility.

Copyright © 2024 - Adriano Fontanari

Physicians typically dedicate approximately 40% of their working hours to documenting patient information, leading to significant burnout. Nabla is a leading Generative AI startup that addresses this issue by automatically generating clinical notes during patient visits. 

Nabla is also a rare example of Product-Led Growth in the healthcare industry. 

Today, I had the opportunity to interview Chloé Simpson, the Head of Business & Strategy at Nabla. Chloé, a data scientist turned strategist, shared insights about her professional journey and best practices in the field of product data management.

Hello Chloé, thanks for joining the Digital Medicine Beats Newsletter. You have quite a unique experience starting as a data scientist and moving towards strategy. Could you tell me more about your professional path?

Hello, I'm Chloé Simpson, currently the Head of Business and Strategy at Nabla. My role revolves around understanding the needs of our customers, who are primarily large health systems in the US. I work closely with them to identify the best ways to implement our AI solutions with their clinicians.

I'm an engineer by background, I hold a Master's of data science from Stanford University, where I discovered and applied data science and AI to healthcare, by working on healthcare data from Stanford Hospital. This is really how I began my career in healthcare.  I then joined Owkin, a startup that is applying machine learning and technologies from medical research to drug discovery, and later collaborated with the French Red Cross. Finally, I joined Nabla to continue working on healthcare, this time on the strategic side.

I transitioned from a highly technical data science role to gradually taking on less technical responsibilities. I found myself increasingly drawn to working on the strategic aspects of the product development lifecycle, as I gained insights from these diverse experiences.

I would say having a T-shaped background is a superpower for an engineer.

My background allows me to understand the intricacies and potential of AI technology, and I can communicate its value to our customers in a language they understand. This helps us ensure that our solutions align with their goals and requirements.

How can evidence and trust be generated in new AI solutions, which are crucial for innovative digital health products?

The whole point of generating evidence is to ensure the accuracy of an AI model. With Nabla Copilot, we automatically generate the transcript and note of the clinical encounter. One of our main focuses is ensuring the accuracy of the transcript since that's where most of the evidence lies. We currently use a mix of open-source models like Whisper or GPT-4 by OpenA, and our own algorithms, fine-tuned to the field of medicine, to achieve the best accuracy. We have developed our own evaluation framework for three reasons: First, to make sure there are no hallucinations in the note. Second, to test different models, and third to provide the most sophisticated product (more info on Nabla.com).

Overall, accuracy is paramount to AI applications in healthcare. We are mindful that clinicians have limited attention spans, so our product needs to be reliable and deliver on our promise to save them time. 

Additionally, generating robust evidence to showcase the impact of the solution. We recently published a manuscript on the impact of Nabla on the clinician workflow (Tierney et al., 2024).

Do you have any best practices for collaboration between the data and product teams? Is there something that can be scaled across other organizations?

Well, I think our team is focused on collaboration. What's even more important is that we include clinical expertise in the process of building our product.

One crucial aspect is ensuring that data scientists have regular exposure to clinicians. To achieve this, we have implemented two best practices:

• External Validation: we allow clinicians to send feedback on Nabla Copilot, which we use to improve our models. Understanding the feedback from clinicians is vital in building the best product. Over the past six months, we have received over 6000 feedback submissions from clinicians, which is a significant number.

• Internal Validation: the second point is working with clinical advisors. Our Chief Medical Officer and Clinical Director are crucial in shaping Nabla Copilot. Recently, we added coding automation, to streamline the assignment of codes and the billing process., after the visit had taken place. Our machine learning and product team have regular meetings and touchpoints with our executive clinical team to better understand how to implement different features. This way, we make sure we do not waste time going in the wrong direction. Having this clinical exposure is important and has proven to be valuable in our collaboration efforts.

What strategies do you employ to enhance the adoption and usage of your product in the healthcare industry?

Our product-led growth methodology is an aspect worth mentioning, especially in the healthcare industry. It allows users to immediately access our product and start getting value from utilizing our Nabla Copilot. Although this approach is not commonly seen in healthcare, it has been instrumental to Nabla in building a large clinician community.

When it comes to selling to health organizations, such as health systems, the process is different, as it usually involves piloting the product first with a small group of physicians. During this pilot, the adoption of our product is tested. To increase the chances of clinicians embracing our product, we work on building the features that suit their needs and clinical workflow.

To provide examples, we place great emphasis on developing features that allow clinicians to personalize copilot. They can customize instructions in their templates and adapt them based on their preferences. Our goal is to provide the most personalized experience possible.

By focusing on boosting product-led growth, we simultaneously increase the chances of adoption during the pilot phase in health systems. This is because early users who discover the product on their own become our best ambassadors and push for quick organization-wide deployment. This alignment of strategies is crucial.

In the process, we focus on consultations per month or week as the main metric. This allows us to track the way our newly developed features boost the adoption and usage of our technology.

How do you track and analyze user analytics and generate evidence? Are there any specific tools that you use?

We have been using Mixpanel to track and analyze user metrics. 

Every Monday, during our product team meetings, we review the metrics on Mixpanel to assess our progress and determine our priorities for the week. 

Additionally, we have executive meetings where we also analyze the metrics to gain insights into our overall strategy and project performance. 

On a monthly basis, we hold all-hands recap sessions that include a component of metrics analysis. 

So, while we check the data every day on Mixpanel, we have more comprehensive reviews on a weekly and monthly basis.

Is there any point that you think would be interesting for data scientists, product data managers, or strategic leaders to know?

I've always been fascinated by the intersection of healthcare and AI. Reading about advancements in healthcare and how AI can make a positive impact has shaped my passion. So, I would suggest diving into the world of healthcare and AI to discover the exciting possibilities it holds.

I believe it is crucial to validate your work in clinical settings and to push for real-world user feedback. 

It's not just about publishing articles or case studies in prestigious medical journals. It's about deploying your solutions in healthcare organizations and gathering metrics on their actual usage. Seeing the impact of your work in practice is incredibly rewarding and helps align everyone within the company. 

The data science team wants to achieve the best performance, the product team wants to see positive metrics, and the strategy team wants to leverage this data to inform their decision-making. 

So, my advice would be to prioritize obtaining these metrics as soon as possible. It's a cool and valuable thing to do.

What counterintuitive advice would you give to yourself, looking at your journey five years ago?

 I think the counterintuitive advice I would give to myself five years ago is not to be afraid of switching careers and jobs. 

Initially, I believed I needed to start in one direction and become an expert in that field. However, over the years, I've realized the value of having an overview of different aspects and job opportunities within a company. 

 By exploring different roles, I have gained valuable insights and experiences. So, my advice would be to embrace the idea of switching careers and exploring different job opportunities.

Before we wrap up, is there anything specific you would like to ask the community?

Yes, we are currently looking to hire talented individuals in sales, and tech roles. Additionally, we are actively seeking more clinicians and health systems to utilize Nabla. 

Key Takeaways

• On Product Metrics (the sooner the better): implementing a way to measure analytics is fundamental, it ensures strategic alignment and informs the product strategy

• On the Collaboration between the Data and Product team: data serves as the connecting thread within the organization in two ways:

  • everyone can access up-to-date product metrics through a dedicated dashboard; 

  • there are regular meetings (on a weekly and monthly basis), at various levels of the organization to discuss the key metrics.

• On the Collaboration between Clinicians and the Product/Data Teams: including clinical expertise in the process of building healthcare products is crucial. Clinicians look for custom solutions that adapt to their workflow not vice versa. Nabla implements external validation (via user-reported feedback) and internal validation (via Clinical advisors) practices to ensure feedback from clinicians is incorporated into their product development. Data scientists have regular touchpoints with the clinical team.

• On Product-led growth: PLG in healthcare (when applicable) has the potential to foster a sense of community and trust surrounding a new solution, thereby eventually driving its adoption. Additionally, it serves as a crucial means of gathering real-world user feedback, which can be utilized to enhance the AI model and further advance commercial opportunities in the future.

• On Career advice: data scientists should not be afraid to switch careers and explore different departments. A T-shaped profile is a superpower!

References:

• Tierney et al., (2024): “Ambient Artificial Intelligence Scribes to Alleviate the Burden of Clinical Documentation.” URL: https://catalyst.nejm.org/doi/full/10.1056/CAT.23.0404?query=CON&cid=DM2321351

Copyright © 2024 - Adriano Fontanari

Startup Weekend (2013) - Brescia (IT)

Over the next few years, I traveled to 8 countries and 2 continents. During that time, I studied and worked with different healthcare companies. These companies included startups, corporations, VC firms, healthcare providers, and non-profits.

Every time was a humbling experience packed with valuable learnings. I outline the most significant ones in this blog post.

---

My 7 key learnings for innovators

1) The things you do affect others' lives, and this effect isn't always measurable

The story is the one of a cold winter day in which I was visiting a loved one. The palliative care nurse came for a routine care visit and started using the software I co-designed (she was not aware of it).

In this situation what you want for your dear one is an efficient tool that reduces screen time and increases the the opportunity for direct communication between the nurse and the patient.

No industry can offer an opportunity to improve people's lives like health care. At the same time, the quality of digital health software cannot be fully captured by a KPI, the only way to understand its real impact is to see how the software is used in the real world.

2) The Missing Chair (for the Patient)

SJD Barcelona Children's Hospital (2019) - Barcelona (ES)

Jeff Bezos used to leave an empty chair in the meeting room. The chair symbolizes the customer not being present during decision-making. I feel like it should be the same in healthcare.

Despite there has been uptake of user-centered design in everyday product development efforts, people tend to forget that the place where healthcare products are built is far from where healthcare happens.

This is why having product management processes with several touch-points with patients across the continuum is paramount. Instead of showing wireframes and asking for feedback, product management teams should visit hospitals regularly, observing and talking with doctors, nurses, and other healthcare professionals.

(Biodesign) Need Finding Workshop - StarShip Health Innovation Fellowship @IESE Business School (2019) - Barcelona (ES)

I believe It should be mandatory for people working on digital health solutions to have regular (and mandatory) visits to a GP or Hospital.

3) You should publish in peer-reviewed journals to work in Digital Health Care

This is a bit opinionated but I see it as a fundamental step to appreciate how evidence is generated. Working with senior research and publishing in peer-review journals, helped me understand the time it takes to generate good-quality research.

Research has no deadlines. Conversely, startups by definition are meant to grow and can be acquired, merged, and dismantled even before the pilot paper of a technology has been published.

4) Best products do not (always) win...and most innovations are incremental (sorry for that)

Developing a 10x tech product does not set a company up for success. This is a harsh reality of health care than other industries.

Getting Product-Market Fit in Digital Health means figuring out the stakeholder alignment. The payer may only see a valuable solution for a doctor as a cost.

My recommendation here is two-fold: i. map out the whole user journey (and all stakeholders involved), ii. understand the culture and barriers to adoption, and connect the dots with a payer. Getting stakeholder alignment right is probably the most important thing to do in healthcare innovation.

5) You will soon realize data is the bottleneck of every digital health innovation

"All roads lead to...the EMR Provider" - Every digital health innovator

You may want to have quick access to longitudinal patient data. The reality is data is fragmented across different systems and providers and you will start learning about the importance of data interoperability. Not to mention data quality issues but this would be a topic for another discussion.

My key takeaway is that existing data bottlenecks in health care are not a bug but a feature of the system. From the innovator's standpoint, I highlight the importance of making the solution interoperable by design. So that it will not create another "data roach motel,"[1] (data can enter but not leave the solution) but will make the innovation an integral part of the system.

Native integrations with leading solutions and a fully interoperable platform are overlooked sources of competitive advantage.

6) A beginner's mindset, teamwork, and community are everything

EIT Health Innovators Community Meeting - Vienna 2022 - (GER)

No matter your experience, each new digital health project feels like starting from scratch. Each innovation has its specific needs and requires different expertise to work together.

Alongside the chair of the patient, innovators need to let empathy enter the meeting room and leave ego outside it.

It is seldom the case a single individual has a full understanding of the all components of the solution. This is why effective digital health innovators need to speak different languages (here I mean the clinical, design, engineering, regulatory, business, and data ones).

Indeed, Innovators are all part of the same big community, working tirelessly to build valuable solutions to improve people's lives.

Digital Health Today Ambassadors @Frontiers Health (2019) - Berlin (GER)

7) Play only long-term games

“The days are long but the decades are short” - Sam Altman

No quotes can probably better summarise what it means to work in digital health.

People working in this industry bear the burden that the impact of the solution may be seen one day, and that day can be far from today.

Patience is a necessary skill to work in the industry, there are no shortcuts to building great digital health products. You get what you put in.

The Startup Mantras (e.g. fake it until you make it; move fast and break things) do not work in the industry. Remember Theranos?

Additionally, the SaaS exponential growth rat rate myth of the 2010s is over (and most likely it has never started in healthcare).

This means embracing the reality.

For instance, B2B Sales cycles are often long (at least 6-8 months) and tech will hardly disrupt healthcare to replace services. Conversely, I believe in the importance of a service component to ensure solutions are meaningfully bridged to the system. This can also be a key component to sustain the business (making money only from tech is hard).

Playing long-term games means truly "caring" about the issues of the healthcare systems and embracing all the hurdles and struggles it takes to solve them.

Startups may not make it but the potential impact on patients is worth the journey.

Talk: "Innovation in Healthcare - From Tech Push to Need Pull" @Forum Sistema Salute (2019) - Florence (IT)

After 10 years, what's next?

Many things can be said about how I saw evolving digital health in the last decade.

10 years ago the US route was the only way to build a successful digital health business. Now that digital health become mainstream and an integral part of everyone's lives), Europe has also become a mature market to consider.

I would like to close this reflection article by saying thank you to all the AMAZING people (colleagues, friends, mentors, leaders) who shared a part of the journey.

What is next about me? I have always seen myself as an entrepreneur/intrapreneur. I will continue supporting companies building great digital health products and also giving back to the community my learnings.

P.s. A final message to current and aspiring innovators: digital health innovation is fun!

MIT-HMS Healthcare Innovation Bootcamp (2019) - Boston (US)

---

References

- [1]: Neff, Gina, and Dawn Nafus. _Self-Tracking_. The MIT Press Essential Knowledge Series. Cambridge, Massachusetts: The MIT Press, 2016.

---

Copyright © 2023 - Adriano Fontanari

Do I need to be a doctor to build a successful health tech firm?

Should I invest in this health tech startup founded by a serial entrepreneur?

Should I trust this company for a long-term strategic partnership?

This report is an attempt to answer these questions and understand the determinants of startup success (IPO / M&A) in the health tech sector, by investigating the role played by firm and founder characteristics.

The research is based on 921 health tech companies founded between 2008-2010 and performing a 10-year cohort analysis.

The TL;DR answer is that the determinants of startup success in health tech are:

- the founder's experience (n. of years) in the healthcare sector

- the presence of a corporate investor

- the growth stage of the company: the more mature the business is (early and late-growth vs seed stage), the higher the probability of being successful).

#healthtech #digitalhealth #vc #research #funding

Access the full report here: https://adry-health.com/adry-health.com/resources/determinants-of-startup-success-adryhealth.pdf

Beyond this apparently simple yet powerful act of self-tracking lies the future of health care. Indeed, the wide adoption of digital health technologies is changing how we manage our health. Since the 2010s individuals have started taking a more and more role in managing their health.

In this article, I discuss how digital Twins could improve doctor-patient relationships by facilitating more informed decision-making, providing personalized insights to patients 365/7, predicting the most effective therapy, and becoming an effective prevention tool (an area for which it is hard to build a sustainable digital health business).

Read more on: https://adry-health.com/adry-health.com/articles/Crossing+the+Chasm+of+Quantified+Health+-+Towards+LLM-enabled+Digital+Twin

#LLM #digital-twins #digital-health #quantified-self #quantified-health #AI

Read the full article: [Karger Digital Biomarkers]

#innovation, #strategy, #product marketing, #venture capital, and beyond

Subscribe now