5.0. Bayes’ Theorem

While Bayes’ Theorem is traditionally a part of Probability and Statistics, it often finds mention in mathematical methods courses—especially in engineering—due to its critical applications in prediction, signal processing, machine learning, and even solving inverse problems related to partial differential equations (PDEs). Understanding Bayes’ Theorem equips students with the tools to update the likelihood of a hypothesis based on new evidence, which is particularly useful in computational applications involving uncertainty. This topic bridges the gap between deterministic PDE models and probabilistic inference models, offering a deeper insight into decision-making under uncertainty.

Bayes' Theorem is essential in various fields, particularly in engineering and statistics. It allows practitioners to revise their beliefs about probabilities when new information becomes available. This capability is vital in complex fields like machine learning and signal processing, where uncertainty frequently arises. By understanding Bayes' Theorem, students can learn how to effectively manage this uncertainty and make better-informed decisions, which is particularly important when dealing with models that have probabilistic elements.

Imagine a doctor diagnosing a rare disease. Initially, the doctor knows that only 1% of the population has this disease (prior probability). After conducting a test, the doctor learns that the test is positive. Instead of simply assuming the patient has the disease just based on the test, the doctor uses Bayes' Theorem to reassess the likelihood, factoring in the accuracy of the test and the overall prevalence of the disease. This process ensures that decisions are made on a solid statistical basis rather than just initial intuitions.

Before delving into Bayes' Theorem, let’s recall a few fundamental probability concepts: • Sample Space (S): Set of all possible outcomes. • Event (E): A subset of the sample space. • Conditional Probability: 𝑃(𝐴∩𝐵) 𝑃(𝐴|𝐵) = , provided 𝑃(𝐵)> 0 𝑃(𝐵)

To effectively understand Bayes' Theorem, it's important to have a strong grasp of basic probability concepts. The sample space is the collection of all possible outcomes in a given situation. An event is any specific outcome or set of outcomes within that sample space. Conditional probability is a key concept, defined as the likelihood of an event occurring given that another event has occurred. For example, if we're considering the probability of rain given that it's cloudy, the probability is calculated using conditional probability.

Consider tossing a coin. The sample space consists of two outcomes: heads or tails. If someone tells you that they got heads, then the event you’re interested in is getting heads. If we know it’s a fair coin, the conditional probability of getting heads after observing it's a fair coin remains 50%. This understanding of basic probabilities sets the foundation for diving deeper into Bayes' Theorem.

Bayes’ Theorem describes the probability of an event based on prior knowledge of conditions that might be related to the event. Formula: 𝑃(𝐵|𝐴 )⋅𝑃(𝐴 ) 𝑖 𝑖 𝑃(𝐴 |𝐵) = 𝑖 ∑𝑛 𝑃(𝐵|𝐴 )⋅𝑃(𝐴 ) 𝑗=1 𝑗 𝑗 Where: • 𝐴 ,𝐴 ,...,𝐴 : Mutually exclusive and exhaustive events 1 2 𝑛 • 𝐵: An event whose probability depends on the occurrence of 𝐴 𝑖 • 𝑃(𝐴 ): Prior probability 𝑖 • 𝑃(𝐵|𝐴 ): Likelihood 𝑖 • 𝑃(𝐴 |𝐵): Posterior probability 𝑖

Bayes' Theorem mathematically expresses how to update the probability of an event based on new evidence. The formula can be broken down as follows: the left side shows the posterior probability of A given B, while the right side consists of the likelihood of B given A multiplied by the prior probability of A, normalized by the total probability of B across all events. The theorem incorporates prior beliefs and adjusts them in light of new evidence, allowing for a more nuanced understanding of probability.

Think of a detective solving a crime. Initially, they have certain assumptions about who the suspects are (prior probability). As new evidence comes in—like fingerprints or alibis—the detective assesses the probability of each suspect being guilty based on this evidence (likelihood). The detective then updates their beliefs about the suspects after evaluating all this new evidence, similar to how Bayes’ Theorem updates probabilities.

Let: • 𝑃(𝐴 ∩𝐵) = 𝑃(𝐵|𝐴 )⋅𝑃(𝐴 ) 𝑖 𝑖 𝑖 • From the total probability theorem: 𝑛 𝑃(𝐵)= ∑𝑃(𝐵|𝐴 )⋅𝑃(𝐴 ) 𝑗 𝑗 𝑗=1 Now: 𝑃(𝐴 ∩ 𝐵) 𝑃(𝐵|𝐴 )⋅𝑃(𝐴 ) 𝑖 𝑖 𝑖 𝑃(𝐴 |𝐵) = = 𝑖 𝑃(𝐵) ∑𝑛 𝑃(𝐵|𝐴 )⋅𝑃(𝐴 ) 𝑗=1 𝑗 𝑗

The derivation starts with understanding the relationship between joint probabilities and conditional probabilities. The joint probability of A and B occurring together can be expressed in terms of conditional probability. By applying the total probability theorem, we can derive the general formulation of Bayes' Theorem. This derivation highlights the mathematical foundation of the theorem, showing how it combines different probabilities to yield a new understanding of event likelihoods.

Imagine you're baking cookies and trying to determine the likelihood of having chocolate chips (event A) and nuts (event B) in the cookies. The joint probability would involve knowing both ingredients are present together. The overall likelihood of having cookies with nuts would involve considering all possible cookie recipes (sum of joint probabilities from all recipes). Deriving Bayes' Theorem is like systematically sorting through recipes to determine your likelihood accurately, taking into account conditions like whether chocolate chips are in the mix.

• Prior Probability 𝑃(𝐴 ): Our belief in event 𝐴 before evidence. 𝑖 𝑖 • Likelihood 𝑃(𝐵|𝐴 ): How probable is the evidence 𝐵 given that 𝐴 is true. 𝑖 𝑖 • Posterior Probability 𝑃(𝐴 |𝐵): Updated belief in 𝐴 after observing 𝐵. 𝑖 𝑖

Understanding the various components of Bayes’ Theorem is crucial for its application. The prior probability refers to what we believe about an event before seeing any new information. The likelihood indicates how probable observed evidence is under the scenario that our hypothesis is true. Finally, the posterior probability reflects our updated belief after taking the new evidence into account. Each term plays a pivotal role in how we refine our assumptions.

Think of a weather forecaster predicting if it will rain (event A). Before looking at any data (prior probability), they might think it’s likely to rain if it was rainy yesterday. When newly gathered evidence comes in, like specific weather patterns (likelihood), they adjust their forecast based on this new information. Ultimately, after all factors are considered (posterior probability), they present a more accurate prediction of rain.

Example: Suppose a disease affects 1% of the population. A diagnostic test has: • True Positive Rate = 99% • False Positive Rate = 5% Find the probability that a person actually has the disease given a positive test. Let: • 𝐷: Has the disease • 𝐷′: Does not have the disease • 𝑇: Test is positive Given: • 𝑃(𝐷) = 0.01, 𝑃(𝐷′)= 0.99 • 𝑃(𝑇|𝐷) = 0.99, 𝑃(𝑇|𝐷′)= 0.05 Now, use Bayes’ Theorem: 𝑃(𝑇|𝐷) ⋅𝑃(𝐷) 𝑃(𝐷|𝑇) = 𝑃(𝑇|𝐷)⋅𝑃(𝐷)+𝑃(𝑇|𝐷′)⋅𝑃(𝐷′) 0.99⋅0.01 0.0099 0.0099 = = = ≈ 0.1667 0.99⋅0.01+ 0.05⋅0.99 0.0099+ 0.0495 0.0594 Conclusion: There’s only a 16.67% chance the person actually has the disease even after testing positive.

This example illustrates how Bayes’ Theorem can lead to counterintuitive results. Despite the test having a high true positive rate, the overall chance that a person has the disease given a positive test is still relatively low because of the disease's rarity in the general population. The application of the theorem involves determining the relevant probabilities and accurately carrying out the calculations. It emphasizes the importance of considering both rates—truthful positives and false positives—when making medical decisions.

Imagine you have a very rare coin. When flipped, it shows heads 99% of the time if it’s real but also shows heads 5% of the time if it’s fake. You find heads when flipping; most people would assume it's the rare coin. However, using Bayes' theorem helps you realize that the likelihood of it being the rare coin is much less because there are so many more fake coins. This unexpected probability helps shape better judgement and decision-making.

Bayes’ Theorem is applied in various real-world and PDE-related contexts:

1. Signal Processing: Estimating noise-reduced signals using Bayes’ filter.
2. Inverse Problems: Reconstructing unknown sources or conditions in a PDE from observed data using Bayesian Inference.
3. Machine Learning: Algorithms like Naive Bayes classifiers and probabilistic models.
4. Structural Reliability: Estimating the likelihood of system failure under uncertainty.
5. Medical Imaging (PDE + Bayes): Inferring organ boundaries or tumor locations from PDE-modeled signals like MRI.

Bayes' Theorem has a wide range of applications across industries and research areas. In signal processing, it enhances data quality by filtering out noise, improving the accuracy of received signals. In mathematics and physics, Bayesian inference helps solve inverse problems, where one infers hidden parameters from observable data. Machine learning employs Bayes' Theorem in various models, enriching the capabilities of classification algorithms. In structural engineering, it assists in predicting the reliability of systems, while in medical imaging, it offers insights into recognizing anatomical structures from complex signal data. Its versatility makes it a crucial concept across fields.

Consider a safety engineer evaluating a bridge's integrity. Using Bayes' Theorem, they could assess signal data from stress tests (like vibrations and deformations) to determine the likelihood of a structural failure—balancing observed data against historical performance of similar structures. In this way, Bayes' theorem helps ensure public safety while enhancing engineering decision-making.

In the continuous domain, the formula becomes: 𝑓 (𝑏|𝑎)⋅𝑓 (𝑎) 𝐵|𝐴 𝐴 𝑃(𝐴|𝐵) = 𝑓 (𝑏) 𝐵 Where: • 𝑓 : Conditional density 𝐵|𝐴 • 𝑓 : Prior density 𝐴 • 𝑓 : Marginal density of evidence 𝐵 This is widely used in Bayesian statistics and data assimilation techniques in simulations involving PDEs.

In cases involving continuous random variables, Bayes' Theorem adapts to incorporate probability densities instead of discrete probabilities. The formulas represent the relationship among conditional densities, prior densities, and marginal densities. Understanding this extension is crucial because many applications, especially those involving physical measurements, rely on continuous data. Bayesian statistics uses these concepts to effectively update beliefs based on new data points in a continuous framework, further bridging statistical inference with real-world data modeling.

Think of measuring the height of plants in a garden. If you initially assume that the height is normally distributed (prior density), when you start measuring them and get actual data points (evidence), you can update your beliefs about the average height, which can be reflected as a conditional density. The process of continually adjusting your height expectations based on new measurements showcases the essence of Bayes’ theorem in action in a continuous setting.

Bayes’ Theorem is a cornerstone of probabilistic inference and is especially powerful in decision-making under uncertainty. It enables us to update prior beliefs in light of new evidence and is vital in fields ranging from machine learning to inverse problems in PDEs. Mastery of this theorem enhances analytical thinking and offers a probabilistic framework to approach real-world and computational challenges in engineering.

Bayes’ Theorem stands out as fundamental in statistics and applications like machine learning, medicine, and engineering. Its capacity to update beliefs as new information emerges is a game-changer in decision-making processes, particularly where uncertainty is involved. Understanding and applying this theorem will develop students' analytical skills and offer insights into how to manage uncertainty in a structured way, ultimately preparing them for real-world challenges.

Consider a stock market analyst who uses past market behavior (prior beliefs) to predict stock performance. As real-time data flows in—such as financial reports and economic fluctuations—the analyst revises their predictions using Bayes' Theorem, leading to better investment decisions. This adaptability emphasizes why mastering such a theorem is crucial for performance across various fields, especially in today's data-driven world.

--