Regular Expressions are a concise and powerful formal language for describing patterns of characters. They are the standard notation for specifying the structure of tokens for a lexical analyzer because the patterns for tokens (like identifiers, numbers, or keywords) are inherently "regular."

A regular expression defines a regular language, which is a set of strings that match the pattern described by the RE.

Formal Definition (Simplified):
Regular expressions are defined recursively using a small set of basic operations:
1. Empty String (epsilon): epsilon is a regular expression, denoting the language containing only the empty string. L(epsilon)=epsilon.
2. Symbol (a): For any symbol a in the alphabet Sigma, a is a regular expression, denoting the language containing only the string a. L(a)=a.
3. Concatenation (RS): If R and S are regular expressions, then RS is a regular expression, denoting the language L(R)L(S).
- Example: (a)(b) or simply ab matches "ab".
4. Alternation (Union) (R∣S): If R and S are regular expressions, then R∣S is a regular expression, denoting the language L(R)cupL(S).
- Example: a | b matches "a" or "b".
5. Kleene Star (Zero or More) ($R$): If R is a regular expression, then $R$ is a regular expression, denoting the language $L(R)$, which is the set of strings formed by concatenating zero or more strings from L(R).
- Example: a* matches {"", "a", "aa", "aaa", ...}.
6. Parentheses (for Grouping): Used to group subexpressions and define the order of operations.

Additional Common Shorthands (Syntactic Sugar):
- Kleene Plus (R+): Matches one or more occurrences of R. Equivalent to $RR$.
- Example: a+ matches {"a", "aa", "aaa", ...}.
- Optional (R?): Matches zero or one occurrence of R. Equivalent to R∣epsilon.
- Example: (.)? matches "" or any single character.
- Character Classes ([ ]):
- [abc] matches 'a', 'b', or 'c'. Equivalent to a | b | c.
- [a-z] matches any lowercase letter.
- [0-9] matches any digit.
- [A-Za-z] matches any uppercase or lowercase letter.
- [^a] matches any character except 'a'.
- Any Character (.): Matches any single character (except newline, typically).
- Escaping Special Characters (\): To match a character that has special meaning in REs (like , +, ?, ., |, (, ), [, ], \), you must escape it with a backslash. E.g., \. matches a literal dot.

Examples of Regular Expressions for Common Tokens in Programming Languages:
- Integer Literals: One or more digits.
- [0-9]+ matches: 0, 123, 98765
- Floating-Point Literals: Digits, optional decimal point and more digits, optional exponent.
- [0-9]+\.[0-9] (e.g., 123., 123.45)
- \.[0-9]+ (e.g., .45)
- ([0-9]+\.[0-9]|\.[0-9]+)([Ee][+-]?[0-9]+)? (More complete for scientific notation)
able matches: 3.14, 10., .5, 1.2e-3, 1E+5
- Identifiers: Starts with a letter or underscore, followed by zero or more letters, digits, or underscores.
- [A-Za-z_][A-Za-z0-9_] (e.g., my_variable, count, _temp, x1)
- Keywords: Specific fixed strings.
- if, else, while, for (Each keyword is its own regular expression).
- Operators: \+\+ (for ++ increment operator)
- >=(for greater than or equal)
- ==(for equality comparison)
- String Literals: A sequence of characters enclosed in double quotes. The characters inside can be anything except a double quote, possibly with escape sequences.
- "([^\"\]|\.)" (A simplified version; real string literals handle more escape sequences) Matches: "hello world", "A quote: \"".

The process of defining token patterns with regular expressions is often the first step in designing the lexical analysis phase of a compiler.