How does a gui detect whether the mouse cursor has become inside the widget, such as the rounded rectangle button, the normal rectangular button, the ellipse, the circle, and a graph under a curve...etc?
I know about some possible algorithms such as determining the widget's coordinates and determining its area, mathematically, or using the widget's color so that the event is triggered when the mouse pointer is on the widget's color, but I think the actual method used in common GUIs is different from that.
Really old-school method (win16, Win32, omitting the modern apis): Windows has a structure called Region which supports union, intersection, diff, etc. Region was used for hit-testing, marking portions of windows which needed repaint, etc. It's basically an ordered array of rows, where each row has a starting y position and height. Each row is an array of columns, where each column has a starting x position and width. The whole region was packed into a single contiguous allocation. Hit testing is by binary search.
A window had a set of child windows (hierarchical). A control was a subtree. Each window maintained (or could compute) its bounding region. Hit testing flowed from parent to child. Since most child windows were rectangular, it wasn't as expensive as it sounds, but could handle, at higher memory and computational cost, non-rectangular controls.
You could gain speed (and simpler implementation) by using a bitmap instead of a region structure. If you switch to GPU, several faster methods become possible.
How do I use bitmap for that?
Approach 1: each control has a separate monochrome bitmap. 1 for inside, 0 for outside.
Approach 2: one 32-bit bitmap for the window. Assign a different color to each control.
Is this method related to render pass and offscreen buffer?
I see some guis use a single color for all elements, this makes me think that different colors can be used for each widget as a bottom layer, while the actual color of the element that we want to show to the user is on the top layer.
Is my guess correct?
Yes, a separate render pass which produces an off screen buffer.