Scrolling will generally be fine since it is usually handled off-thread for most browsers as long as there are no active scroll events attached. Clicking should be delayed but not necessarily for the full duration of the red (though the first one looks like a single executing JS function so it would be for the full duration of that block).
The red blocks are any point where the main thread is blocked (including from input) for 100ms. A lot of separate 100ms events can make it look like a long block or a single really long event can also show up (which could be a way worse experience).
How did you measure being able to click on a link during one of the red blocks? You’d have to have a test machine handy with a similar CPU and connectivity and then try to click in the UI immediately after it starts to render.