I've heard that the boarding process itself is rarely the limiting factor in flights. They're usually waiting on other plane-related things (refueling? Pre-flight checks? I can't recall the details).

If it were, they probably wouldn't be doing their 8-group boarding process that takes 20 minutes just to let people start boarding, because gate-time is expensive for them.