Whenever ladders are skipped in a level, you can also skip access points.
We will remove this access point.

Create a design, and skip ladders where needed
Open the required Bay
In the Ladders dialog, click the Skip Access Points for Ladders skipped checkbox

Click Apply

Your model will look like this.
