For reliable data transmission, 802.11ax standard employs various orders of modulation schemes with a forward error correction method performing different coding rates (CR). Higher-order modulation schemes can enhance the data rate, but at the same time increasing the possibility of data corruption and bit error occurrence. Moreover, in wireless communications, each modulation scheme can be used with different guard intervals and channel bandwidths. A shorter guard interval increases the data rate at the cost of increasing the interferences and data loss. A longer guard interval solves the issue but at the cost of the performance reduction due to wasting the useful bandwidth. With regards to channel bandwidth, although wider channels increase the data rate, they are subject to more signal interference. This can get even worse in the high-density deployment of 802.11ax where many users are placed in close distance and the signal interferences are strong. Thus, aside from the modulation orders and coding rates parameters, the efficiency of modulation schemes relies on the channel bandwidth and guard interval which demands the proper selection of these parameters. Consequently, this work proposes a simulation model to optimize the performance of the 802.11ax network regarding the modulation schemes under high-density conditions. The model includes all available modulation schemes and their corresponding coding rates along with the channel bandwidth and guard interval. The model is further implemented and the most efficient values for performance optimization are determined on the basis of bit error rate, throughput and its efficiency, end-to-end delay, loss ratio, and jitter.