Bias and unfairness of test items in language proficiency tests have been an issue of scholarly investigation. Unfair test items and cultural biases in international English language proficiency tests such as TOEFL and IELTS have received considerable attention. However, such unfairness and biases in proficiency tests conducted at the national level such as Bangladesh Civil Service examination (henceforth referred to as BCS) have not been investigated. This paper attempts to examine item biases and inconsistencies in English language testing of BCS preliminary examination. Qualitative data has been collected from stakeholders using a semi-structured interview technique. A detailed content analysis of the English section of the BCS examination is conducted. Most test items have been found to be biased towards testing knowledge of English literature which is inconsistent with validity constructs of the language proficiency test. Stakeholders have been found to perceive the test as ineffective in measuring their English language proficiency.