CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation

Kaiwen Yan1 *, Hongcheng Guo1 * †, Xuanqing Shi2, Jingyi Xu1, Yaonan Gu3, Zhoujun Li1,
1Beihang University 2Tsinghua University 3National University of Singapore
* Equal contribution. † Corresponding Author.
Paper Code Data Leaderboard

Introduction

With the rapid advancement of Large Language Models (LLMs), the demand for robust instruction-following capabilities in code generation tasks has grown significantly. Code generation not only facilitates faster prototyping and automated testing, but also augments developer efficiency through improved maintainability and reusability of code. In this paper, we introduce CodeIF, the first benchmark specifically designed to assess the abilities of LLMs to adhere to task-oriented instructions within diverse code generation scenarios. CodeIF encompasses a broad range of tasks, including function synthesis, error debugging, algorithmic refactoring, and code explanation, thereby providing a comprehensive suite to evaluate model performance across varying complexity levels and programming domains. We conduct extensive experiments with LLMs, analyzing their strengths and limitations in meeting the demands of these tasks. The experimental results offer valuable insights into how well current models align with human instructions, as well as the extent to which they can generate consistent, maintainable, and contextually relevant code. Our findings not only underscore the critical role that instruction-following LLMs can play in modern software development, but also illuminate pathways for future research aimed at enhancing their adaptability, reliability, and overall effectiveness in automated code generation.

Pipeline

DataSet Pipeline

The CodeIF framework, illustrated in Figure, is meticulously crafted to enhance code generation by leveraging constraint instructions derived from real-world coding tasks. This process involves a methodical collection and refinement of constraints, integral for ensuring the relevance and applicability to practical scenarios. By integrating these constraint instructions with advanced language models (LLMs) and a rigorous human review process, CodeIF succeeds in constructing a robust and high-quality evaluation dataset. This dataset not only benchmarks the performance of LLMs in generating code but also fosters the development of more intelligent, understanding, and contextually aware coding assistants. The assembly of the dataset follows a structured protocol, ensuring the inclusion of diverse programming tasks and scenarios, thereby broadening the framework's applicability and effectiveness across various coding environments and challenges.

Dataset

Instruction Analysis Instruct Type Analysis

The CodeIF framework is meticulously designed to enhance automatic code generation capabilities by leveraging real-world constraints. It divides challenges into different levels of complexity and covers multiple programming languages to ensure comprehensive evaluation. Extensive Coverage: Supports Go, Python, Java, and C++, with task designs closely aligned with real-world coding scenarios. Structured Complexity: Provides datasets categorized into simple and difficult levels, aiding in targeted evaluation of language models under varying complexities. Detailed Insights: Offers in-depth insights into model performance across various programming instructions, helping identify areas for improvement. Explore the powerful capabilities of CodeIF in improving the precision and effectiveness of code generation models.

Model Result

Model Result

Figure above compares the performance of leading LLMs across four programming languages: C++, Java, Python, and Go, highlighting key trends at both the model and language levels.

BibTeX


@misc{yan2025codeifbenchmarkinginstructionfollowingcapabilities,
  title={CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation}, 
  author={Kaiwen Yan and Hongcheng Guo and Xuanqing Shi and Jingyi Xu and Yaonan Gu and Zhoujun Li},
  year={2025},
  eprint={2502.19166},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2502.19166}, 
}