A new study from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) demonstrates that large language models (LLMs) can be used to automatically generate realistic and functional software benchmarks. The researchers developed a method where an LLM, given a simple text description of a desired programming task, can produce both the code for the benchmark and the test cases to evaluate it. This approach, named ‘LM-generated benchmark evolution,’ aims to address the growing challenge of creating diverse and up-to-date benchmarks to test the rapidly advancing capabilities of AI coding assistants. The system can iteratively refine and expand these benchmarks, creating more complex tasks over time. Initial tests show these AI-generated benchmarks are effective at revealing performance gaps in existing code-generation models that standard benchmarks might miss. For more details, read the full article at https://technologyreview.com/2024/05/15/1090000/ai-can-now-design-benchmarks-to-test-its-own-coding-skills/