Sunwoo Kim, a student from the 2nd cohort of the KRAFTON AI Fellowship, has successfully published a first-author paper at the International Conference on Learning Representations (ICLR), one of the world’s top three AI conferences. Furthermore, the paper has been selected as a Spotlight paper, recognizing it as an outstanding and innovative research achievement among the accepted papers. How did an undergraduate student accomplish such an extraordinary feat even before graduating? We had the opportunity to interview Sunwoo Kim to learn more about his journey and insights. 😊
(Photo 1) Today's Main Feature: Sunwoo Kim from the 2nd Cohort of KRAFTON AI Fellowship
Q. Hello! Could you briefly introduce yourself and share your motivation for applying?
I am Sunwoo Kim, a student at Seoul National University majoring in Statistics, currently studying and researching AI. I am particularly interested in AI that goes beyond merely mimicking data and instead evolves through interactions with the real world.
As I studied AI through university courses, books, and online lectures, I always had a strong desire to conduct research myself. The KRAFTON AI Fellowship attracted me because it offered the opportunity to lead a research project while providing generous support in both human and material resources (such as computing power). Of course, the highly competitive scholarship was also one of the key factors that solidified my decision to apply. 😊
As I studied AI through university courses, books, and online lectures, I always had a strong desire to conduct research myself. The KRAFTON AI Fellowship attracted me because it offered the opportunity to lead a research project while providing generous support in both human and material resources (such as computing power). Of course, the highly competitive scholarship was also one of the key factors that solidified my decision to apply. 😊
Q. What projects did you work on during the AI Fellowship?
During the AI Fellowship, I primarily worked on two major projects. In the beginning, I conducted research and wrote a paper on diffusion models. Later, I participated in the Smart Zoi project, which focuses on an AI feature for KRAFTON’s upcoming life simulation game, inZOI.
I can explain the diffusion model research in more detail later, but in simple terms, we worked on a generative method that enables models—pre-trained on internet data and producing relatively low-quality outputs—to generate images with desired characteristics (e.g., following prompts more accurately or creating more aesthetically pleasing images).
The second project, Smart Zoi, involved utilizing Small Language Model (SLM). The goal was to develop an on-device AI agent that controls characters within inZOI. My primary role was enhancing the reasoning and planning capabilities of the AI using SLM.
I can explain the diffusion model research in more detail later, but in simple terms, we worked on a generative method that enables models—pre-trained on internet data and producing relatively low-quality outputs—to generate images with desired characteristics (e.g., following prompts more accurately or creating more aesthetically pleasing images).
The second project, Smart Zoi, involved utilizing Small Language Model (SLM). The goal was to develop an on-device AI agent that controls characters within inZOI. My primary role was enhancing the reasoning and planning capabilities of the AI using SLM.
(Photo 2) Paper Accepted at ICLR 2025
Q. It seems that the research you mentioned earlier is the one that was accepted at ICLR. Could you explain it in more detail?
Yes, that’s correct. The research is titled “Test-time Alignment of Diffusion Models without Reward Over-optimization.” As I mentioned earlier, this study focuses on alignment, which involves adjusting pre-trained diffusion models to better adhere to the desired characteristics we specify.
To put it formally, we define the problem as generating samples that maximize a given reward function, which measures how well the generated output meets the desired characteristics. Traditional approaches typically address this by fine-tuning the diffusion model with additional training. However, this process can lead to issues like reward hacking or reward over-optimization, where the model learns to exploit the reward function in unintended ways. For example, the generated images might achieve high reward scores numerically but fail to align with the actual desired characteristics.
In our paper, we proposed a method that enhances the reward signal without requiring additional fine-tuning during the generation process. This approach mitigates reward hacking and allows us to obtain higher-quality results without the need for extra training.
To put it formally, we define the problem as generating samples that maximize a given reward function, which measures how well the generated output meets the desired characteristics. Traditional approaches typically address this by fine-tuning the diffusion model with additional training. However, this process can lead to issues like reward hacking or reward over-optimization, where the model learns to exploit the reward function in unintended ways. For example, the generated images might achieve high reward scores numerically but fail to align with the actual desired characteristics.
In our paper, we proposed a method that enhances the reward signal without requiring additional fine-tuning during the generation process. This approach mitigates reward hacking and allows us to obtain higher-quality results without the need for extra training.
Q. That’s a sharp and insightful approach, addressing a major gap in existing research. What motivated you to pursue this study?
The motivation behind this research was a technical discovery. In fact, the issue of reward hacking in fine-tuning-based methods has been repeatedly mentioned in various previous papers. As I was thinking about this problem, I decided to mathematically formulate the training objective function of existing fine-tuning-based methods. Through this process, I realized that it doesn’t necessarily have to be approached as an optimization problem but could instead be framed as an inference problem. This led me to wonder: If we want to generate samples from a distribution where the reward is high while utilizing a pre-trained model, what kind of inference method should we use? After exploring different approaches, I concluded that Sequential Monte Carlo (SMC) would be a suitable method, which became the starting point of this research.
Q. Even if you came up with the idea, implementing it must not have been easy. Were there any difficulties during the research process?
There were three major challenges I faced during the research process.
The first challenge was selecting the research topic. Before my internship, I took a project-based course at school, and during that time, I was interested in so many topics that I struggled to settle on just one. As a result, I wasn’t able to conduct in-depth research. In the KRAFTON AI Fellowship, I wanted to avoid making the same mistake, so instead of diverging into multiple ideas, I focused on quickly converging on one topic and delving deep into it. In this regard, the feedback from experienced mentors was incredibly helpful.
The second challenge was somewhat ironic. When I first tested the proposed methodology, the results were far better than I had expected, but even I found it difficult to clearly explain why. It was particularly surprising that our method outperformed fine-tuning approaches without requiring additional training. While I could have simply stated, “The results turned out well,” and submitted the paper, I didn’t want to leave it at that. To explain this, I analyzed mathematically why reward hacking occurs in traditional fine-tuning methods and confirmed that similar trends appeared in toy examples. Additionally, I worked on theoretical proofs to clarify the advantages of our proposed method. The process of logically justifying why my approach worked, after initially assuming it might work, was challenging—but looking back, it was an invaluable experience.
The final challenge was the last three weeks before the paper submission deadline. Initially, I had planned to submit the paper in January, but as the research progressed, I felt that the findings were coming together well, so I decided to submit earlier than planned. Because of this, I had to complete additional experiments, finalize the proofs, and finish writing the paper within a short period. The only way to overcome this was simply to push through it head-on. For those three weeks, I worked incredibly hard, often sleeping in KRAFTON’s nap room (by the way, it’s really comfortable and great!).
The first challenge was selecting the research topic. Before my internship, I took a project-based course at school, and during that time, I was interested in so many topics that I struggled to settle on just one. As a result, I wasn’t able to conduct in-depth research. In the KRAFTON AI Fellowship, I wanted to avoid making the same mistake, so instead of diverging into multiple ideas, I focused on quickly converging on one topic and delving deep into it. In this regard, the feedback from experienced mentors was incredibly helpful.
The second challenge was somewhat ironic. When I first tested the proposed methodology, the results were far better than I had expected, but even I found it difficult to clearly explain why. It was particularly surprising that our method outperformed fine-tuning approaches without requiring additional training. While I could have simply stated, “The results turned out well,” and submitted the paper, I didn’t want to leave it at that. To explain this, I analyzed mathematically why reward hacking occurs in traditional fine-tuning methods and confirmed that similar trends appeared in toy examples. Additionally, I worked on theoretical proofs to clarify the advantages of our proposed method. The process of logically justifying why my approach worked, after initially assuming it might work, was challenging—but looking back, it was an invaluable experience.
The final challenge was the last three weeks before the paper submission deadline. Initially, I had planned to submit the paper in January, but as the research progressed, I felt that the findings were coming together well, so I decided to submit earlier than planned. Because of this, I had to complete additional experiments, finalize the proofs, and finish writing the paper within a short period. The only way to overcome this was simply to push through it head-on. For those three weeks, I worked incredibly hard, often sleeping in KRAFTON’s nap room (by the way, it’s really comfortable and great!).
Q. It must have been incredibly challenging. Without the support of those around you, it would have been even more difficult. Was there a mentor at KRAFTON who helped you the most?
I conducted my research with Minkyu and Dongmin from KRAFTON’s Deep Learning Division, specifically the Data-centric Research team. Through this collaboration, I learned so much. Many people say that asking the right questions is crucial for good research, and I deeply agree—but I also think it’s one of the hardest things to do.
My mentors helped me think through the significance and impact of my research topics and guided me on how to structure the storytelling, which helped me clarify which problems to focus on. This was the most valuable part of my research journey and played a key role in my growth as a researcher. Additionally, they provided insightful feedback on methodology, experimental design, paper writing, and even the details of the rebuttal process. Thanks to them, I was able to gain valuable experience in research and paper writing, and without their support, achieving an ICLR Spotlight would have been impossible.
My mentors helped me think through the significance and impact of my research topics and guided me on how to structure the storytelling, which helped me clarify which problems to focus on. This was the most valuable part of my research journey and played a key role in my growth as a researcher. Additionally, they provided insightful feedback on methodology, experimental design, paper writing, and even the details of the rebuttal process. Thanks to them, I was able to gain valuable experience in research and paper writing, and without their support, achieving an ICLR Spotlight would have been impossible.
Q. You put in a lot of hard work, but it seems like writing this paper was a rewarding experience, especially with great people by your side. Can you share a bit of what you’re proud of about your paper?
The alignment of diffusion models has a wide range of applications. Beyond the image generation experiments presented in the paper, this method can also be applied to protein-generation diffusion models, contributing to drug discovery. If applied to the widely used Diffusion Policy, it could also have implications in robotics.
In this context, I believe our work is valuable because we mathematically identified the cause of reward hacking in existing alignment techniques and proposed a novel test-time algorithm to address this issue. I hope this research will contribute to future studies on diffusion model alignment. It is also gratifying to see that the AI research community recognized this contribution, leading to the opportunity to present it as a Spotlight paper at ICLR.
In this context, I believe our work is valuable because we mathematically identified the cause of reward hacking in existing alignment techniques and proposed a novel test-time algorithm to address this issue. I hope this research will contribute to future studies on diffusion model alignment. It is also gratifying to see that the AI research community recognized this contribution, leading to the opportunity to present it as a Spotlight paper at ICLR.
(Photo3) A News article about KRAFTON’s Papers Featured in Media (Digital Times)
(Translation) "According to the industry on the 17th, KRAFTON is set to present five papers at 'ICLR 2025,' one of the world's top three AI conferences, in April. The submitted papers cover three topics in simulation and animation, one in audio synthesis, and one in AI for studio data centers. The technologies disclosed in these papers are expected to have applications beyond the gaming industry, potentially extending to other sectors, including military use."
Q. Did you expect such results when you decided to join the Fellowship? Or was this news just as unexpected for you?
It was completely unexpected news. Throughout my Fellowship experience, I encountered many things I hadn’t anticipated, but the most surprising was being able to complete a paper and submit it to a conference in such a short period. I’ve already talked a lot about my research experience, but in the end, I think it was the environment provided by KRAFTON that allowed me to fully dedicate myself to research and pour everything into it within such a short time. Beyond that, I also gained many unforgettable experiences and memories, such as working on a large-scale corporate project like Smart Zoi, attending G-Star, and participating in team workshops.
Q. What was the biggest lesson you learned from the Fellowship?
The biggest change after the Fellowship was the way I approach research. Previously, I thought research was about finding a “good methodology” through novel ideas, and whenever I came up with a new method, I would try to fit it into a problem it could solve. While I haven’t completely broken this habit, I’ve come to realize that the most important aspect of research is finding a “good problem” or “good question” to address.
Through my mentors’ feedback and my own reflections on the impact of my research, I eventually understood that the key was not just about applying methods but about identifying the right problems to solve. This shift in mindset has also changed the way I read research papers. Now, before focusing on the methodologies used, I first try to clearly understand what problem the paper is trying to solve.
Most importantly, when choosing a new research topic or planning long-term projects, I now start by thinking about the problems I genuinely want to solve in both the short and long term.
Through my mentors’ feedback and my own reflections on the impact of my research, I eventually understood that the key was not just about applying methods but about identifying the right problems to solve. This shift in mindset has also changed the way I read research papers. Now, before focusing on the methodologies used, I first try to clearly understand what problem the paper is trying to solve.
Most importantly, when choosing a new research topic or planning long-term projects, I now start by thinking about the problems I genuinely want to solve in both the short and long term.
Q. It’s really gratifying to hear that the Fellowship had such a positive impact on how you approach research. What are your future plans and goals for continuing your research?
In the long term, I want to study AI that goes beyond just mimicking data and can improve through feedback from the real world. Over the past few years, large models in language and vision have proven that, given enough data, they can replicate it effectively. However, AI models become truly powerful—or even disruptive—when they go beyond imitation and create something new. Since this “new” form of data is scarce in existing databases, AI models must learn through interactions with the real world, such as human preference feedback.
Research in this area is already progressing across various domains. My research topic, diffusion model alignment, and the recently publicized DeepSeek-R1 are examples of this. However, there is still a long way to go. Our understanding of emergent abilities in large models remains limited. As AI capabilities continue to advance, concerns about safety will only become more pressing. At the same time, the limitations of current methodologies are clear, and overcoming these constraints is essential for AI models to evolve beyond mere data imitation and continue improving.
I see my research during the Fellowship as the first step in this journey. For my next step, I am currently working on robot learning research. In the field of robotics, there has been growing interest in imitation learning methods that leverage LLMs and diffusion model training techniques. However, due to data scarcity, pre-trained models often perform poorly, and my goal is to enhance them further.
Another area of interest is drug discovery. Protein design for drug development requires creating proteins that do not yet exist in the real world. Since these proteins are entirely new, predicting their effects is much more challenging. To address this, real-world feedback, such as laboratory experiments, must be effectively incorporated into the AI learning process.
The Fellowship provided the starting point for this journey and helped me develop a broader perspective on AI advancements. Most importantly, my positive experiences at KRAFTON gave me the confidence to continue on this path.
Research in this area is already progressing across various domains. My research topic, diffusion model alignment, and the recently publicized DeepSeek-R1 are examples of this. However, there is still a long way to go. Our understanding of emergent abilities in large models remains limited. As AI capabilities continue to advance, concerns about safety will only become more pressing. At the same time, the limitations of current methodologies are clear, and overcoming these constraints is essential for AI models to evolve beyond mere data imitation and continue improving.
I see my research during the Fellowship as the first step in this journey. For my next step, I am currently working on robot learning research. In the field of robotics, there has been growing interest in imitation learning methods that leverage LLMs and diffusion model training techniques. However, due to data scarcity, pre-trained models often perform poorly, and my goal is to enhance them further.
Another area of interest is drug discovery. Protein design for drug development requires creating proteins that do not yet exist in the real world. Since these proteins are entirely new, predicting their effects is much more challenging. To address this, real-world feedback, such as laboratory experiments, must be effectively incorporated into the AI learning process.
The Fellowship provided the starting point for this journey and helped me develop a broader perspective on AI advancements. Most importantly, my positive experiences at KRAFTON gave me the confidence to continue on this path.
Q. Thank you for sharing your experiences at KRAFTON. Lastly, KRAFTON AI Fellowship will be recruiting its 3rd cohort this December. Do you have any words of advice for students considering applying?
For undergraduate students interested in AI research, opportunities like the AI Fellowship Internship are rare. All selected Fellows have the chance to lead their own research projects or participate in projects they are passionate about while working alongside experienced and highly skilled mentors throughout the process.
Additionally, ample computing resources are provided— I could access up to four A100 GPUs at any time! On top of that, the generous scholarship and salary, along with delicious daily meals, allow you to focus entirely on your research while enjoying a comfortable and fulfilling lifestyle. These factors made my time in the Fellowship incredibly efficient and productive.
Lastly, I’ve heard that the selection exam for the Fellowship is crafted with high-quality and carefully curated problems. This means that even the application process itself can be a great learning experience for undergraduate students. If you’re interested, there’s no reason to hesitate—just go for it!
Additionally, ample computing resources are provided— I could access up to four A100 GPUs at any time! On top of that, the generous scholarship and salary, along with delicious daily meals, allow you to focus entirely on your research while enjoying a comfortable and fulfilling lifestyle. These factors made my time in the Fellowship incredibly efficient and productive.
Lastly, I’ve heard that the selection exam for the Fellowship is crafted with high-quality and carefully curated problems. This means that even the application process itself can be a great learning experience for undergraduate students. If you’re interested, there’s no reason to hesitate—just go for it!
Through Sunwoo Kim’s interview, we got a glimpse of the diverse experiences offered by the KRAFTON AI Fellowship and the incredible achievements that can result from those experiences. If you are an undergraduate student who wants to grow and develop just like Sunwoo, don’t miss the opportunity to apply for the Fellowship, which opens every December! 😊
Introduction and reviews of the KRAFTON AI Fellowship
KRAFTON AI Fellowship application page (opens every December!)
KRAFTON AI research paper list
Introduction and reviews of the KRAFTON AI Fellowship
KRAFTON AI Fellowship application page (opens every December!)
KRAFTON AI research paper list