My custom image generator script crashed after a 500 image batch last night.

I was running a local stable diffusion model to create some concept art for a project. After about 500 images, the whole thing just froze and the command line threw a memory error. I checked my task manager and saw the VRAM usage was maxed out on my GPU. I had to restart the process and add a line to clear the cache every 50 images, which fixed it. Has anyone else hit a wall with long generation jobs and found a better fix?

3 comments

3 Comments

phoenix_perez9726d ago

Totally feel your pain with that memory crash. Running big batches overnight just to find a frozen screen is the worst. Clearing the cache every few images is a solid workaround, it's what I ended up doing too. Have you tried messing with the garbage collection settings in your script at all?

hollyg5926d agoMost Upvoted

Ugh, @phoenix_perez97, garbage collection is a total lifesaver.

xena27721d ago

Yeah that garbage collection tip is a game changer honestly. I mean, @phoenix_perez97, it reminds me of when I tried to render this huge animation loop and my whole system just gave up halfway through. Had to dig into the settings and basically force it to clean up more often, which helped a ton. Still a pain to babysit it though, feels like you can never fully trust it to run on its own. Maybe it's just me but I always expect the worst now after a few crashes.