CUDA OUT OF MEMORY

[gulzainali98]

I am trying to calculate lpips metric on directory of images. i use following code to load and calculate

for i in range(len(ground_truth)):

                p= lpips.im2tensor(lpips.load_image(predictions[i]))
                g= lpips.im2tensor(lpips.load_image(ground_truth[i]))
                if use_gpu:
                    p=p.cuda()
                    g=g.cuda()
                mean_total=mean_total+loss_fn.forward(g,p).mean()
                im_counter=im_counter+1
                del p
                del g
                torch.cuda.empty_cache()

However, I am getting cuda out of memory error even with alexnet backbone and A100 40GB GPU!

[gulzainali98]

When i use GTX2080 with pytorch 1.5 I get following error:

Traceback (most recent call last): File "eval.py", line 69, in <module> p= lpips.im2tensor(lpips.load_image(predictions[i])) File "/opt/conda/lib/python3.6/site-packages/lpips/__init__.py", line 113, in im2tensor [:, :, :, np.newaxis].transpose((3, 2, 0, 1))) RuntimeError: unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation.

[huluwaXYZ]

I am trying to calculate lpips metric on directory of images. i use following code to load and calculate

for i in range(len(ground_truth)):

                p= lpips.im2tensor(lpips.load_image(predictions[i]))
                g= lpips.im2tensor(lpips.load_image(ground_truth[i]))
                if use_gpu:
                    p=p.cuda()
                    g=g.cuda()
                mean_total=mean_total+loss_fn.forward(g,p).mean()
                im_counter=im_counter+1
                del p
                del g
                torch.cuda.empty_cache()

However, I am getting cuda out of memory error even with alexnet backbone and A100 40GB GPU!

Try to replace with
mean_total=mean_total+loss_fn.forward(g,p).mean().cpu().detach().numpy()
in your code

[piddnad]

Hi, I encountered the same question, could you Explain how to solve it?

My codes are like this:

    total_lpips, total_lpips_convert = 0, 0
    with torch.no_grad():
        loss_fn_vgg = lpips.LPIPS(net='vgg').cuda()
        for img in tqdm(img_names):
            gt_pth = os.path.join(gt_dir, img)
            gt = np.array(Image.open(gt_pth).convert("RGB")).astype(np.uint8)
            gt = transforms.ToTensor()(gt)
            gt = transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))(gt).cuda()

            pred_pth = os.path.join(predicted_dir, img)
            pred = np.array(Image.open(pred_pth).convert("RGB"))
            pred = transforms.ToTensor()(pred)
            pred = transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))(pred).cuda()

            lpips_vgg = loss_fn_vgg(gt, pred).detach().cpu().numpy()
            total_lpips += lpips_vgg

            img_counts += 1
            del gt, pred, lpips_vgg
            torch.cuda.empty_cache()

[smandal94]

Shouldn't the forward pass be done under "with torch.no_grad():" in lpips_2imgs.py, lpips_2dirs.py and lpips_1dir_allpairs.py? This solved the CUDA out of memory issue for me.

[ourpubliccodes]

Which line would you put this sentence on? Is this right below?