Loading data error

[KululuMi]

Hello and thanks for the code.

I used this command to run the train.py with pytorch0.4.0, CUDA9.0, CUDNN 7.1.4

CUDA_VISIBLE_DEVICES=0,1 python3 train.py --dist_url /workdir/mixiaoyue/nonexistent_file --cfgs_file cfgs/anet.yml --checkpoint_path checkpoint/$id --batch_size 4 --world_size 2 --cuda --sent_weight 0.25 --mask_weight 1.0 --gated_mask |tee log.txt

However,I got the bug when video data loading:

What can I do for running it?

[KululuMi]

I'm not sure if the image was uploaded successfully. Here is the content of bug、

video: 9g1omTdpoYs
Traceback (most recent call last):
File "train.py", line 550, in
main(args)
File "train.py", line 241, in main
train_loader, valid_loader, text_proc, train_sampler = get_dataset(args)
File "train.py", line 154, in get_dataset
sample_listpath=args.train_samplelist_path,
File "/workdir/mixiaoyue/densecap/data/anet_dataset.py", line 155, in init
results[i] = r.get()
File "/home/hadoop-waimai/anaconda2/envs/densecap/lib/python3.6/multiprocessing/pool.py", line 608, in get
raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '('/workdir/mixiaoyue/densecap/feature_root/anet/training/sJFgo9H6zNo', 298, defaultdict(<class 'list'>, {6: [(2962, array([0.73074257]), 0.28082304007157594, 0.18079329689298143, tensor([

[LuoweiZhou]

@KululuMi I'd suggest you run the code under the single-GPU setting first. Also, when you run it with dist data parallel, make sure you set --dist_url file:///workdir/mixiaoyue/nonexistent_file instead. We have seen a few issues with the multiprocessing package with this version of pytorch, for example issue 11. You can search in the repo Issues for more. Besides, what's your python3 and OS version?

[KululuMi]

@LuoweiZhou Thanks to your reply.I use python3.6 and Linux version 3.10.0-862.mt20181120.113.el7.x86_64.

I've tried a new command follow your advice:

CUDA_VISIBLE_DEVICES=0 python3 train.py --dist_url file:///workdir/mixiaoyue/nonexistent_file --cfgs_file cfgs/anet.yml --checkpoint_path checkpoint/$id --batch_size 4 --world_size 1 --cuda --sent_weight 0.25 --mask_weight 1.0 --gated_mask |tee log.txt

However, I still have the same problem.

And then I edited train.py and anet_dataset.py scripts follow #11

But there's no response when the program is running somewhere:

/workdir/mixiaoyue/densecap/train.py:123: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
options_yaml = yaml.load(handle)

[LuoweiZhou]

@KululuMi The last msg is a warning regarding YAML and does not account for the issue. When you say you followed #11, you mean you have disabled multiprocessing? BTW, are you using Ubuntu 16.04?

[KululuMi]

@LuoweiZhou I use CentOS 7.5.1804. I followed #11 :
made pin_memory =True, and edited the train.py anet_dataset.py scripts with this :
from torch.multiprocessing import Pool, Process, set_start_method, cpu_count
try:
set_start_method('spawn')
except RuntimeError:
pass

I've checked there was no defunct process. But there's no response when the program is running somewhere:

[LuoweiZhou]

@KululuMi I've never tested the code on CentOS but I suspect multiprocessing is the issue here. You can refer to the following comments and disable it for now:
#11 (comment)
LuoweiZhou#3

[KululuMi]

@LuoweiZhou Thanks a lot! I have tried your suggestion and solved the problem. However, I met another question. When running the program, GPU was not used, and Loss was always nan. I don't know what went wrong

[LuoweiZhou]

@KululuMi When you say "GPU was not used", can you monitor your nvidia-smi and make sure the utilization is consistently 0%? Because the way in which we load feature files (individual *.npy files), the training speed is largely affected by data loading speed (if you have a slow hard disk I/O). This results in 0% utilization sometimes. For the nan issue, you need to first locate which loss in particular caused the issue (i.e., cls, reg, sentence, mask).

[KululuMi]

@LuoweiZhou Thanks for your reply. I'm sure the utilization of my GPU is consistently 0%, I've tried 5 times. and print(arg.cuda) is always True.

The loss is normal after reducing the learning rate, but when I tried to use pretrained model, the process showed a new bug:

But I did not change the network architecture, or input.

[KululuMi]

@LuoweiZhou I've solved the gpu utilization problem, that's a bug in docker. But when I use pretrained model, it's always the same Bug