python out of memory_显存充足，但报错 out of memory

用cascade_rcnn_cbr200_vd_fpn_dcnv2_nonlocal_softnms.yml训练自己的数据集

显存充足，但报错 out of memory，请问该怎么解决这个问题？

`python3 -u tools/train.py -c configs/dcn/cascade_rcnn_cbr200_vd_fpn_dcnv2_nonlocal_softnms.yml -o pretrain_weights=models/cascade_rcnn_cbr200_vd_fpn_dcnv2_nonlocal_softnms --use_tb=True --tb_log_dir=tb_log_caltech/scalar --eval

P.S. batch_size已经设置为1，

尝试多进程方式 python -m paddle.distributed.launch --selected_gpus 0,1,2,3 tools/train.py ... 也有同样的问题

实在不知道要怎么做了，求指教...

2020-04-06 17:36:49,707-INFO: 6707 samples in file dataset/coco/annotations/instances_val2007.json

2020-04-06 17:36:49,712-INFO: places would be ommited when DataLoader is not iterable

W0406 17:36:50.419684 27808 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.0, Runtime API Version: 10.0

W0406 17:36:50.422271 27808 device_context.cc:245] device: 0, cuDNN Version: 7.6.

2020-04-06 17:36:51,523-INFO: Loading parameters from models/cascade_rcnn_cbr200_vd_fpn_dcnv2_nonlocal_softnms...

2020-04-06 17:36:51,524-WARNING: models/cascade_rcnn_cbr200_vd_fpn_dcnv2_nonlocal_softnms.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]

loading annotations into memory...

Done (t=0.20s)

creating index...

index created!

2020-04-06 17:36:53,468-WARNING: Found an invalid bbox in annotations: im_id: 5387, area: -10.0 x1: 348, y1: 176, x2: 348, y2: 196.

2020-04-06 17:36:53,481-WARNING: Found an invalid bbox in annotations: im_id: 5765, area: -10.0 x1: 71, y1: 174, x2: 71, y2: 197.

2020-04-06 17:36:53,686-INFO: 15649 samples in file dataset/coco/annotations/instances_train2007.json

2020-04-06 17:36:53,699-INFO: places would be ommited when DataLoader is not iterable

I0406 17:36:54.286912 27808 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 4 cards are used, so 4 programs are executed in parallel.

W0406 17:37:00.890586 27808 fuse_all_reduce_op_pass.cc:74] Find all_reduce operators: 730. To make the speed faster, some all_reduce ops are fused during training, after fusion, the number of all_reduce ops is 604.

I0406 17:37:01.025799 27808 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1

I0406 17:37:28.992170 27808 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True

I0406 17:37:29.978662 27808 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0

W0406 17:37:42.174067 508 operator.cc:181] deformable_conv raises an exception paddle::memory::allocation::BadAlloc,

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackStringstd::string(std::string&&, char const*, int)

1 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long)

2 paddle::memory::allocation::AlignedAllocator::AllocateImpl(unsigned long)

3 paddle::memory::allocation::AutoGrowthBestFitAllocator::AllocateImpl(unsigned long)

4 paddle::memory::allocation::Allocator::Allocate(unsigned long)

5 paddle::memory::allocation::RetryAllocator::AllocateImpl(unsigned long)

6 paddle::memory::allocation::AllocatorFacade::Alloc(paddle::platform::Place const&, unsigned long)

7 paddle::memory::Alloc(paddle::platform::Place const&, unsigned long)

8 paddle::memory::Alloc(paddle::platform::DeviceContext const&, unsigned long)

9 paddle::framework::Tensor paddle::framework::ExecutionContext::AllocateTmpTensor(paddle::framework::DDim const&, paddle::platform::CUDADeviceContext const&) const

10 paddle::operators::DeformableConvCUDAKernel<:platform::cudadevicecontext float>::Compute(paddle::framework::ExecutionContext const&) const

11 std::_Function_handler >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&)

12 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const

13 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const

14 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&)

15 paddle::framework::details::ComputationOpHandle::RunImpl()

16 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*)

17 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<:framework::blockingqueue> const&, unsigned long*)

18 std::_Function_handler<:unique_ptr std::__future_base::_result_base::_deleter> (), std::__future_base::_Task_setter<:unique_ptr std::__future_base::_result_base::_deleter>, void> >::_M_invoke(std::_Any_data const&)

19 std::__future_base::_State_base::_M_do_set(std::function<:unique_ptr std::__future_base::_result_base::_deleter> ()>&, bool&)

20 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

Error Message Summary:

ResourceExhaustedError:

Out of memory error on GPU 2. Cannot allocate 66.797119MB memory on GPU 2, available memory is only 31.062500MB.

Please check whether there is any other process using GPU 2.

If yes, please stop them, or start PaddlePaddle on another GPU.

If no, please decrease the batch size of your model.

at (/paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:69)

F0406 17:37:42.174147 508 exception_holder.h:37] std::exception caught,