When i run the server i get to experience this error message…
Pre-loading model 100
[2018-11-10 10:48:18,319 INFO] Loading model 100
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524590031827/work/aten/src/THC/THCGeneral.cpp line=70 error=30 : unknown error
Traceback (most recent call last):
File “/home/stano/OpenNMT-py-master/onmt/translate/translation_server.py”, line 239, in load
out_file=open(os.devnull, “w”))
File “/home/stano/OpenNMT-py-master/onmt/translate/translator.py”, line 35, in build_translator
onmt.model_builder.load_test_model(opt, dummy_opt.dict)
File “/home/stano/OpenNMT-py-master/onmt/model_builder.py”, line 142, in load_test_model
model = build_base_model(model_opt, fields, use_gpu(opt), checkpoint)
File “/home/stano/OpenNMT-py-master/onmt/model_builder.py”, line 264, in build_base_model
model.to(device)
File “/home/stano/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 393, in to
return self._apply(lambda t: t.to(device))
File “/home/stano/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 176, in _apply
module._apply(fn)
File “/home/stano/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 176, in _apply
module._apply(fn)
File “/home/stano/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 176, in _apply
module._apply(fn)
[Previous line repeated 1 more times]
File “/home/stano/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 182, in _apply
param.data = fn(param.data)
File “/home/stano/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 393, in
return self._apply(lambda t: t.to(device))
File “/home/stano/anaconda3/lib/python3.6/site-packages/torch/cuda/init.py”, line 161, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /opt/conda/conda-bld/pytorch_1524590031827/work/aten/src/THC/THCGeneral.cpp:70
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “server.py”, line 118, in
debug=args.debug)
File “server.py”, line 25, in start
translation_server.start(config_file)
File “/home/stano/OpenNMT-py-master/onmt/translate/translation_server.py”, line 80, in start
self.preload_model(opt, model_id=model_id, **kwargs)
File “/home/stano/OpenNMT-py-master/onmt/translate/translation_server.py”, line 116, in preload_model
model = ServerModel(opt, model_id, **model_kwargs)
File “/home/stano/OpenNMT-py-master/onmt/translate/translation_server.py”, line 188, in init
self.load()
File “/home/stano/OpenNMT-py-master/onmt/translate/translation_server.py”, line 241, in load
raise ServerModelError(“Runtime Error: %s” % str(e))
onmt.translate.translation_server.ServerModelError: Runtime Error: cuda runtime error (30) : unknown error at /opt/conda/conda-bld/pytorch_1524590031827/work/aten/src/THC/THCGeneral.cpp:70
It is able to read the the model but the server terminates indefinately.
@pltrdy @vince62s can you help me solve this or explain to me how i can go about it
below is my conf.json file
{
“models_root”: “./available_models”,
“models”: [
{
“id”: 100,
“model”: “kikuyu_kiswahili_test_model_step_100000.pt”,
“timeout”: 600,
“on_timeout”: “to_cpu”,
“load”: true,
“opt”: {
“gpu”: 0,
“beam_size”: 5
},
“tokenizer”: {
“type”: “sentencepiece”,
“model”: “mymodel.model”
}
}
]
}