OpenNMT Forum

Case feature error on eval

Hello,
I am getting Missing case feature error while training at the evaluation step
This is the data.yml file, I am using.

model_dir: run/
data:
  train_features_file: src-train.txt
  train_labels_file: tgt-train.txt
  eval_features_file: src-val.txt
  eval_labels_file: tgt-val.txt
  source_vocabulary: src-vocab.txt
  target_vocabulary: tgt-vocab.txt
  source_tokenization:
    type: OpenNMTTokenizer
    params:
      mode: aggressive
      case_feature: true
      sp_model_path: src.model
      # sp_nbest_size: 64
      # sp_alpha: 0.1
      joiner_annotate: true
      segment_numbers: true
  target_tokenization:
    type: OpenNMTTokenizer
    params:
      mode: aggressive
      case_feature: true
      sp_model_path: tgt.model
      # sp_nbest_size: 64
      # sp_alpha: 0.1
      joiner_annotate: true
      segment_numbers: true
train:
   save_checkpoints_steps: 2000
eval:
   external_evaluators: bleu

Error Message:

INFO:tensorflow:Step = 10000 ; steps/s = 0.41, source words/s = 9277, target words/s = 10388 ; Learning rate = 0.000884 ; Loss = 1.437881
INFO:tensorflow:Saved checkpoint run/ckpt-10000
INFO:tensorflow:Running evaluation for step 10000
Traceback (most recent call last):
  File "/usr/local/bin/onmt-main", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/opennmt/bin/main.py", line 224, in main
    hvd=hvd)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/runner.py", line 236, in train
    moving_average_decay=train_config.get("moving_average_decay"))
  File "/usr/local/lib/python3.6/dist-packages/opennmt/training.py", line 122, in __call__
    early_stop = self._evaluate(evaluator, step, moving_average=moving_average)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/training.py", line 149, in _evaluate
    evaluator(step)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/evaluation.py", line 304, in __call__
    ordered_writer.push(prediction)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/utils/misc.py", line 382, in push
    return self._try_notify()
  File "/usr/local/lib/python3.6/dist-packages/opennmt/utils/misc.py", line 368, in _try_notify
    self._callback_fn(value)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/evaluation.py", line 284, in <lambda>
    self._model.print_prediction(prediction, stream=output_file))
  File "/usr/local/lib/python3.6/dist-packages/opennmt/models/sequence_to_sequence.py", line 358, in print_prediction
    sentence = self.labels_inputter.tokenizer.detokenize(tokens)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/tokenizers/tokenizer.py", line 119, in detokenize
    return self._detokenize(tokens, sequence_length)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/tokenizers/tokenizer.py", line 144, in _detokenize
    return self._detokenize_string(tokens)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/tokenizers/opennmt_tokenizer.py", line 47, in _detokenize_string
    return self._tokenizer.detokenize(tokens)
RuntimeError: Missing case feature

Hi,

case_feature is not supported for online tokenization.

(We should probably raise an error when starting the training.)