OpenFst Forum

You need to be a registered user to participate in the discussions.

You can start a new discussion here:

You can use the formatting commands describes in TextFormattingRules in your comment.
If you want to post some code, surround it with <verbatim> and </verbatim> tags.
Auto-linking of WikiWords is now disabled in comments, so you can type VectorFst and it won't result in a broken link.
You now need to use <br> to force new lines in your comment (unless inside verbatim tags). However, a blank line will automatically create a new paragraph.
 Subject Comment

Error installing OpenFST under Ubuntu

KeesKoenen - 2017-08-17 - 14:02

Want to install OpenFST 1.6.2. Ran ./configure, make, make check. I get this output. Please advice. Thanks. PS Kind of a newbie at this, pls be gentle

<verbatim>fst_test.o: In function fst::internal::CompactFstImpl<fst::ArcTpl<fst::TropicalWeightTpl<float> >, fst::(anonymous namespace)::CustomCompactor<fst::ArcTpl<fst::TropicalWeightTpl<float> > >, unsigned short, fst::DefaultCompactStore<std::pair<int, fst::TropicalWeightTpl<float> >, unsigned short>, fst::DefaultCacheStore<fst::ArcTpl<fst::TropicalWeightTpl<float> > > >::Init(fst::Fst<fst::ArcTpl<fst::TropicalWeightTpl<float> > > const&, std::shared_ptr<fst::DefaultCompactStore<std::pair<int, fst::TropicalWeightTpl<float> >, unsigned short> >)': fst_test.cc:(.text+0x3c35f): undefined reference to fst::Int64ToStr(long, std::string*)

make[3]: Entering directory '/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/src/test' /bin/bash ../../libtool --tag=CXX --mode=link g++ -std=c++11 -o fst_test fst_test.o ../lib/libfst.la -lm -ldl libtool: link: g++ -std=c++11 -o .libs/fst_test fst_test.o ../lib/.libs/libfst.so -lm -ldl -Wl,-rpath -Wl,/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/lib Makefile:645: recipe for target 'fst_test' failed make[3]: Leaving directory '/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/src/test' Makefile:1054: recipe for target 'check-am' failed make[2]: Leaving directory '/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/src/test' Makefile:358: recipe for target 'check-recursive' failed make[1]: Leaving directory '/home/kkoenen/stt/kaldi-master/tools/openfst-1.6.2/src' Makefile:414: recipe for target 'check-recursive' failed </verbatim>

Same problem with openfst-1.6.3.

KyleGorman - 2017-08-17 - 17:06

We can fix this on our side but you don't need to run "make check", just "sudo make install" should suffice.

KyleGorman - 2017-08-18 - 15:03

Hi Kees, the error doesn't make a ton of sense to me. There is no function fst::Int64ToStr in OpenFst 1.6.3, as far as I can tell.

You may want to try doing a clean install (i.e., go to where OpenFst headers and source are installed and remove them, then install again).

You may want to try this outside of the Kaldi framework for installing OpenFst, since we don't maintain that, the Kaldi people do, and I don't know anything about it.

patch for failed cross-compile on openfst-1.6.3

BoonPangLim - 2017-08-03 - 14:02

Hi, I get an error linking when trying to compile using a cross-compiling toolchain:

<pre> make[4]: Entering directory '/scratch/bplim/crossbuild2/tmp/openfst-1.6.3/src/extensions/far' /bin/sh ../../../libtool --tag=CXX --mode=link armv8rpi3-novumind-linux-gnueabihf-g++ -std=c++11 -o farcompilestrings farcompilestrings.o libfstfarscript.la ../../script/libfstscript.la ../../lib/libfst.la -lm -ldl libtool: link: armv8rpi3-novumind-linux-gnueabihf-g++ -std=c++11 -o .libs/farcompilestrings farcompilestrings.o ./.libs/libfstfarscript.so ../../script/.libs/libfstscript.so ../../lib/.libs/libfst.so -lm -ldl -Wl,-rpath -Wl,/scratch/bplim/crossbuild2/tmp/build/rpi3/openfst/lib /home/share/crosstool/rpi3/lib/gcc/armv8rpi3-novumind-linux-gnueabihf/6.3.0/../../../../armv8rpi3-novumind-linux-gnueabihf/bin/ld: warning: libfstfar.so.8, needed by ./.libs/libfstfarscript.so, not found (try using -rpath or -rpath-link) ./.libs/libfstfarscript.so: undefined reference to fst::IsSTList(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' ./.libs/libfstfarscript.so: undefined reference to fst::IsSTTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' </pre>

I'm not sure if this is completely correct but this patch (followed by rerunning aclocal && automake && autoreconf) seems to fix it:

<pre>

openfst-1.6.3/src/extensions/far/Makefile.am 2017-06-27 13:12:19.000000000 -0700 +++ openfst-1.6.3/src/extensions/far/Makefile.new 2017-08-02 19:24:37.034355239 -0700

@@ -23,7 +23,7 @@ bin_PROGRAMS = farcompilestrings farcreate farequal farextract farinfo farisomorphic farprintstrings

-LDADD = libfstfarscript.la ../../script/libfstscript.la +LDADD = libfstfarscript.la libfstfar.la ../../script/libfstscript.la ../../lib/libfst.la -lm $(DL_LIBS) farcompilestrings_SOURCES = farcompilestrings.cc </pre> BoonPangLim - 2017-08-03 - 14:05 Update with cleaner versions: ERROR: <verbatim> make[4]: Entering directory '/scratch/bplim/crossbuild2/tmp/openfst-1.6.3/src/extensions/far' /bin/sh ../../../libtool --tag=CXX --mode=link armv8rpi3-novumind-linux-gnueabihf-g++ -std=c++11 -o farcompilestrings farcompilestrings.o libfstfarscript.la ../../script/libfstscript.la ../../lib/libfst.la -lm -ldl libtool: link: armv8rpi3-novumind-linux-gnueabihf-g++ -std=c++11 -o .libs/farcompilestrings farcompilestrings.o ./.libs/libfstfarscript.so ../../script/.libs/libfstscript.so ../../lib/.libs/libfst.so -lm -ldl -Wl,-rpath -Wl,/scratch/bplim/crossbuild2/tmp/build/rpi3/openfst/lib /home/share/crosstool/rpi3/lib/gcc/armv8rpi3-novumind-linux-gnueabihf/6.3.0/../../../../armv8rpi3-novumind-linux-gnueabihf/bin/ld: warning: libfstfar.so.8, needed by ./.libs/libfstfarscript.so, not found (try using -rpath or -rpath-link) ./.libs/libfstfarscript.so: undefined reference to fst::IsSTList(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' ./.libs/libfstfarscript.so: undefined reference to fst::IsSTTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' collect2: error: ld returned 1 exit status </verbatim> PATCH: <verbatim> openfst-1.6.3/src/extensions/far/Makefile.am 2017-06-27 13:12:19.000000000 -0700 +++ openfst-1.6.3/src/extensions/far/Makefile.new 2017-08-02 19:24:37.034355239 -0700 @@ -23,7 +23,7 @@ bin_PROGRAMS = farcompilestrings farcreate farequal farextract farinfo farisomorphic farprintstrings -LDADD = libfstfarscript.la ../../script/libfstscript.la +LDADD = libfstfarscript.la libfstfar.la ../../script/libfstscript.la ../../lib/libfst.la -lm$(DL_LIBS)

farcompilestrings_SOURCES = farcompilestrings.cc </verbatim>

Pre-determinization

MarinaPetkovic - 2017-07-31 - 04:52

Hi, I was wondering if there is an openfst implementation of an algorithm described in the paper "An Optimal Pre-Determinization Algorithm for Weighted Transducers" by Allauzen and Mohri?

KyleGorman - 2017-08-14 - 13:47

No sorry, but you could make your own?

There is (was?) an implementation in the old AT&T FSM tools.

How to properly link my own program with openfst's libfst.so?

WeiyiZheng - 2017-07-26 - 19:31

I encountered this problem when using fst/fst.h and libfst.so with my own c++ code. not sure what I did wrong, does anyone have a hint?

And this is the error <verbatim>

E0726 16:23:20.319480 640899 generic-register.h:71] GenericRegister::GetEntry: vector-fst.so: cannot open shared object file: No such file or directory E0726 16:23:20.319733 640899 fst.h:246] Fst::Read: Unknown FST type vector (arc type = standard): <unspecified> </verbatim>

Can you try using <fst/fstlib.h> istead?

How to static link openFST with -fPIC?

Hi, I'd like to compile openFST to produce a static link that would later be linked into a shared library. So I need to compile it with the -fPIC flag, but automake is really confusing and I don't know what to edit to accomplish that. Halp! BTW thank you for the amazing library <3

Label IDs for matchers

I want to use sigma matcher and add it to isyms.txt. Is there a special integer reserved for all the matchers like we have 0 reserved for epsilon?

KyleGorman - 2017-07-24 - 12:01

No, it just needs to not be 0 and to not overlap any other non-"special" labels. Sometimes I use 1, 2, 3 for phi, sigma, rho. sometimes I use -1, -2, -3 (though we don't do much testing for negative labels so bad things could happen).

KyleGorman - 2017-07-24 - 12:01

BTW, if you're working with fstspecial, there are specific flags for setting the special labels for the different types of matchers (you also have to select the matcher to convert to).

I selected the fst::RhoMatcher as the matcher type and it works with whatever label I provide for <rho>. Thanks Kyle

Pynini installation difficulties

LeahVelleman - 2017-07-08 - 20:49

I've successfully built and installed OpenFST. The cython commands listed here do not run successfully for me, giving the following errors:

> cython -3 --cplus -o pynini.cc pynini.pyx

Error compiling Cython file:

... if self._mode == b"r": return self._reader.far_type() elif self._mode == b"w": return self._writer.far_type() else: return "closed" ^

pynini.pyx:2426:13: Unicode literals do not support coercion to C types other than Py_UNICODE/Py_UCS4 (for characters) or Py_UNICODE* (for strings).

> cython -3 --cplus -o pywrapfst.cc pywrapfst.pyx

Error compiling Cython file:

... # Prints FST to stringstream, then returns resulting string. cdef fst.SymbolTable *ssymbols_ptr = NULL if ssymbols is not None: ssymbols_ptr = ssymbols._table cdef stringstream sstrm fst.PrintFst(deref(self._fst), sstrm, "<pywrapfst>", ^

pywrapfst.pyx:1717:42: Unicode literals do not support coercion to C types other than Py_UNICODE/Py_UCS4 (for characters) or Py_UNICODE* (for strings).

If I attempt to install Pynini anyway, python setup.py install runs without errors or warnings, but then attempting to use the code (either by importing it in python or by running python setup.py test) gives this error:

> python setup.py test running test running egg_info writing pynini.egg-info/PKG-INFO writing dependency_links to pynini.egg-info/dependency_links.txt writing top-level names to pynini.egg-info/top_level.txt reading manifest file 'pynini.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pynini.egg-info/SOURCES.txt' running build_ext copying build/lib.macosx-10.12-x86_64-3.6/pywrapfst.cpython-36m-darwin.so -> copying build/lib.macosx-10.12-x86_64-3.6/pynini.cpython-36m-darwin.so -> pynini_test (unittest.loader._FailedTest) ... ERROR

================================================================== ERROR: pynini_test (unittest.loader._FailedTest)

Ran 1 test in 0.001s

FAILED (errors=1) Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0> error: Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>

This is on MacOS 10.12.5, with XCode 8.3.3, and with python 3.6.1 and cython installed through macports.

LeahVelleman - 2017-07-08 - 21:18

Update: replacing the relevant "..." strings in the relevant .pyx files with b"..." strings allows cython to run without error messages, but after doing this and reinstalling pynini, python setup.py test gives the same error as before.

LeahVelleman - 2017-07-08 - 21:45

In fact, it looks like the cython errors and the conversion to python 3.6 is a red herring. When I do the whole process under python 2.7 (with the original, unmodified versions of those .pyx and .cc files) I get essentially the same error:

Traceback (most recent call last): File "setup.py", line 88, in <module> test_suite="pynini_test") File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/test.py", line 211, in run self.run_tests() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/test.py", line 234, in run_tests **exit_kwarg File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 94, in init self.parseArgs(argv) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 149, in parseArgs self.createTests() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 158, in createTests self.module) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName module = __import__('.'.join(parts_copy)) File "/Users/leahvelleman/Downloads/pynini-1.6/pynini_test.py", line 16, in <module> from pynini import * ImportError: dlopen(/Users/leahvelleman/Downloads/pynini-1.6/pynini.so, 2): Symbol not found: __ZN3re23RE23Arg12parse_stringEPKcmPv Referenced from: /Users/leahvelleman/Downloads/pynini-1.6/pynini.so Expected in: flat namespace in /Users/leahvelleman/Downloads/pynini-1.6/pynini.so

LeahVelleman - 2017-07-08 - 22:45

OK, got it sorted I think. The solution was to

1) Rebuild and reinstall re2 from source. 2) Switch a few "..." strings out for b"..." strings as described above. 3) Run cython to regenerate the files as described in README.rst. 4) Install and test Pynini.

I think step 2 would have been necessary for anyone trying to get this to work with python 3.6, but step 1 may have been due to some idiosyncrasy of my own setup. In any case, leaving this here in case someone else hits the same errors.

KyleGorman - 2017-07-24 - 11:58

Thanks Leah for the report there, I'll update the documentation for the next release. I wonder if I can simply eliminate all the "b"" literals and see if that works on both Python 2 and 3? (I love Python 3 but I develop mostly with 2.7 for ...employer reasons).

NatashaNayga - 2017-06-23 - 17:04

I have an fst file, but fstcompose doesn't like it. I made the fst file using fstinvert, but this returns a bad fst header error: ERROR: FstHeader::Read: Bad FST header: inverselexicon.fst<br> I currently have no experience with openfst, so any help would be appreciated.<br> Thanks!

KyleGorman - 2017-06-27 - 16:49

Where did the FST file come from?

Note that OpenFst data files are not guaranteed to be compatible across platforms, since some file formats are byte order-dependent. (And this is so we can memory-map them, which makes IO much faster.) If you want to share files across platforms, share the arc list produced by fstprint instead, and then use fstcompile to turn that into a platform-specific FST data file.

Incorrect C++ code

PaulHewlett - 2017-06-12 - 11:51

I have just compiled the library on alpine-linux and the gcc compiler is giving a lot of warnings at line 174 of arc-map.h.

The warning points out that some code is incorrectly indented inside an if statement giving the impression that it is wrapped by the if when - because no curly braces are present - it is not. My reading of the rest of arc-map.h implies that this is in fact an error and curly braces should be added. The offending call to fst->SetFinal() is in the MAP_REQUIRE_SUPERFINAL clause.

If not an error fst->SetFinal() should be suitably outdented

PaulHewlett - 2017-06-12 - 11:53

<verbatim> case MAP_REQUIRE_SUPERFINAL: { if (s = superfinal) { A final_arc = (*mapper)(A(0, 0, fst->Final(s), kNoStateId)); if (final_arc.ilabel = 0 || final_arc.olabel = 0 || final_arc.weight = Weight::Zero()) fst->AddArc(s, A(final_arc.ilabel, final_arc.olabel, final_arc.weight, superfinal)); fst->SetFinal(s, Weight::Zero()); } break;

</verbatim>

KyleGorman - 2017-06-27 - 16:52

That appears to be fixed internally, and should be correct in the next release. Thanks for the report.

How can I get SFST Library?

KyleGorman - 2017-05-30 - 14:36

I don't believe it's ready for release yet.

Non-functional FST at HCLG stage

SurbhiSinghal - 2017-05-02 - 17:35

For a simple corpus that has 2 sentences A B CD AB C D

I used kaldi's recipe till CLG FST compilation and then my code because my Acoustic Model is Sphinx format. For a simple case like the one I mentioned above, I get the following error at the determinization step: <verbatim>FATAL: StringWeight::Plus: unequal arguments (non-functional FST?)</verbatim> However, if I use simple work around of Encode->Determinize->Decode using Thrax's Optimize function , it finishes compilation. I wonder what trick (that probably kaldi uses) in taking care of such kinds of trigram pronunciations.(that it maps the 2 same trigram pronunciations to different acoustic ID.) Thanks,

I am not sure

SurbhiSinghal - 2017-05-02 - 17:47

I apologize, my corpus was : A B CD AB C D

KyleGorman - 2017-05-30 - 14:37

You'd have to ask at the Kaldi forums, but probably the same trick? It's a relatively well-known one.

Optimal use of ReplaceFst

PafnoutiT - 2017-04-26 - 12:25

I have a reasonably sized graph (about 100MB) on which I want to run a replace with a small one (100kB) on an output label that appears 3 times.

The baseline runtime is 16 seconds. fstreplace takes 4 seconds, so if I pipe this into my binary it takes a total of 20 seconds. If I use ReplaceFst in the C++ the total runtime is 22 seconds.

Is there a way of making ReplaceFst more efficient? One of the issues of using fstreplace is that on short tasks with a baseline time one 1 second, it adds a flat time of 4 seconds, whereas there is almost no overhead with ReplaceFst.

KyleGorman - 2017-05-30 - 14:40

fstreplace uses the scripting interface, which means that functions and classes have to be looked up in static registries, several layers of pointer dereferencing, and virtual dispatch, so that might be the additional time spent? ReplaceFst is also "on-the-fly" (lazy) whereas there is no reasonable way to use that from within a binary. So if you build a ReplaceFst but don't visit it all in the next step, it may do less work.

Installation Problem

YangYu - 2017-04-19 - 09:32

I am using CentOs6.7; gcc version 4.8.2; When I installed openfst 1.6.0 or 1.6.2: when I put ./configure , everything runs smoothly when I put make, it runs for a while and some error occurs:

Here are the errors:

libtool: link: g++ -std=c++11 -o .libs/fstarcsort fstarcsort.o ../script/.libs/libfstscript.so ../lib/.libs/libfst.so -lm -ldl -Wl,-rpath -Wl,/usr/local/lib g++: ../script/.libs/libfstscript.so: No such file or directory g++: ../lib/.libs/libfst.so: No such file or directory

KyleGorman - 2017-05-30 - 14:42

None of us have old CentOS machines so it's difficult to replicate this. Could you just try with a more recent GCC? (That's quite old by Linux standards.)

BurakBOELEK - 2017-03-29 - 04:13

Hello;

I just took over a project and I am trying to run a speech recognition. I am not sure if this is the correct place to ask, But I will try my chance...

My code is here :

<verbatim> commands.add("/bin/sh"); commands.add("-c"); commands.add(execPath + "add-deltas ark:" + tempKlasor + "/stereo.ark ark:- | " + execPath + "nnet-forward --feature-transform=" + modelPath + "/word_model/final.feature_transform " + "--no-softmax=true --class-frame-counts=" + modelPath + "/word_model/ali_train_pdf.counts " + "--use-gpu=no " + modelPath + "/word_model/final.nnet ark:- ark:- | " + execPath + "/decode-faster-mapped --acoustic-scale=0.1 --allow-partial=true --max-active=1000 " + "--beam=7 " + modelPath + "/word_model/final.mdl " + modelPath + "/phone_model/HCLG.fst ark:- ark,t:" + tempKlasor + "/stereo.hyp ark:- | " + execPath + "ali-to-phones --per-frame=true " + modelPath + "/word_model/final.mdl ark:- ark,t:" + tempKlasor + "/stereo.phones"); <verbatim>

I expect this code to create stereo.phones file. But it did not create the file. I checked the file permissions. They all seem ok.

I am not experienced in speech recognition I hope you can help me...

Thanks

mutable versions of classes in pywrapfst

DavidVanLeeuwen - 2017-03-28 - 08:53

Hello,

I am working with pywrapfst to make my fsts. When I have used a more complicated function, such as replace(), the fst (or the symbol tables) are no mutable anymore. I still need to add arcs to the fst. How can I create a fully mutable copy from an unmutable fst? SHould I re-create one arc-by-arc?

Cheers,

david

KyleGorman - 2017-05-30 - 14:58

Hi David, I can't replicate this. The return value of replace is pywrapfst._MutableFst (underlyingly it's an VectorFst), which supports the .add_arc(state, arc) method.

As for symbol table access, if you want to mutate the symbol tables attached to an FST in-place, use .mutable_input_symbols() and .mutable_output_symbols() methods to get mutable references into the underlying tables.

Or, somewhat more slowly, make copies of the current symbol tables with .isymbols().copy() or .osymbols().copy(), mutate these, and then assign them back using .set_input_symbols(syms) and .set_output_symbols(syms).

The only way for an FST to not be mutable in pywrapfst is if it was read from disk and it wasn't mutable when it was on disk either.

Compilation of Openfst 1.6.2 fails with Intel Compiler

PeterSmit - 2017-03-23 - 16:07

OpenFST compilation with intel compiler fails with the following error:

libtool: compile:  icpc -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -ipo -O2 -xSSE4.2 -fp-model precise -fp-model source -ansi-alias -std=c++11 -MT mapped-file.lo -MD -MP -MF .deps/mapped-file.Tpo -c mapped-file.cc -o mapped-file.o &#62;/dev/null 2&#62;&1
./../include/fst/vector-fst.h(693): error: a function cannot be both constexpr and virtual
constexpr uint32 Flags() const final { return kArcValueFlags; }
^
detected during:
instantiation of class &#34;fst::MutableArcIterator&#60;fst::VectorFst&#60;Arc, State&#62;&#62; [with Arc=fst::LogArc, State=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 707
instantiation of &#34;void fst::VectorFst&#60;A, S&#62;::InitMutableArcIterator(fst::VectorFst&#60;A, S&#62;::StateId, fst::MutableArcIteratorData&#60;fst::VectorFst&#60;A, S&#62;::Arc&#62; *) [with A=fst::LogArc, S=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 443
implicit generation of &#34;fst::VectorFst&#60;A, S&#62;::~VectorFst() [with A=fst::LogArc, S=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 443
instantiation of class &#34;fst::VectorFst&#60;A, S&#62; [with A=fst::LogArc, S=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 443
instantiation of &#34;fst::VectorFst&#60;A, S&#62;::VectorFst() [with A=fst::LogArc, S=fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;]&#34; at line 77 of &#34;./../include/fst/register.h&#34;
instantiation of &#34;fst::FstRegisterer&#60;FST&#62;::FstRegisterer() [with FST=fst::VectorFst&#60;fst::LogArc, fst::VectorState&#60;fst::LogArc, std::allocator&#60;fst::LogArc&#62;&#62;&#62;]&#34; at line 41 of &#34;fst.cc&#34;

Could this be fixed?


DanielRichardG - 2017-08-07 - 18:44

Hello PeterSmit,

Yes, I've encountered this too. The Intel C++ compiler appears to be a bit more strict on C++ compared to g++.

I researched this a bit at the time and found that this use of constexpr is incorrect. The following patch fixes the issue:

diff -ru openfst-1.6.2/src/include/fst/complement.h openfst-1.6.2/src/include/fst/complement.h
--- openfst-1.6.2/src/include/fst/complement.h   2017-03-13 15:54:39.000000000 -0400
+++ openfst-1.6.2/src/include/fst/complement.h   2017-06-08 19:15:38.128407003 -0400
@@ -246,7 +246,7 @@
pos_ = a;
}

-  constexpr uint32 Flags() const final { return kArcValueFlags; }
+  /*constexpr*/ uint32 Flags() const final { return kArcValueFlags; }

void SetFlags(uint32, uint32) final {}

diff -ru openfst-1.6.2/src/include/fst/vector-fst.h openfst-1.6.2/src/include/fst/vector-fst.h
--- openfst-1.6.2/src/include/fst/vector-fst.h   2017-03-13 15:54:43.000000000 -0400
+++ openfst-1.6.2/src/include/fst/vector-fst.h   2017-06-08 18:10:12.585208111 -0400
@@ -690,7 +690,7 @@
kNoOEpsilons | kWeighted | kUnweighted;
}

-  constexpr uint32 Flags() const final { return kArcValueFlags; }
+  /*constexpr*/ uint32 Flags() const final { return kArcValueFlags; }

void SetFlags(uint32, uint32) final {}



Coin

EditaStewart - 2017-03-23 - 07:08

I would need to see your grammar to know whether it's a bug in the grammar or a bug in Thrax itself. مواقع تعدين البيتكوين

KyleGorman - 2017-05-30 - 15:35

@PeterSmit we can fix this in the next version. It does appear to be against the spec.

Do not link against python libraries on mac

MichkaPopoff - 2017-03-21 - 18:42

Hi,

I am one of the maintainers of the homebrew package manager. We have a check for libraries that link against specific python libraries. This does not allow to run openfst with different Python versions on Mac (for example system Python 2 and brewed Python 2).

The fix is pretty easy, see the explanation the error message from our tests:

<verbatim> homebrew/science/openfst: * python modules have explicit framework links These python extension modules were linked directly to a Python framework binary. They should be linked with -undefined dynamic_lookup instead of -lpython or -framework Python. /usr/local/opt/openfst/lib/python2.7/site-packages/pywrapfst.so </verbatim>

I hope this can be fixed. If you need any help I can also have a look at the code. Thanks in advance.

KyleGorman - 2017-05-30 - 14:48

HI Michka, sorry we're just seeing this! I'd be glad to fix this but I'm not sure I understand the error. Or rather, I don't know what causes -lpython to be used, since this compilation is all done inside of several layers of autoconf magic (see openfst-1.6.2/src/extensions/python/Makefile.am). Could you perhaps suggest a patch to that file to address this problem? (please feel free to email me directly: my email address is in the README.)

Python extension error

WeijianLin - 2017-03-07 - 20:42

I successfully installed openfst It also passed 'make check'.

And this is what i got when i tried to install python extension: Collecting openfst Downloading openfst-1.6.1.tar.gz (223kB) 100% |████████████████████████████████| 225kB 1.7MB/s Installing collected packages: openfst Running setup.py install for openfst ... error Complete output from command /Users/weijian/anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-3hgZ8Q/openfst/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-MksT_q-record/install-record.txt --single-version-externally-managed --compile: running install running build running build_ext building 'pywrapfst' extension creating build creating build/temp.macosx-10.6-x86_64-2.7 gcc -fno-strict-aliasing -I/Users/anaconda2/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/anaconda2/include/python2.7 -c pywrapfst.cc -o build/temp.macosx-10.6-x86_64-2.7/pywrapfst.o -std=c++11 -Wno-unneeded-internal-declaration -Wno-unused-function In file included from pywrapfst.cc:461: /usr/local/include/fst/util.h:16:10: fatal error: 'type_traits' file not found #include <type_traits> ^ 1 error generated. error: command 'gcc' failed with exit status 1 I tried both --enable-python and --enable-far during configuration, still got the same error.

Any thoughts?

KyleGorman - 2017-05-30 - 15:36

I'd guess your compiler doesn't have full C++11 support, since it doesn't have the type_traits header. Upgrade and try again?

Dream

EditaStewart - 2017-03-05 - 16:32

In the shell you have the possibility to set up strings as ilabels and olabels (with isymbols.txt and osymbols.txt) But when I followed the QuickTour and wanted to do it all in the C++-Editor I couldn't find out, how to give the strings to an vectorfst. I can only give them integers as labels, because of the definition of stdarc. Would be glad if someone could point out that to me. PS Is it even effective to make it all in the C++-editor (I am using Eclipse)? تفسير الاحلام

KyleGorman - 2017-03-17 - 14:43

Hi Edita, there's no way to store a string inside of a "label", but if you want to keep track of which label integers go with which strings, use the fst::SymbolTable class. Say you've already populated the symbol table syms, then you can do something like:

Compilation on Ubuntu 16.10 (yakkety)

MoiriGamboni - 2017-02-28 - 03:59

Hello,

I am getting the following error when compiling version 1.6.1 on Ubuntu 16.10:

<verbatim> In file included from ./../include/fst/fstlib.h:72:0, from ./../include/fst/script/arciterator-class.h:10, from arciterator-class.cc:4: ./../include/fst/shortest-path.h: In function 'void fst::internal::SingleShortestPathBacktrace(const fst::Fst<Arc>&, fst::MutableFst<Arc>*, const std::vector<std::pair<typename Arc::StateId, long unsigned int> >&, typename Arc::StateId)': ./../include/fst/shortest-path.h:77:8: error: inconsistent deduction for 'auto': 'auto' and then 'int' for (auto state = f_parent, d = kNoStateId; state = kNoStateId; ^~~~ </verbatim>

Version 1.6.0 gives me a similar error at another location:

<verbatim> In file included from ./../include/fst/state-map.h:15:0, from ./../include/fst/arcsort.h:14, from ./../include/fst/accumulator.h:15, from ./../include/fst/label-reachable.h:14, from ./../include/fst/lookahead-matcher.h:17, from ./../include/fst/matcher-fst.h:14, from fst.cc:14: ./../include/fst/arc-map.h: In member function 'fst::GallicToNewSymbolsMapper<A, G>::ToArc fst::GallicToNewSymbolsMapper<A, G>::operator()(const FromArc&)': ./../include/fst/arc-map.h:968:14: error: inconsistent deduction for 'auto': 'int' and then 'auto' for (auto i = 0, p = state_; i < w1.Size(); ^~~~ </verbatim>

Version 1.5.4 seems to work correctly. Any ideas about how to fix this?

Thank you

MoiriGamboni - 2017-02-28 - 08:22

I changed auto to int and compilation finished without problems.

KyleGorman - 2017-03-17 - 14:45

This is a known compiler bug. We've worked around it in 1.6.2, BTW.

(BTW, not always safe to assume that int is the same as StateId. Use StateId instead.)

Nontrivial examples of the C++ interface

VictorMiller - 2017-02-02 - 16:19

Are there pieces of example code using the C++ interface to the library for non-trivial FSMs?

Also, how do I just create an acceptor (as opposed to a transducer) -- the StdArc takes output symbols and weights as arguments. Is there a variant which just takes input symbol and destination state? I noticed that there is no boolean semiring described. Do you just use the TropicalSemiring in that case?

VictorMiller - 2017-02-03 - 11:01

I've created a transducer and I want to minimize it. However, when I call Minimize I get the message: FATAL: Refusing to minimize a non-deterministic FST with allow_nondet = false

What is allow_nondet, and how to I set it to true?

KyleGorman - 2017-02-22 - 11:45

Hi Victor, yes we just use the tropical semiring, usually, though you could write your own.

You almost surely do not want to use allow_nondet (though you can set it with a flag invocation like --allow_nondet). Rather you want to (attempt to) determinize first. For some information on generic FST optimization, check out this snippet here:

compilation on iOS?

EstherJudd - 2017-01-11 - 04:53

Has anybody tried compiling openFST for iOS and using it in an app? I know this is Google code, so maybe not so likely ;). The compilation seems to go ok but when we use fst::Read to try and read in an FST we get an error:

It happens regardless of whether we have a vector FST or change it to a const FST using fstconvert. Do the models also have to be created using the same platform, or can we create them on linux and use on MacOS / iOS?

KyleGorman - 2017-01-23 - 12:57

Hi Esther, we do strive to have OpenFst compile on any platform with POSIX, C99, and C++11 support and that ought to include iOS. (I also note, without actually knowing anything for a fact, that Google.app for iOS supports voice search, and that probably means that OpenFst is inside it.) We do occasionally get patches for OpenFst on non-Linux platforms and incorporate them, too.

That error would appear to suggest it was unable to dlopen the SO for const FST support (libfst-const.so on my system). That usually occurs in turn when the LD_LIBRARY_PATH does not include whatever directory this is found in. To debug, you could try simply finding that SO and then making sure dlopen can find it.

(Out of curiosity, what happens if you

#include <fst/const-fst.h>

then use ConstFst::Read ?)

KyleGorman - 2017-01-23 - 12:59

Note also that we use memory-mapped IO in OpenFst (for efficiency reasons), which means that serialized FSTs are not portable across platforms due to differing byte orderings; however your error suggests that this isn't an issue here.

Looking for "Runtime Code" to simulate the application of a transducer

KennethRBeesley - 2017-01-09 - 20:00

Could some kind soul point me to any algorithmic "runtime code" that takes an OpenFST FST and an input string (basically a sequence of integers), and simulates the application of the FST to the input string and returns any results?

That is, straightforward application can be done by building the input string into an FST, composing the input-FST on the input side of the main FST, and then extracting the output projection of the resulting FST. But I'm looking for C++ runtime code that takes an input string, walks through the main FST algorithmically, without modifying it, collecting matching output strings, and finally returning the set of output strings related to the original input string.

Any help would be much appreciated.

KyleGorman - 2017-01-23 - 12:19

That sounds pretty difficult (seems like you'd have to reinvent large portions of composition?), but I too would love to see a worked example.

KennethRBeesley - 2017-01-25 - 18:07

Difficult but not impossible. Here's a rough outline of what would need to be done. Assume that the arcs of the FST are marked with ints representing Unicode code point values (both on the input side and the output side of each arc label). And assume that the input is a single string, coded as a sequence of ints representing Unicode code point values. And assume that the FST is determinized, with Tropical Semiring weighting. The algorithm would start at the Start state of the FST; let that be the initial Current State; and at the beginning of the Input Sequence. Let the Current Weight be initialized to zero, and let the Current Output Sequence be an empty sequence of int. In the simplest case. where the FST has no epsilons (zeros) on the input side, you would take the first int of the input sequence (let that be the Current Input Symbol) and use a Matcher object to select the set of exit arcs leading from the Current State that match the Current Input Symbol on the input side. If there is just one such matching exit arc, "consume" the input symbol, collect the corresponding output int (output symbol) on the matching arc and append it to the Current Output Sequence, and add the weight of the matching arc to the Current Weight, and let the destination state of the arc be the new Current State. (If there's no match, then Fail.) Proceed like that until you simultaneously run out of input symbols and reach a Final State---that's a Success. Then collect or output the Current Output Sequence and its weight. Complications: at any Current State, if there are exit arcs with Epsilon on the input side, they need to be "explored" without consuming the Current Input Symbol. At any Current State, if there are multiple exit arcs that match the Current Input Symbol, or multiple epsilon arcs, or at least one epsilon arc and at least one matching arc, then you have a Branch State, and the algorithm would need to store the current State (Current State, position in the Input Sequence, Current Output Sequence, Current Weight) for future backups, and iterate through those exit arcs, exploring each one individually. After any path being explored succeeds or fails, the algorithm would need to backup to explore other yet-unexplored exit arcs at Branch States, restoring the State at the Branch as appropriate and exploring all remaining exit paths. Other complications: input-side epsilon loops; epsilon arcs leading from Final States where input succeeded. The algorithm could be implemented as Depth-First or Breadth-First. (Any corrections and suggestions would be appreciated.)

Such algorithmic "runtime code" that simulates the composition of an input string and the extraction of the output-projection language was implemented in the Xerox Finite State Toolkit (lexc and xfst) and in Foma, a very good clone. And it was deemed in both projects to be much more efficient than doing the straightforward composition and output-projection extraction. The Foma runtime code is available, and I'm trying to get a handle on it. I worked at Xerox for years but never down at such low levels. Both Xerox and Foma implemented unweighted FSTs, so the code for OpenFST FSTs would need to be modified slightly to collect the weight along each path.

I had hoped that such algorithmic runtime code might already be available for OpenFST FSTs. Hence my posting. Any pointers to such existing code would still be much appreciated.

[After finding or writing such runtime code for lookup of a single input string in a single FST, the next step would be to allow multiple FSTs and algorithmically simulate the composition of those FSTs while looking up the input sequence. This would be useful in cases where the multiple FSTs, if straightforwardly composed, would blow up in size.]

Delayed composition via Python API

KilianGebhardt - 2017-01-06 - 15:57

I have a rather large FST F and want to compose it with an input string w. From the result I want to find the shortest path. For this I use in pynini: shortestpath(acceptor(w) * F) which takes long. I suspect that the composition is fully computed before the shortest path is computed. Is there some way to enforce delayed composition from Python?

KyleGorman - 2017-01-23 - 12:02

No, sorry, no easy way to expose delayed composition at the moment.

Just to play the devil's advocate:

* do you know for a fact that this would be faster with delayed computation of the cascade? (i.e., is it faster when you do it like that in C++?) I would not assume that it is necessarily so---just that it takes up less memory under the default garbage collection / cache settings; it depends. * Have you confirmed that it's the composition and not the shortest-path operation that is responsible for the slow computation? (i.e., by assuming the composition result and then doing shortestpath on that?) * Would calling .optimize() on F (or manually optimizing with epsilon-removal, determinization, minimization, etc.) help?

FST determinizability

MariyaCelinTA - 2016-12-21 - 00:33

I am working with WFST for modelling the pronunciation variations for dysarthric speakers. For my work I need to add multiple pronunciations for a single dysarthric speaker. I have tried them with HTK by adding multiple pronunciations in the dictionary, however the dictionary size was very large and it ended up in a poor performance. So, I thought of using weighted finite state transducers for modelling the pronunciation variations. I have added such pronunciation variations in my Lfst.txt. However during fstdeterminizestar, the L.fst and G.fst (bigram language model) are not composed because L.fst is not determinizable. So, I made the cyclic fst as an acyclic, then they got determinized, but during composition with H.fst (monophone acoustic model) I face the same issue. I get the following error doing fstdeterminize for my fst: FATAL: StringWeight::Plus: unequal arguments (non-functional FST?) w1 = 616 w2 = 719

I doubt whether, the multiple transitions introduced by me (to take care of the multiple pronunciation) at each state will be an issue? However, the multiple transitions have different outputs. I don't understand where did it go wrong. I always had a doubt that, Is HTK (by adding multiple pronunciations in dictionary) and WFST is the same. Or is WFST only a representation for speech recognition, is there any difference between HTK and WFST?

Can anyone please provide me assistance in adding the multiple pronunciation issues with Openfst.

KyleGorman - 2017-01-23 - 11:31

Hi Mariya,

Having multiple pronunciation variants in one's lexicon is a known problem in the literature on WFST speech recognition and OpenFst implements some solutions. FST disambiguation (http://www.openfst.org/twiki/bin/view/FST/DisambiguateDoc) is an algorithm specifically for making L \circ G determinizable and would seem to be applicable here.

The error you're getting from fstdeterminize indicates that the FST to be determinized is non-functional (in the sense that it is a one-to-many relation). fstdeterminize does support non-functional determinization, though this can be very slow; in general, determinization has poor guarantees: it is not guaranteed to terminate (the FST may have the "twins property": http://www.cs.nyu.edu/~mohri/pub/twins.pdf) and even for determinizable FSTs the complexity may be exponential in the size of the input.

If disambiguation doesn't help, here, you may want to see if you can use another trick: determinizing a weighted transducer as if it were an unweighted acceptor (http://www.cs.nyu.edu/~mohri/pub/dmk.pdf). This technique is used by fst::Optimize, a C++ function distributed with Pynini (pynini.opengrm.org) and Thrax (thrax.opengrm.org).

Both HTK and OpenFst use WFSTs, but I don't know much about the HTK internal representation thereof or how HTK handles multiple pronunciation variants.

Check if FST is empty?

EstherJudd - 2016-12-09 - 04:55

What is the best way to check in c++ whether the output of a composition between two FSTs results in a non-empty FST?

I tried with fst::Verify but it does not seem to give me the right answer.

EstherJudd - 2016-12-12 - 11:01

Another option might be to check if there is more than one state in the FST? I really need to find a fallback option when a composition between a pronunciation FST and a constraint grammar yields an empty FST.

EstherJudd - 2016-12-13 - 03:32

I found the solution. I can just check whether fst.NumStates() == 0.

KyleGorman - 2017-01-23 - 11:18

Hi Esther, that works for any subclass of ExpandedFst. Delayed FSTs (like ComposeFst) do not have a NumStates method, since we do not necessarily know the answer yet. A solution that works for all FST types (all subclasses of Fst) is to check if the Start() == kNoStateId. (which is -1).

changes in reader->GetFst from 1.5.2 to 1.5.3?

EstherJudd - 2016-09-29 - 06:10

I used to be able to get an FST from a FAR archive in c++ using

However with version 1.5.3 I get a compilation error and it works if I change to *fst but that seems to be incompatible with the older version that we still have running on our server. What is the best way to solve this apparent change (besides upgrading our server version)?

KyleGorman - 2016-10-11 - 18:52

This returns a pointer, not a reference, now, simply for cross-API consistency (other functions of the form "GetFst" all return a pointer as well). You're welcome to upgrade to any version >= 1.5.3 to get a version that returns a pointer. I can't think of a reasonable way to write code that can handle both, sorry.

EstherJudd - 2016-10-18 - 04:43

Ok, no problem. I will tell our software coders to update to the latest version of openfst.

Showing characters in the input label in PyFST

JoseOrtiz - 2016-09-28 - 19:41

I'm using the method linear_chain to accept a String. When I convert it into a fst binary to then into a DOT format, I get integers instead of the characters. Also, I have a SymbolTable for each of the corresponding letters being read.

What I need is to show the characters instead, be by the command line or by coding directly into Python. Sorry if this is not the place to ask. Any help or reference would be greatly appreciated.

EstherJudd - 2016-09-29 - 03:51

I have done it on the command line by typing:

fstdraw --isymbols=isyms.txt --osymbols=osyms.txt binary.fst binary.dot

I haven't been using this in python but maybe you can just do this?:

draw("binary.dot",input_symbols="isyms.txt",output_symbols="osyms.txt")

KyleGorman - 2016-10-11 - 18:55

The OpenFst developers don't contribute to pyfst, so you might be better off asking at their GitHub site: https://github.com/vchahun/pyfst

(Though note that project appears to have been abandoned: the last commit was 3 years ago, and it's unlikely to compile against any reasonable recent version of OpenFst).

JoseOrtiz - 2016-10-18 - 17:45

Thank you both. The suggestion by Esther worked perfectly. I started using PyFST instead of pywrapfst because I was unable to install it correctly in my laptop, however, I still wish to install it. Is the pywrapfst still being maintained?

Once again, thank you for taking the time to answer.

KyleGorman - 2017-01-23 - 17:24

pywrapfst is very much maintained; we use it every day. If you're having installation problems, does it compile?

+= and *= equivalence functions for weights

MaartenVanCasteren - 2016-08-08 - 11:17

Would it be possible to add functions to the weight templates that work equivalent to += and *= in C++, to avoid unnecessary copying? They could be .Times(const weight& w) and .Plus(const weight& w). The normal Times() and Plus() could then potentially be implemented using these.

KyleGorman - 2016-10-11 - 19:07

That certainly could be done. (Nit: you actually need separate LeftTimes and RightTimes methods, since some semirings are non-commutative as mentioned in http://www.openfst.org/twiki/bin/view/FST/FstWeightRequirements.) I'm not sure this is going to be worth much for the float weight types since everything ought to fit in CPU registers, and for composite weight types, I wonder if there's copy elision here. Do you know for a fact there's not? (I.e., what happens if you compile with -fno-elide-constructors on your system?)

Label with String

JoseOrtiz - 2016-07-15 - 12:23

Greetings,

I am currently trying to set the Labels to more than one character per Label. Would this be possible?

EstherJudd - 2016-08-23 - 08:33

I have a SymbolTable where the symbols sometimes consist of two characters, so this shouldn't be a problem. I would advise you to work with a SymbolTable that maps each label to a number and train the FST with the numbers instead of the labels.

pywrapfst SymbolTable allow_negative_labels flag?

LouisaConwill - 2016-07-11 - 15:52

Hello,

I am writing a Python application to traverse FSTs. I want to read in a binary FST (which does not have a symbol table associated with it), read in a symbol table, and set it as the binary FST's symbol table. The code that I've written looks like this:

This code works great when the symbol table does not have negative labels; however, it does not work when the symbol table has negative labels. The error I get is: "ERROR: SymbolTable::ReadText: Bad non-negative integer "-4"" (the symbol table I'm trying to read in has -4 as one if its labels).

Is there a way to set allow_negative_labels to true using either SymbolTable.read or SymbolTable.read_text? I know this is possible in C++ and when using the openfst command line tool, but I'm struggling to figure out how to do this in Python.

Thank you!

KyleGorman - 2016-10-11 - 19:16

There is a flag --allow_negative_labels for the binaries, but the ability to use them is not exposed to pywrapfst yet. I can do so, but I really wouldn't recommend using negative labels if you can possibly avoid it. I'll update the thread once/if I add this support.

KyleGorman - 2016-10-13 - 20:37

Okay, I've added the ability to specify that you want to permit negative labels for pywrapfst.SymbolTable.read_text. That'll be in the next release. But l I still strongly discourage using negative labels.

openfst 1.5.3 causes ld to segfault with gcc 4.7.2

DanielPovey - 2016-06-29 - 15:12

On our grid in JHU, when compiling OpenFst 1.5.3, we get a segfault in "ld". I have never heard of "ld" segfaulting before so something pretty funky must be going on. What compiler versions do you think it should work on?

jtrmal@a14 ~/soft/openfst-1.5.3/src/script /bin/sh ../../libtool --verbose --tag=CXX --mode=link g++ -std=c++11 -version-info 4:0:0 -o libfstscript.la -rpath /usr/local/lib arciterator-class.lo arcsort.lo closure.lo compile.lo compose.lo concat.lo connect.lo convert.lo decode.lo determinize.lo difference.lo disambiguate.lo draw.lo encode.lo encodemapper-class.lo epsnormalize.lo equal.lo equivalent.lo fst-class.lo info.lo intersect.lo invert.lo isomorphic.lo map.lo minimize.lo print.lo project.lo prune.lo push.lo randequivalent.lo randgen.lo relabel.lo replace.lo reverse.lo reweight.lo rmepsilon.lo script-impl.lo shortest-distance.lo shortest-path.lo stateiterator-class.lo synchronize.lo text-io.lo topsort.lo union.lo weight-class.lo verify.lo ../lib/libfst.la -lm -ldl libtool: link: rm -fr .libs/libfstscript.so.4.0.0 libtool: link: g++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.7/crtbeginS.o .libs/arciterator-class.o .libs/arcsort.o .libs/closure.o .libs/compile.o .libs/compose.o .libs/concat.o .libs/connect.o .libs/convert.o .libs/decode.o .libs/determinize.o .libs/difference.o .libs/disambiguate.o .libs/draw.o .libs/encode.o .libs/encodemapper-class.o .libs/epsnormalize.o .libs/equal.o .libs/equivalent.o .libs/fst-class.o .libs/info.o .libs/intersect.o .libs/invert.o .libs/isomorphic.o .libs/map.o .libs/minimize.o .libs/print.o .libs/project.o .libs/prune.o .libs/push.o .libs/randequivalent.o .libs/randgen.o .libs/relabel.o .libs/replace.o .libs/reverse.o .libs/reweight.o .libs/rmepsilon.o .libs/script-impl.o .libs/shortest-distance.o .libs/shortest-path.o .libs/stateiterator-class.o .libs/synchronize.o .libs/text-io.o .libs/topsort.o .libs/union.o .libs/weight-class.o .libs/verify.o -Wl,-rpath -Wl,/home/jtrmal/soft/openfst-1.5.3/src/lib/.libs ../lib/.libs/libfst.so -ldl -L/usr/lib/gcc/x86_64-linux-gnu/4.7 -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crtn.o -Wl,-soname -Wl,libfstscript.so.4 -o .libs/libfstscript.so.4.0.0 collect2: error: ld terminated with signal 11 [Segmentation fault] DanielPovey - 2016-06-29 - 19:40 BTW, this is on debian 7.10. Apparently OpenFst 1.5.3 says it should work from gcc version 4.7 which is what we are using. DanielPovey - 2016-06-29 - 20:25 I have found more information from valgrind and submitted a bug-report in bugzilla, I think it's a linux binutils bug. DanielPovey - 2016-06-29 - 21:40 I have documented in the last comment on https://sourceware.org/bugzilla/show_bug.cgi?id=20317 that the bug can be fixed by taking a certain line from a more recent binutils version 2.26 (this fix works for me; I have to compile binutils from source with the fix). So it looks like it's not a bug in OpenFst but a bug in binutils that it just happens to trigger. Our binutils is version 2.22. Dan Log In ShortestDistance very slow? PafnoutiT - 2016-06-21 - 11:42 I have an fst (actually a Kaldi lattice, it's an acyclic graph in the tropical semiring) on which I wanted to use fst::ShortestDistance to the final state, however it's very slow. It takes about 1.5 sec to process my lattice, whereas if I loop myself over the states and arcs to compute the same quantity it happens in 0.03 seconds. What could I have done wrong? KyleGorman - 2016-10-11 - 19:22 It's possible that automatic queue selection is failing because certain property bits are not yet set. You could try turning up logging to see what queue is selected (see http://www.openfst.org/twiki/bin/view/FST/FstAdvancedUsage#State_Queues), or if you're comfortable, could simply assert that the FST is (e.g.,) acyclic (and possibly topologically sorted) using SetProperties() before calling ShortestDistance. It is also possible that the explicit queue is less efficient than manual iteration for this simple case, in which case you should feel free to stick with your implementation. Log In Shortest distance very slow? PafnoutiT - 2016-06-21 - 11:41 I have an fst (actually a Kaldi lattice, it's an acyclic graph in the tropical semiring) on which I wanted to use fst::ShortestDistance, however it's very slow. It takes about 1.5 sec to process my lattice, whereas if I loop myself over the states and arcs to compute the same quantity it happens in 0.03 seconds. What could I have done wrong? Log In Python extension compilation error AlexBogatu - 2016-06-18 - 12:50 Hi, I am trying to use the python extension but I have some problems when I compile the source. I have done the following: - download openfst 1.5.3 - ./configure with --enable-far - make & make install - pip install openfst This last command fails with the following error: /usr/local/include/fst/script/fst-class.h:277:3: note: fst::script::FstClass::FstClass() FstClass() : impl_(nullptr) {} ^ /usr/local/include/fst/script/fst-class.h:277:3: note: candidate expects 0 arguments, 1 provided At global scope: cc1plus: warning: unrecognized command line option "-Wno-unneeded-internal-declaration" [enabled by default] error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 Am I missing something? Thank you! MarxerR - 2016-08-10 - 10:54 I get a similar error with clang: building 'pywrapfst' extension clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c pywrapfst.cc -o build/temp.macosx-10.11-x86_64-2.7/pywrapfst.o -std=c++11 -Wno-unneeded-internal-declaration -Wno-unused-function pywrapfst.cc:41102:22: error: no matching constructor for initialization of 'fst::script::FstClass' __pyx_v_tfst = new fst::script::FstClass(__pyx_v_self->_reader->GetFstClass()); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /usr/local/include/fst/script/fst-class.h:281:3: note: candidate constructor not viable: no known conversion from 'const fst::script::FstClass *' to 'const fst::script::FstClass' for 1st argument; dereference the argument with * FstClass(const FstClass &other) ^ /usr/local/include/fst/script/fst-class.h:383:12: note: candidate constructor not viable: no known conversion from 'const fst::script::FstClass *' to 'fst::script::FstClassImplBase *' for 1st argument explicit FstClass(FstClassImplBase *impl) : impl_(impl) {} ^ /usr/local/include/fst/script/fst-class.h:279:12: note: candidate template ignored: could not match 'Fst<type-parameter-0-0>' against 'const fst::script::FstClass *' explicit FstClass(const Fst<Arc> &fst) : impl_(new FstClassImpl<Arc>(fst)) {} ^ /usr/local/include/fst/script/fst-class.h:276:3: note: candidate constructor not viable: requires 0 arguments, but 1 was provided FstClass() : impl_(nullptr) {} ^ 1 error generated. error: command 'clang' failed with exit status 1 KyleGorman - 2016-10-11 - 19:24 There's no reason to call pip. The version up there is not compatible with arbitrary OpenFst versions. If you do --enable-python then make and make install, you have already installed pywrapfst for whatever version of OpenFst you're building. Log In Changing Stateid value. JoseOrtiz - 2016-06-18 - 12:30 I'm trying to read files in GFA format and then parse into an openfst format. In GFA format, each new line starts with a character that identifies what type of line it is. What I've done so far is if it starts with H, it ignore the rest of the line and go the next one. If it starts with S, it means it's a state, so it proceeds to identify each part of the line which consist of five strings including the S. After this, I add a state, but I want my state to hold the value of one of the strings identified in the line, which is a numerical value. As I understand, when you use an AddState, it return a Stateid which begins at zero, then increments by one. If it helps, I'm currently using a StdVectorFst object for the AddState function. I am reading about the SetState function from the vector_fst_base_impl class, would I be able to use this funcion to first get the StateId I want to change and then assign the new value? If so, would this cause any errors? Thanks. DoganCan - 2016-06-19 - 16:40 Hi Jose In OpenFst, state IDs are integers. They are used for uniquely identifying states. They are not tied to the states they identify. Most OpenFst operations do not preserve state IDs. Cheers, Dogan Log In fst/fstlib.h header not found PaulPoveda - 2016-05-31 - 10:31 Hello everyone, I tried to install OpenFst to use it with SphinxTrain and it seem to work but I got stuck with the following error : checking fst/fstlib.h usability... no checking fst/fstlib.h presence... no checking for fst/fstlib.h... no configure: error: fst/fstlib.h header not found But the OpenFst binaries are well installed in /usr/local/lib/fst and the header too in /usr/local/include/fst.. Can anyone give me a clue about the issue ? KyleGorman - 2016-10-11 - 19:26 Assuming that /usr/local/include/fst/fstlib.h exists and is readable for your user, you may need to ensure that that directory is in your include path. At the compiler level you will want to use the -I flag to pass the directory, but I'm not sure how to do this with SphinxTrain. There may be something wrong with your environment (this directory should very much be in your include path) or with the SphinxTrain configuration setup, but this doesn't seem to be an issue with OpenFst per se. Log In ArcIterator construction is very slow with delayed composition RyanPrenger - 2016-05-03 - 16:04 Hi. I've written a decoder loosely based on Kaldi's simple-decoder that is reasonably fast on an FST that was composed offline (lex + language model). When I try using the decoder on the FST created with delayed composition it's around 100 times slower. All the time seems to be spent in the ArcIterator construction steps. The language model is around 500 megs, and the lex is around 15 megs. Would one expect such a large slow down? How do people usually get decoders to perform with a large language model (the composition results in an FST about double the size)? KyleGorman - 2016-10-11 - 19:34 It could be quite slow to work with a delayed composition, particularly if the offline composition is slow. The former is used to implement the latter, and many tricks which speed up offline composition should make working with a delayed composition faster too. These include use of specific matchers (or modifying the inputs, e.g. with arc-sorting or setting the FST's properties) so that ComposeFst chooses the best matcher), state tables, and composition filters (see http://www.openfst.org/twiki/bin/view/FST/FstEfficiency), and selecting the best associativities for cascaded compositions. You might just try to make the offline composition fast and then port that solution to the delayed composition-based decoder. Log In Speed of weighted FST loading in c++ EstherJudd - 2016-04-12 - 04:28 I have created a large FST for g2p. Using this command I have been able to get some speed savings when loading. I apply this command in a shell script when building the FST. <br> <verbatim>fstconvert --fst_type=const --fst_align=true in.ofst out.ofst</verbatim> <br> I have also been able to reduce the size of my FST by creating a compressed version using: <br> <verbatim>fstcompress --gzip=true out.fst out_compr.fst</verbatim> <br> Now I read in my final FST in c++ but I use the StdVectorFst. I was reading that it is faster to use StdConstFst but when I use that, I get an error saying that that class is not derived from MutableFst. I use the following command to read in the compressed FST: <br> <verbatim> StdVectorFst *decompressed = new StdVectorFst(); Decompress(remap_fst_name,decompressed,true); </verbatim> <br> So my question is, how can I increase the load speed for this large FST? MichaelRiley - 2016-05-27 - 10:11 fstcompress is for small file footprint; expands on load and is not fast. constfst is a bit faster to load than vectorfst not memory-mapped and much faster memory-mapped (but then is demand paged as used). you can't mutate a constfst. EstherJudd - 2016-06-14 - 04:11 Thanks, I guess my question is if I convert a VectorFST to a constFST using the fstconvert command, what is the best way to load it faster in c++? I found this website that talks about this conversion and when I type fst-info it does say it is memory-mapped: http://www.lvcsr.com/openfst-133-mapped-files.html And if I did want to make the resulting FST smaller without losing accuracy, what other things can I do besides fstcompress? KyleGorman - 2016-10-11 - 19:38 One generic method for reducing the size (in terms of arcs and states) is to use the Optimize method in optimize.h (a header shipped with Thrax and Pynini). It merges identically-labeled multi-arcs, removes epsilons, and applies determinization and minimization where they are known to terminate. See the documentation there for the specific details. It does about as well as one can hope to do automatically without knowing how the FST will be used. Log In Possible bug in ReplaceFstMatcher? GautamT - 2016-02-25 - 22:59 ReplaceFstMatcher in replace.h instantiates MultiEpsMatcher for each of the component fst with mode kMultiEpsList and non terminal labels as possible epsilon. It does this for both MATCH_INPUT and MATCH_OUTPUT case. Under the condition ofMATCH_INPUT, this seems wrong to me. Let's look at the 2 possible scenarios under MATCH_INPUT. 1. epsilon on replace is false: Under this condition input labels continue to remain and do not become epsilons. Input symbols could represent a domain different from output symbols and therefore input labels with values matching those of the non terminals shouldn't be considered epsilons. Replace operation takes place based on the output label and not input label. Current behaviour would match self loop (kNoLabel) against these. Dropping this behaviour would also result in performance gain since one would not have to do a Find on the underlying machine for each of the non terminal label to match against self loop. 2. epsilon on replace is true: Under this condition, input label accompanying a non terminal output label is always re-written to epsilon irrespective of whether or not its value is also that of a non terminal. Matching self loop against input in this case is tricky because one would need to match against corresponding output label. To begin input labels are sorted to use this matcher, to match against output labels that are not sorted does not seem to be a good idea. My thoughts are that for MATCH_INPUT if epsilon on replace is false then we don't need to use MultiEpsMatcher for each underlying machine, or not perform AddMutlipleEpsLabel. If epsilon on replace is true then not use this matcher and the user wouldn't be able to take advantage of the option to not always cache. The other possible option would be to mandate that if epsilon on replace is true then input label match that of output label for non terminal arc and check this on detecting a non terminal label in the output arc. To confirm to myself that this is a bug, I constructed a root fst and a sub fst and created a replace fst that is used for a composition with another fst. The replace fst is the rhs fst or fst2 so that it is matched on the input side. The fsts are created in such a way as to expose this problem. The result was that if one were to not do a lazy replace, i.e. fully expand it and then use it for composition, one would get a different result from using a lazy replace or replace fst. For reproducibility, here are the fsts: root fst 0 1 1 5 1 sub fst 0 1 5 7 1 fst1 or lhs fst for composition 0 1 1 1 1 expanded fst (fstreplace root 10 sub 5 expanded) using 5 as the non terminal to refer to sub fst and root label being a non existent label (non recursive) 0 1 1 0 1 2 5 7 2 3 0 0 3 As can be seen the composition fst1 o expanded fst would result in an empty fst. However when not expanding and doing the composition it would result in a non empty fst with a final state since ReplaceFstMatcher would treat the input label 5 in the replace fst from the sub fst as epsilon and therefore composition proceeds. Am I missing something? GautamT - 2016-02-25 - 23:01 Somehow the fst formatting was off, fixing it. root fst 0 1 1 5 1 sub fst 0 1 5 7 1 fst1 or lhs fst for composition 0 1 1 1 1 expanded fst (fstreplace root 10 sub 5 expanded) using 5 as the non terminal to refer to sub fst and root label being a non existent label (non recursive) 0 1 1 0 1 2 5 7 2 3 0 0 3 Log In G.fst in ngram format work fail on arm JerryLin - 2016-02-20 - 09:13 Hi all, I built G.fst in ngram format on x86 , and run on x86. it works fine I built G.fst in ngram format on arm , and run on arm, it throws bad_malloc exception. if you built G.fst in vector format on arm, and run on arm, it works fine. How to solve it ? Thank you. MichaelRiley - 2016-05-27 - 10:13 Possibly changed endianity when crossing platforms. Fix - don't do that. KyleGorman - 2016-10-11 - 19:40 Right, because IO is memory-mapped, don't expect serialized FSTs to work on a platform with a different endianity. It's a small price to pay for faster IO. Log In How to compute edit distance of a given text line SopheaPRUM - 2016-02-18 - 20:50 Hello, Given a text line containing mistakes (Example "AL WUYS LO W PRICES", the correct text line should be "ALWAYS LOW PRICES"). Considering that i have a lexicon and language model. How can i compute the edit distance of this given text line? Please help Best regards KyleGorman - 2016-10-11 - 19:43 The last example here (http://www.openfst.org/twiki/bin/view/FST/FstExamples) involves making an edit transducer and then computing the edit distance. The paper on Pynini (https://aclweb.org/anthology/W/W16/W16-2409.pdf) has an example of using a noisy channel model and a language model to resolve this kind of ambiguity, as well. Log In TypeError: unsupported operand type(s) for -: 'int' and 'RestrictedProperty' MariamAAMER - 2016-02-04 - 13:01 Hi, I am trying to create a script. The part that doesn't work is the pasted below: class MyGrating(Structure): wg_W = PositiveNumberProperty(required = True) wg_L = PositiveNumberProperty(required = True) taper_L = PositiveNumberProperty(required = True) wg_W_Grating = PositiveNumberProperty(required = True) period_Grating = PositiveNumberProperty(required = True) DC_Grating = PositiveNumberProperty(required = True) n_periods_Grating = PositiveNumberProperty(required = True) width_Grating = PositiveNumberProperty(required = True) Out_wg = PositiveNumberProperty(default=1)# 1 si queremos que dibuje guia, 0 si no la queremos def define_elements(self, elems): inside the define_elements, i have a for loop, and inside it, i have todo make an arithmetic operation between two of the above defined properties. I get an error message that says: TypeError: unsupported operand type(s) for -: 'int' and 'RestrictedProperty' Somebody could help me to resolve it. Thanks to everybody Log In Linking problem on 64bit arch DodoIvanecky - 2016-01-08 - 09:54 I am trying to create a simple application with openFST. While fst.Write is working for my simple graph, functions like RmEpsilon or Determinize are not possible to link. In the first case I am getting: undefined reference to fst::ReverseProperties(unsigned long, bool)' and it is O.K. since: uint64 ReverseProperties(uint64 inprops, bool has_superinitial); But why/from where is fst::ReverseProperties(unsigned long, bool) called, I have no idea. Any hint is appreciated. DodoIvanecky - 2016-01-08 - 11:22 Looks like I see the same problem with several Property function. Calling Determinize I got undefined reference to fst::DeterminizeProperties(unsigned long, bool, bool)'. Which is O.K. because: uint64 DeterminizeProperties(uint64 inprops, bool has_subsequential_label, bool distinct_psubsequential_labels) DodoIvanecky - 2016-01-08 - 12:33 Solved. Problem is compiler. Needs to be at least 4.8. The reported problem was with 4.6 version. AnnaWarry - 2016-08-27 - 19:52 If you need install the las version visit http://www.csee.ogi.edu/~sproatr/Courses/TextNorm/tutorial.html this web have a great manual. Anna Warry Unix Administrator in https://www.good-fundraising-ideas.com/ Log In converting standard lattice format into and FST file MuhammadSalem - 2015-11-11 - 08:11 Hi i have Standard lattice format files, and i want to convert them into WFST to use them in Kaldi each file contains more than one NULL node, which were used in htk to decrease number of arcs, now if i want to convert those files to FST What should those null nodes be in the symbol file Not that i am working with the specific symbol file as input symbol file and output symbol file in order to finish the process Log In Minimization for fsts with negative weights -- I get weird results when I try to minimize a tropical weight fst with negative weights as shown below. Additionally, the minimization takes quite long (around 20 seconds) although the fst only has 2 states.  cat bar.txt
0      1      1      1      -5
1      1      2      2      -1
1      0
$fstcompile bar.txt bar.ofst$ fstminimize bar.ofst bar.min.ofst
$fstprint bar.min.ofst 0 1 1 1 -16777220 1 1 2 2 1 16777216  As far as I understand, this fst is already minimal so fstminimize shouldn't really do anything. If I convert all weights to positive weights, the problem vanishes $ cat baz.txt
0      1      1      1      5
1      1      2      2      1
1      0
$fstcompile baz.txt baz.ofst$ fstminimize baz.ofst baz.min.ofst

DoganCan - 2014-05-11 - 19:06

Hi Zeeshan,

Did you build the shared object lattice4-arc.so using Paul's gist? It seems like it is not in the working directory or in the dynamic library search path LD_LIBRARY_PATH (DYLD_LIBRARY_PATH on OS X).

Cheers, Dogan

ZeeshanAhmed - 2014-05-13 - 17:57

Hi Dogan,

Thanks for the reply. It worked fine on pure Linux system but on cygwin it's not working though the lattice4-arc.dll is in the working directory. The problem is that on cygwin shared libraries for openFst are not built. Any idea on how to build openFst shared libraries on cygwin?

Thanks, Zeeshan

DoganCan - 2014-05-13 - 18:26

Hi Zeeshan,

I have no idea how things work under cygwin. That being said, are you sure your problem is related to shared libraries not being built? I think your problem might have to do with the shared object file being named lattice4-arc.dll instead of lattice4-arc.so. Try renaming that file to lattice4-arc.so. If that does not work, try lattice4-arc.so.dll. Good luck.

Cheers, Dogan

Error installing openGRM on Mac 10.9.2

SepidehA - 2014-04-05 - 13:38

Hi all, I have installed OpenFST successfully on my mac 10.9.2 with --enable-far. But then when I want to configure my OpenGRM for installing I get " checking fst/fst.h usability... no checking fst/fst.h presence... no checking for fst/fst.h... no configure: error: fst/fst.h header not found" but my fst.h is there in /usr/local/include any ideas?

Thanks

CyrilAllauzen - 2014-04-07 - 14:53

Hi,

It seems this is due by Apple changing the default C++ standard library in the latest version of Xcode (version 5).

CXX=-stdlib=libstdc++ ./configure
should fix this.

EstherJudd - 2014-05-07 - 13:03

I have the same problem. The command above does not fix it. I had it working on a different machine previously but just started working on my iMac with 10.9.2 and the latest version of Xcode installed.

PaulDixon - 2014-05-08 - 01:15

The latest c++11 version of OpenFst (1.4.1) and OpenGrm Ngram (1.21) should just compile without any extra flags or environment variables.

EstherJudd - 2014-05-08 - 11:43

Then why when I run ./configure do I get the following messages? fat binaries are installed in /usr/local/bin and it is in my path

checking fst/fst.h usability... no checking fst/fst.h presence... no checking for fst/fst.h... no configure: error: fst/fst.h header not found

AaronDunlop - 2014-05-08 - 16:10

Here's what worked for me on Mavericks (10.9.2)

In the extracted opengrm-ngram-1.2.1 directory:

# Replace imports of 'unordered_map' with 'tr1/unordered_map' perl -pi -e 's#<unordered_map>#<tr1/unordered_map>#' src/include/ngram/ngram-shrink.h src/include/ngram/ngram-marginalize.h src/include/ngram/ngram-count.h

# Configure with Clang arguments ./configure --prefix=/usr/local CXX="clang++ -std=c++11 -stdlib=libstdc++"

make sudo make install

Note: To configure OpenFST, I used:

./configure --prefix=/usr/local --enable-compact-fsts --enable-const-fsts --enable-far=yes --enable-lookahead-fsts --enable-pdt CXX="clang++ -std=c++11 -stdlib=libstdc++"

Of course, your mileage may vary. But maybe someone will find that helpful.

CyrilAllauzen - 2014-05-12 - 10:55

Aaron: My original message was about compiling OpenGrm NGram 1.1.x. With version 1.2.1, you shouldn't need to -stdlib=libstdc++. Have you tried compiling without it?

Esther: Could you clarify which version of OpenFst and OpenGrm NGram you were using? Which version of Xcode? Also could you look at config.log and see if you're seeing anything suspicious there. Also, if you've recently installed Xcode5, you might need to accept the license: /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -license accept.

AaronDunlop - 2014-05-14 - 17:59

Cyril: Compiling OpenFST without -stdlib=libstdc++ seems to work, but configuring OpenGRM without that flag fails.

AliceWalton - 2014-06-13 - 00:41

I have the same problem when I try to configure Thrax. It shows configure: error: fst/fst.h header not found Using the flag CXX="clang++ -std=c++11" does not fix it.

In OpenFst, if I configure with CXX="clang++ -std=c++11 -stdlib=libstdc++" as suggested by Aaron, I got a fatal error of unordered_map not found. Well I can configure and install OpenFst without this flag but I cannot configure Thrax.

Thank you very much for your help.

JatishKhanna - 2015-09-26 - 05:33

In short: 1. Install - openfst-1.5.0 or higher version http://www.openfst.org/twiki/bin/view/FST/WebHome

2. Set the LD_LIBRARY_PATH=/usr/local/bin [all the library *.so by default to be installed ]

3. Clean up if already tried [ make clean ]

4. Try installing it back [make && make install]

walah Done

Intersection gives a FATAL error

ErikAxelson - 2014-03-07 - 04:03

Hi all,

I have two transducers with no symbol tables:

0       0       9      9      0
0       0       6      6      0
0       1       8      8      0
0       1       7      7      0
0       1       3      3      0
1       1       9      9      0
1       1       6      6      0
1       2       8      8      0
2       2       9      9      0
2       2       6      6      0
2       3       4      4      0
3       3       9      9      0
3       3       6      6      0
3       3       8      8      0
3       3       7      7      0
3       3       3      3      0
3       4       5      5      0
4       4       9      9      0
4       4       6      6      0
4       5       8      8      0
5       5       9      9      0
5       5       6      6      0
5       6       8      8      0
5       6       7      7      0
5       6       3      3      0
6       6       9      9      0
6       6       6      6      0
6       0


and

0       0       5      5      0
0       0       4      4      0
0       1       3      3      0
1       2       6      6      0
2       3       5      5      0
2       3       4      4      0
2       2       3      3      0
2       4       9      9      0
2       2       8      8      0
2       2       7      7      0
3       3       5      5      0
3       3       4      4      0
3       2       3      3      0
3       2       8      8      0
3       2       7      7      0
4       4       5      5      0
4       4       4      4      0
4       5       3      3      0
5       5       5      5      0
5       5       4      4      0
5       0


When I try to intersect them (t1 is the first transducer and t2 the second):

    RmEpsilon(t1);
RmEpsilon(t2);

ArcSort(t1, OLabelCompare<StdArc>());
ArcSort(t2, ILabelCompare<StdArc>());

EncodeMapper<StdArc> encoder(kEncodeLabels,ENCODE);
EncodeFst<StdArc> enc1(*t1, &encoder);
EncodeFst<StdArc> enc2(*t2, &encoder);

IntersectFst<StdArc> intersect(enc1, enc2);


I get an error message "FATAL: ComposeFst: 1st argument cannot match on output labels and 2nd argument cannot match on input labels (sort?)."

What am I doing wrong?

PaulDixon - 2014-03-08 - 09:10

You need to sort the encoded fsts before the intersection.

ErikAxelson - 2014-03-12 - 04:50

It seems that using Encode instead of EncodeFst solves the problem, because then I'm able to sort the arcs before intersection:

Encode<StdArc>(t1, &encoder); Encode<StdArc>(t2, &encoder);

ArcSort(t1, OLabelCompare<StdArc>()); ArcSort(t2, ILabelCompare<StdArc>());

IntersectFst<StdArc> intersect(*t1, *t2);

OpenFst website back up

CyrilAllauzen - 2014-02-14 - 18:20

The OpenFst website was down for a few days due to some hardware issue. The site is back up but some things (such as the doxygen links) are not working properly yet. casino offers We're working on resolving the remaining issues.

OliverAdams - 29 Jan 2014 - 21:09

I've built OpenFst in my home directory.

I'm aiming to compile the following toy code:

1 #include <fst/fstlib.h> 2 3 int main(void) { 4 fst::StdVectorFst fst; 5 }

So far I've run a number of commands that are variations of the following theme:

I get various undefined reference errors. I feel I should probably have append the commands with -l , but for the obvious variations I get errors along the lines of:

/usr/bin/ld: cannot find -lopenfst collect2: error: ld returned 1 exit status

How can I get it to compile? Any help would be greatly appreciated.

Bug in weight-class.h: ConvertKeyToSoFilename() ?

FedericoFlego - 26 Jan 2014 - 06:54

Hello,

I've got a problem running the fstprune binary with my user defined arc type.

It works fine if I do something like:

> fstprune myfst.fst > /dev/null

but if I do

> fstprune --weight=8 myfst.fst > /dev/null

I get

ERROR: GenericRegister::GetEntry : tropical_LT_tropical.so: cannot open shared object file: No such file or directory Segmentation fault

I believe this is due to function ConvertKeyToSoFilename() in weight-class.h:

virtual string ConvertKeyToSoFilename(const string &key) const { return key + ".so"; }

which should probably be:

virtual string ConvertKeyToSoFilename(const string& key) const { string legal_type(key); ConvertToLegalCSymbol(&legal_type);

return legal_type + "-arc.so"; }

Thanks a lot for your feedback!

Federico

DoganCan - 27 Jan 2014 - 03:19

Hi Federico,

I believe, you need a shared object for your user defined weight type (see attached code) as well. Make sure the the DSO is named tropical_LT_tropical.so and put it somewhere dlopen can find.

Also, you will get the following error with that fstprune call since 8 can not be parsed as a pair weight.

fstprune --weight=8 test.fst > /dev/null
FATAL: StrToWeight: Bad weight = "8", source = WeightClass, line = 0


fstprune --weight=8,0 test.fst > /dev/null


Cheers, Dogan

#include <fst/arc.h>
#include <fst/script/weight-class.h>

using namespace fst;

namespace fst {
namespace script {
REGISTER_FST_WEIGHT(LexicographicWeight<TropicalWeight, TropicalWeight>);
}
}


FedericoFlego - 27 Jan 2014 - 08:40

Thanks a lot Dogan!

Actually I've got the shared object (named tropical_LT_tropical-arc.so) which is loaded correctly when I do for example fstprint. I see in the code DSOs are generally loaded adding the suffix -arc:

legal_type + "-arc.so"

It's only in src/include/fst/script/weight-class.h

that the suffix is not added. After changing function ConvertKeyToSoFilename() in weight-class.h from:

  virtual string ConvertKeyToSoFilename(const string &key) const {
return key + ".so";
}


to:

  virtual string ConvertKeyToSoFilename(const string &key) const {
//return key + ".so"; //ff257: is this a bug?
string legal_type(key);
ConvertToLegalCSymbol(&legal_type);

return legal_type + "-arc.so";
}


it worked fine. Thanks for confirming that I need to add REGISTER_FST_WEIGHT. I didn't do it initially because it doesn't seem to be mentioned in the on-line documentation, but after a grep of the src for REGISTER_FST I had found it. Then yes, the last missing bit was to pass 8,0 rather than 0. I've just tried and it works perfectly!

Thanks a lot!

Federico

FedericoFlego - 27 Jan 2014 - 08:46

Apologies for the bad formatting... And I don't see an option to fix it

DoganCan - 28 Jan 2014 - 03:55

Hi Federico,

I am glad it worked out. Regarding the change you made in the source code, I don't think it is a bug. OpenFst binaries can dynamically load user defined weight types just like user defined arc types. Even though in practice these two mean more or less the same thing, technically these are two different things. In this particular case, fstprune is looking for the user defined weight type (not the arc type) to parse the command line weight parameter. That is why you don't see the same error when you call fstprint (or when you call fstprune without the --weight option) which only needs to load the user defined arc type.

I believe the intended way of handling this issue is by creating another DSO named tropical_LT_tropical.so (in addition to tropical_LT_tropical-arc.so) which only registers the weight type. Patching the library code is not a great idea if you plan to update the OpenFst library later when a new version comes out or if you plan to run your code on different machines -- you will have to remember patching the library -- or if you plan to distribute your code.

Cheers, Dogan

Is there a general approach to eliminating redundant (higher-weighted) paths in cyclic WFSAs?

AlecS - 16 Jan 2014 - 05:36

Given a simple tropical semiring WFSA, defined by the expression A* | (A / 1)* | (B /2)

Is there any general way to reduce this to the semantically equivalent WFSA A* | (B /2)

In other words, how to remove redundant higher-weighted paths in tropical semiring WFSAs with weighted cycles?

Does anyone have a solution to this?

Thanks.

DoganCan - 18 Jan 2014 - 21:12

Hi Alec,

I don't think there is a general solution to your problem provided by the public release of OpenFst at this time. Determinization will do what you want as long as the input automaton has the twins property. The example you gave does not have this property, hence determinization will not work in this case.

Having said that, conclusion section of this paper alludes to a general disambiguation algorithm for weighted automata which does not require the twins property to hold. I believe there is a good chance that algorithm will be a part of the public release of OpenFst once it is published, but this is pure speculation on my end.

Cheers, Dogan

AlecS - 04 Feb 2014 - 05:26

Hi Dogan, Thanks for your input.

Yes I agree, if someone could devise a working disambiguation algorithm for weighted FSAs, that would do the trick.

Also, a determinization algorithm that would determinize all determinizable WFSAs, despite the lack of twins property in certain cases (like the example above), would suffice in my case.

Let’s hope your speculation is correct.

Thanks, Alec

PaulDixon - 07 Feb 2014 - 01:01

mifst toolkit has a determinization algorithm that tries to detect self loops with different weights.

Infinity weight in compiled fst

HuanliangWang - 30 Dec 2013 - 23:45

I get Infinity weight in final state when I compile a text fsm by the command "fstcompile --arc_type=standard --fst_type=const infsm | fstprint > outfsm". The file "infsm" is very large, about 20G. I find a line "3 Infinity" in the file "outfsm". Is this an error? How can I solve it?

Why do I get Infinity weight in final state?

HuanliangWang - 30 Dec 2013 - 23:36

I get a Infinity weight in final state when I compile a text fst into binary format by the command "fstcompile --arc_type=standard --fst_type=const infsm > outfst". The "infsm" file is large fsm. Then I convert the "outfst" into text format by command "fstprint". I find there is a line "3 Infinity" in the converted fsm file. Is the case a error? How can I solve it?

PaulDixon - 31 Dec 2013 - 00:41

What does fstinfo report? Sounds like you have coaccessible states (states not reachable from a final). If the machine is connected the number of accessible and coaccessible states reported by fstinfo should be the same. You can remove the dead end states with fstconnect.

HuanliangWang - 01 Jan 2014 - 21:45

Hi, Paul, I get the fst information by "fstinfo". It is as following: fst type const arc type standard input symbol table none output symbol table none # of states 425041220 # of arcs 1230392842 initial state 0 # of final states 0 # of input/output epsilons 425041217 # of input epsilons 425041217 # of output epsilons 425041217 # of accessible states 425041220 # of coaccessible states 0 # of connected states 0 # of connected components 1 # of strongly conn components 43672597

The number of accessible and coaccessible states is not same.

HuanliangWang - 01 Jan 2014 - 21:51

sorry! It is like as: fst type const

arc type standard

input symbol table none

output symbol table none

# of states 425041220

# of arcs 1230392842

initial state 0

# of final states 0

# of input/output epsilons 425041217

# of input epsilons 425041217

# of output epsilons 425041217

# of accessible states 425041220

# of coaccessible states 0

# of connected states 0

# of connected components 1

# of strongly conn components 43672597

PaulDixon - 02 Jan 2014 - 01:50

Your machine doesn't have final states at at all and is probabaly useless. You could try generating a tiny FST with the same procedure and viewing it using fstdraw to check the structure is what you expect.

HuanliangWang - 06 Jan 2014 - 06:50

thank you!

I find there are two end states in input fsm "infsm". It is unexpected. I will check the error firstly.

error in ubuntu 13.10

YanWang - 19 Dec 2013 - 22:02

../script/.libs/libfstscript.so: undefined reference to dlopen' ../script/.libs/libfstscript.so: undefined reference to dlerror' collect2: ld returned 1 exit status

the openfst-1.3.4 compile error in ubuntu.How can I solve it?

PaulDixon - 20 Dec 2013 - 04:44

Seems the -ldl linker switch is missing or need the order changing. This has been discussed in earlier posts.

FalouuFalouu - 20 Jan 2014 - 06:10

I found a solution (Ubuntu 13.10): use gcc-4.4 and g++-4.4 instead of 4.8.
1. sudo apt-get install gcc-4.4 g++-4.4
2. Temporarily link gcc to gcc-4.4 (and g++ ...) or replace all occurences of gcc and g++ in Makefiles to 4.4 version. Maybe it is not all necessary, but I did it and it worked.

DavidLandan - 22 Jan 2014 - 15:26

I'm seeing the error as well, and various orderings of the -lm & -ldl tags don't seem to change things. Unfortunately, while using gcc/g++ v4.4 is a workaround, you may run into issues later on with other tools that make use of the openfst library (as I discovered).

My src/bin/Makefile.am only has one line with "LD" in it: LDADD = ../script/libfstscript.la ../lib/libfst.la -lm -ldl

putting the -lm & -ldl switches in the middle or end doesn't matter:

libtool: link: g++ -O2 -o .libs/fstarcsort fstarcsort.o -lm -ldl ../script/.libs/libfstscript.so ../lib/.libs/libfst.so ../script/.libs/libfstscript.so: undefined reference to dlopen' ../script/.libs/libfstscript.so: undefined reference to dlerror' collect2: error: ld returned 1 exit status

libtool: link: g++ -O2 -o .libs/fstarcsort fstarcsort.o ../script/.libs/libfstscript.so ../lib/.libs/libfst.so -lm -ldl ../script/.libs/libfstscript.so: undefined reference to dlopen' ../script/.libs/libfstscript.so: undefined reference to dlerror' collect2: error: ld returned 1 exit status

Has anyone else found a way to work through this using gcc/++ tools of v4.7 or later?

YendaTrmal - 31 Jan 2014 - 19:35

Ok, I was able to figure it out, perhaps... Not really sure if that is the correct solution either. I don't really have experience with autotools and such, so I had to mess with everything and I'm not sure in which sequence the things have to done, but everything boils down to changing the Makefile.am files containing the LDADD to -LDADD = ../script/libfstscript.la ../lib/libfst.la -lm -ldl +LDADD = ../script/libfstscript.la ../lib/libfst.la +AM_LDFLAGS = -Wl,-no-as-needed -lm -ldl

YendaTrmal - 31 Jan 2014 - 19:36

Sorry, wrong formatting

-LDADD = ../script/libfstscript.la ../lib/libfst.la -lm -ldl

+AM_LDFLAGS = -Wl,-no-as-needed -lm -ldl

StevenBedrick - 2014-02-16 - 16:12

First, I think there's a slight formatting bug above- from what I can tell, "no-as-needed" is a two-dash argument (i.e., it should be "-Wl,--no-as-needed" (note the two dashes before no-as-needed)).

Second, editing the relevant Makefile.ac files and making this change does in fact result in a successful compilation on modern Ubuntu/GCC combinations, but the resulting binaries ("fstinfo", etc.) still aren't being linked properly against libfst and libfstscript. So, the result of running "make install" is a bunch of broken binaries.

I've poked at it for a while and not had any luck fixing the problem; somebody with stronger automaker-fu than myself might have more luck.

StevenBedrick - 2014-02-17 - 16:01

OK, never mind- I was correct that the issue was with my automake-fu. The problem was twofold: first, I wasn't actually telling automake to regenerate the Makefile.in files from the modified Makefile.am files, and second, once I figured out that that's why my changes to the Makefile.am files weren't having an effect, I next discovered that modern versions of Ubuntu come with a different version of automake/autoconf than the OpenFST distribution was created with. So, to get OpenFST built and installed correctly, here's what you'll need to do:

First you have to modify the relevant Makefile.am files as described above by YendaTrmal (taking into account the typo I identified in my previous comment vis-a-vis double-dashed arguments).

Second, you need to re-generate all of the automake-related files. I did this by running "autoreconf --force --install", which might be more than was needed- see my earlier comments about my lack of automake-fu. Before you can do this, however, you'll need to modify configure.ac in the top-level distribution directory; the AM_INIT_AUTOMAKE directive has a flag that treats all warnings as errors, and the differences in the automake/autoconf versions causes several warnings that will prevent autoreconf from running correctly. The "solution" was to get rid of the "-Werror" flag in the AM_INIT_AUTOMAKE directive. None of the errors seemed particularly dire, but take that observation with a grain of salt (see earlier comments about my lack of automake-fu). Removing "-Werror" caused all the relevant Makefile.in files to be properly regenerated, with the corrected LDADD/AM_LDFLAGS variables.

Third, "./configure"/"make"/"make install" as normal.

That seems to do the trick- the resulting binaries are all properly linked now.

StevenBedrick - 2014-02-17 - 19:55

One more update- apologies to YendaTrmal, "no-as-needed" seems to work just fine as a single-dashed argument. I must have had some other typo going on when I was having trouble with it earlier. disregard that part of my comment.

MathieuMorey - 2014-02-25 - 11:53

The solutions above did not really work for me, but they were a great starting point (thanks a lot, YendaTrmal and StevenBedrick !). I managed to get openFST to compile, install and work as a shared library on Ubuntu 13.10 using the following steps:

1. in "src/lib/Makefile.am", append: libfst_la_LIBADD = -ldl

2. in "configure.ac", replace: AM_INIT_AUTOMAKE([foreign nostdinc -Wall -Werror]) with: AM_INIT_AUTOMAKE([foreign nostdinc -Wall -Wno-extra-portability -Werror])"

3. autoreconf --force --install

then the usual ./configure && make && make install .

You can check it works by running "make check".

fstequal command

LahiruSamarakoon - 19 Dec 2013 - 07:30

Hi all, What is the current status of the "fstequal" command? Is it working properly. My attempts to use it was unsuccessful. Why this command is not listed in the website?

Thanks

PaulDixon - 20 Dec 2013 - 04:34

It goes through two machines and does a state by state and and an arc comparison, setting --v to 1 should give detailed output. There seems to be typo in line 97 of the code

FalouuFalouu - 20 Jan 2014 - 06:09

I found a solution (Ubuntu 13.10): use gcc-4.4 and g++-4.4 instead of 4.8. 1. sudo apt-get install gcc-4.4 g++-4.4 2. Temporarily link gcc to gcc-4.4 (and g++ ...) or replace all occurences of gcc and g++ in Makefiles to 4.4 version. Maybe it is not all necessary, but I did it and it worked.

FalouuFalouu - 20 Jan 2014 - 06:12

Sorry, i accidentally put response in wrong place. Please delete.

Visual Studio OpenFST 1.3.4

RolandSchwarz - 02 Dec 2013 - 05:03

Does anyone have a working solution for compiling OpenFST 1.3.4 on Visual Studio? I found this project here

Any help would be appreciated.

Thanks,

Roland

Compiling errors on Mac Mavericks 10.9

CesineC - 22 Nov 2013 - 11:23

Has anyone gotten OpenFST to compile on OS 10.9? Here are the errors I'm getting:

$./configure --prefix=$OPENFST_HOME --enable-far --enable-pdt &&
make clean
make &&
make install || {
echo "Error building openfst.";
exit 1;
}
....

In file included from fst.cc:21:
In file included from ./../include/fst/fst.h:34:
In file included from ./../include/fst/arc.h:28:
In file included from ./../include/fst/expectation-weight.h:36:
In file included from ./../include/fst/pair-weight.h:29:
In file included from ./../include/fst/weight.h:82:
#include <tr1/unordered_map>
^
1 error generated.
make[3]: *** [fst.lo] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
Error building openfst.

$g++ --version Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Target: x86_64-apple-darwin13.0.2 Thread model: posix  KyleGorman - 22 Nov 2013 - 13:25 I have. My understanding is that tr1 isn't part of any standard, and Apple moved it to a place where autoconf isn't looking for it. You need to set CPPFLAGS to point to that header and LIBS to point to that shared library. See: CesineC - 26 Nov 2013 - 14:15 Hi Kyle, I actually found your brew example which helped me get it to work, thats why i posted on your PR for brew also so that maybe they would accept that the current brew formula wasn't working, and that we might have to use libstdc++... I was hoping that we weren't the only ones using mavericks, but it looks like we are. For those who run into this later, here is some bash which (so far) appears to work well on mavericks:  ./configure --prefix=$OPENFST_HOME  --enable-compact-fsts --enable-const-fsts --enable-far=yes --enable-lookahead-fsts --enable-pdt CXX="clang++ -std=c++11 -stdlib=libstdc++" &&
make clean
make &&
make install || {
echo "Error building openfst.";
exit 1;
}


Availability of linear classifier/tagger FST extension

DavidHugginsDaines - 19 Nov 2013 - 13:25

I notice that there is a section on "Linear FSTs" in the FstExtensions page, however in the 1.3.4 release the referenced header files and fstlinear program aren't present. Is this available elsewhere, or likely to be released soon?

Thanks!

How to compile on 64 bit machine

ErrorNotFound - 22 Oct 2013 - 22:34

There seems to be only one OpenFST package for download. In other places I have seen separate packages for 32 bit and 64 bit. If I have a 64 bit machine, how do I install OpenFST there?

Thanks a lot.

kPosInfinity in old version of fst

ScottLawton - 11 Oct 2013 - 19:32

I'm trying to compile some code (http://bitbucket.org/yotaro/srtk) that apparently depends on an older (unstated) version of fst. It references kPosInfinity:

return LogLatticeWeight(fst::FloatLimits::kPosInfinity, fst::FloatLimits::kPosInfinity);

But, openfst-1.3.3/src/include/fst/float-weight.h just defines PosInfinity (no 'k' prefix).

I tried a superficial fix: deleting the 'k' in srtk -- but that yields "no matching function for call to ‘srtk::LogLatticeWeight::LogLatticeWeight(const float (&)(), const float (&)())’".

I'm a Python guy not a C++ guy ... any suggestions? I assume there's a way to cast it to (float, float), or a wrapper of some sort.

Thanks.

PaulDixon - 12 Oct 2013 - 01:58

You need to add the template parameter and parentheses

fst::FloatLimits<float>::PosInfinity()


Probably, bug: arc-map.h, ArcMap(), case MAP_REQUIRE_SUPERFINAL

NickolayMerkin - 29 Aug 2013 - 05:01

I've done a static analysis of source code with PVS-Studio, and it has reported a probable bug.

V640 The code's operational logic does not correspond with its formatting. The statement is indented to the right, but it is always executed. It is possible that curly brackets are missing. arc-map.h 178

if (final_arc.ilabel = 0 || final_arc.olabel = 0 || final_arc.weight = Weight::Zero()) fst->AddArc(s, A(final_arc.ilabel, final_arc.olabel, final_arc.weight, superfinal)); fst->SetFinal(s, Weight::Zero());

fst-AddArc is conditional (then-branch of the preceded if) fst-SetFinal is unconditional, though it is indented as if it is conditional too.

Optimization issue: functions Times and Divide on logartithmic and tropical weights

NickolayMerkin - 29 Aug 2013 - 04:54

Hello, I've run my application (LV CSR decoder) that uses OTF WFST composition under the AMD profiler and found that much time is spent in two functions: inline TropicalWeightTpl Times and Divide. They do costy checks for NaN and +/-Inf before the arithmetic operation (+ for times, - for divide). But it's obvious that nan+anything is nan, inf+anything is inf, and inf-inf is nan. Thus, these pre-checks are redundant.

When I've eliminated these checks, the speed of my application has been increased for 5%.

PaulDixon - 29 Aug 2013 - 22:32

Just out of interest are using the TropicalWeight type every where in your application? Or just as part of the composition and ArcIterator interface and then float and + operator everywhere else? Thanks.

AlexZatv - 11 Sep 2013 - 15:15

It's almost everywhere in application. Looks like most decoders use log scores, but we with Nickolay use weights.

Which optimization is applied to OpenFst 1.3.3 ?

JihoonKim - 18 Aug 2013 - 21:28

Hi,

OpenFst 1.3.3 is much faster than OpenFst 1.3.1. While fstdeterminize of 1.3.1 takes about 1 hour, that of 1.3.3 takes only less than 1 minute! How can 1.3.3 be this faster? Could anyone explain the reason?

Thanks.

AlexZatv - 19 Aug 2013 - 09:56

Hello! Are you talking about Windows version, or linux? There was some problems in windows standard library (std::hash_map is too slow,boost::hash_map is much faster)

JihoonKim - 20 Aug 2013 - 04:16

Yes, I meant Windows version. It's because of windows standard library ! Thanks for the quick response.

Can I get OpenFst 1.3.3 Visual Studio Project

JihoonKim - 15 Aug 2013 - 04:47

Hi,

I have to build OpenFst in Windows. In the download page, Visual Studio project is just for 1.1 and 1.3.1. It seems that latest version 1.3.3 is much faster than 1.3.1 so I need 1.3.3 to be built in Windows.

Is there anyone who has 1.3.3 for windows ?

PaulDixon - 15 Aug 2013 - 19:14

https://dl.dropbox.com/u/321851/openfst-1.3.3.zip

The memory map FSTs compile but aren't tested. The ngram FSTs aren't finished that why I never uploaded to the contrib section. Also includes VS2012 solutions.

JihoonKim - 16 Aug 2013 - 01:15

Thanks for your quick response. It'd be great help for me.

Merging symbol tables

PrashantMathur - 26 Jul 2013 - 10:28

Hi,

I am a newbie to openfst, so this can sound stupid.

Is there a function which can merge two SymbolTables? I was using A->AddTable(SymbolTable* B) , where I want to merge B in A. function to merge two symbol tables. But I believe AddTable function just appends table B with A and starts indexing with the maximum key. ( I don't know if I am correct).

A 0 a 1 b 2 c 3

B 0 d 1 a 2 b 3

C - Resulting Table 0 a 1 b 2 c 3 d 4

My code needs to do this operation several times, so I thought I better ask first.

Thanks!

PrashantMathur - 26 Jul 2013 - 10:30

sorry the formatting didn't turn out as I planned to .. let me try again  A 0 a 1 b 2 c 3 B 0 d 1 a 2 b 3 

 C 0 a 1 b 2 c 3 d 4 

PrashantMathur - 26 Jul 2013 - 11:10

Is there a function which can merge two SymbolTables?
I was using  A->AddTable(SymbolTable* B) , where I want to merge B in A.
But I believe AddTable function just appends table B with A and starts indexing with the maximum key. ( I don't know if I am correct).
A
eps 0
a 1
b 2
c 3

B
eps 0
d 1
a 2
b 3

C - Resulting Table
eps 0
a 1
b 2
c 3
d 4

My code needs to do this operation several times, so I thought I better ask first.
Thanks!

PaulDixon - 27 Jul 2013 - 05:15

If table A has holes, you can use CompactSymbols after merging the table to give a contiguous table. MergeSymbolTable will merge symbol tables and has optimizations for certain cases. Both are in symbol-table-ops.h

I/O error for large FSTs

SamS - 08 Jul 2013 - 14:49

I'm working on an application where I'd like to store a large FST in binary format, but I'm getting read/write errors. I'm convinced my large FST is well formed and that there might be a problem with big read/write operations. Here's a toy example that produces the same error:

   SymbolTable * syms = new SymbolTable;

StdVectorFst res;
for (int s = 0; s < big_number; s++) {
TropicalWeight final = -log(0.5);
res.SetFinal(s, final);
for (int dest = 0; dest < big_number; dest++) {
}
}

res.SetStart(0);
res.SetInputSymbols(syms);
res.SetOutputSymbols(syms);

return 0;


For me, the big number seems to be too big at around 923. The printed error message is:

ERROR: VectorFst::Read: read failed: badfile.fst


Any insight into what's going on and how to get around this issue would be appreciated. Thanks!

SamS - 08 Jul 2013 - 16:39

And this turned out to be a problem with the local file system. Sorry!

MikySu - 12 Jul 2013 - 12:54

I also had the same error message. i command "fstcompile --arc_type=log --fst_type=const GramFst > GramFst.out.bfsm",but the fst type of GramFst.out.bfsm didn't transform from vector to const. could someone give me some advice?

MikySu - 12 Jul 2013 - 13:16

my file GramFst.out.bfsm and LexFst.out.bfsm both of fst type were vector. i commanded "fstcompose LexFst.out.bfsm GramFst.out.bfsm >a1.bfsm". i checked the information with fstinfo and then the error message came out "ERROR: VectorFst::Read: read failed:a1.bfsm". file GramFst.out.bfsm and LexFst.out.bfsm were fine.

PaulDixon - 13 Jul 2013 - 04:23

What is the format of GramFst? Hoe many states and arcs do the bfsm machine have? Did this work for you fstconvert --fst_type=const < some.vector.fst > new.const.fst

Making an FST with C++: How to define strings as labels?

DennisRitter - 19 Jun 2013 - 10:30

In the shell you have the possibility to set up strings as ilabels and olabels (with isymbols.txt and osymbols.txt) But when I followed the QuickTour and wanted to do it all in the C++-Editor I couldn't find out, how to give the strings to an vectorfst. I can only give them integers as labels, because of the definition of stdarc.

Would be glad if someone could point out that to me.

PS Is it even effective to make it all in the C++-editor (I am using Eclipse)?

PaulDixon - 20 Jun 2013 - 02:11

Use the SymbolTable to assign/get integer ids for the string labels

DennisRitter - 20 Jun 2013 - 10:26

Is that the way you meant it?
SymbolTable* isyms;
isyms = new SymbolTable("isyms.txt");
SymbolTable* osyms;
osyms = new SymbolTable("osyms.txt");

fst.SetInputSymbols(isyms);
fst.SetOutputSymbols(osyms);

// Adds state 0 to the initially empty FST and make it the start state.
fst.SetStart(0); // arg is state ID

// Adds two arcs exiting state 0.
// Arc constructor args: ilabel, olabel, weight, dest state ID.
fst.AddArc(0, StdArc(isyms->Find("a"), osyms->Find("x"), 0.5, 1)); // 1st arg is src state ID

// Adds state 1 and its arc.

// Adds state 2 and set its final weight.
fst.SetFinal(2, 3.5); // 1st arg is state ID, 2nd arg weight

fst.Write("example.fst");

PaulDixon - 20 Jun 2013 - 19:59

Yes, by convention symbol 0 is the for the epsilon label so call oyms->AddSymbol("\<eps\>"); first and make sure the return value is 0.

AddSymbol returns the id if the key already exists. So you can remove all the AddSymbol lines (except for the epsilon) and do this instead

From memory in this case I think you would need to add the Symbol tables after adding all the arcs. Internally I think it will make a copy when you use SetInputSymbols method so might also need to delete the SymbolTables also.

DennisRitter - 21 Jun 2013 - 05:01

Thanks a lot Paul!

PS You now helped me the third time, first time as Edobashira on your old blog :). I was DreeDrunk at that time. Shall I mark my thread with something like a SOLVED-tag?

PaulDixon - 23 Jun 2013 - 16:50

Glad I could help. Not sure if the thread should be marked as solved.

GenericRegister::GetEntry : lookup failed in shared object

MarkusD - 12 May 2013 - 20:01

I'm trying to register my expectation arc, so that it can be used in fstcompile, etc.

But I'm getting the runtime error:

ERROR: GenericRegister::GetEntry : lookup failed in shared object: expectation-arc.so FATAL: No operation found for "CompileFst" on arc type expectation

How can I fix this?

The register call in my code looks like this (my arc is called MDExpectationArc):

#include <fst/const-fst.h> #include <fst/edit-fst.h> #include <fst/vector-fst.h> #include <fst/script/register.h> #include <fst/script/fstscript.h> namespace fst { namespace script { REGISTER_FST(VectorFst, MDExpectationArc); REGISTER_FST(ConstFst, MDExpectationArc); REGISTER_FST(EditFst, MDExpectationArc); REGISTER_FST_CLASSES(MDExpectationArc); REGISTER_FST_OPERATIONS(MDExpectationArc); }}

Here is how I compile: /usr/bin/c++ -fPIC -O2 -shared -o expectation-arc.so expectation-arc-inline.cc.o /c01_data/mdreyer/software/openfst-1.3.3/lib/libfst.so

or this, gives the same error: /usr/bin/c++ -fPIC -O2 -shared -Wl,-soname,expectation-arc.so -o expectation-arc.so expectation-arc-inline.cc.o openfst-1.3.3/lib/libfst.so -ldl -Wl,-rpath,openfst-1.3.3/lib

It used to work for me with an old openfst version, but I'm getting this error now with openfst 1.3.3.

MarkusD - 12 May 2013 - 20:07

For clarity, I'm getting the runtime error when I run this, for example:

fstcompile --arc-type=expectation

where it loads the expectation-arc.so file that I created using the c++ commands above.

PaulDixon - 13 May 2013 - 08:10

Does the underlying weight type of the MDExpectationArc return the string "expectation" from the Type() method? I think the Type() method returns value need to match the name of shared object.

how to use fstreplace tool

HuanliangWang - 09 May 2013 - 21:54

Hi, I want to use fstreplace tool to generate a extended fst by replacing a nonternimal label in root fst by a subfst. But I got error when using the tool. I know it is necessary to preprocess two fsts, such as symbol table. But I don't know the detail. Could anybody tell me the detailed usage procedure?

Thanks,

Huanliang Wang

Regex to FST

RaihanRasool - 07 May 2013 - 12:56

Hi,

Using OpenFst library we can create FST for a given regular expression. But is there any way I could develop a generic RegEx to FST converter that takes 'any' regular expression and build FST ?

PaulDixon - 09 May 2013 - 00:09

Maybe the answer to this question is what you are looking for

MarkusD - 12 May 2013 - 19:55

Also, I think Thrax does something like that: http://www.openfst.org/twiki/bin/view/GRM/Thrax

KennethRBeesley - 2017-01-26 - 22:58

I would also suggest taking a look at the Kleene language: www.kleene-lang.org

Coordinate Descent Solution for Rational Kernels

AbielRoche - 01 May 2013 - 13:59

Is there any available implementation of Coordinate Descent Solution for SVM using Rational Kernels? As it is mentioned in the paper: "Large-Scale Training of SVMs with Automata Kernels". Thank you for your help,

“FATAL: StringWeight::Plus: unequal arguments” when running fstdeterminize.

JiaP - 31 Mar 2013 - 21:51

The transducer in question simply describes a flat list dictionary without any explicit weight. It is very similar to the one given in the “Tokenization” example, except being much larger (about 664,000 states).

When I ran fstdeterminize on this transducer, the command line tool exited with following error message: FATAL: StringWeight::Plus: unequal arguments (non-functional FST?) w1 = 1 w2 = 2

I will try to construct such transducer directly using C++ API. Mean while, I’d appreciate any suggestion.

Jia

PaulDixon - 01 Apr 2013 - 01:59

Are you adding disambiguation symbols like the ones described for the lexicon construction in this paper http://www.cs.nyu.edu/~mohri/pub/hbka.pdf

EricLebigot - 03 Apr 2013 - 22:37

@PaulDixon: I had a look at the paper you link to. As far as I understand, for determinization to be possible, each input string should only produce a single output string, right?

Composition result too large: how to make smaller?

EricLebigot - 25 Mar 2013 - 22:27

Hi,

I am trying to calculate a kernel of the form T∘T^-1 from a transducer T, but the calculation uses up a lot of memory and I have to stop it before it converges (I stopped at 25 GB of swap). What could I try to do in order to calculate the composition result? (My ultimate goal is to calculate a kernel between acceptors A and B through the A∘T∘T^-1∘B composition and getting the (log) total weight between the starting and final states. A difficulty is that A∘T can produce a huge number of paths, and I was hoping that calculating T∘T^-1 could help solve this difficulty.)

I tried to first optimize transducer T, but this failed (a first epsilon removal yields a fatal error about StringWeight::Plus being given unequal arguments, as does determinization).

Here are some characteristics of transducer T, if they matter:

- About 15 nodes and 200k arcs.

- Accommodates input strings of any size through the use of internal loops.

If necessary, I can remove all loops, or some loops (because I know in advance how big the input strings can be); this would multiply the number of nodes and arcs by maybe 100 or 600 (depending on which loops I remove). Could removing loops significantly with the "too much memory used for composition" problem?

I could also limit the number of arcs, but this would make the transducer less expressive, so I would like to avoid this. However, if this is the best way of speeding up the calculation of T∘T^-1 and limiting its memory requirement, I am ready to do that.

What would you suggest, so that T∘T^-1 can really be calculated?

PaulDixon - 26 Mar 2013 - 05:35

In the transducers do you have to traverse many epsilon transitions before reaching a label? This can delay the matching and lead to the generation of many dead end states. These states are normally removed after composition.

I don't think epsilon removal uses the StringWeight are your weights in the log/tropical semiring? Another problem could be negative weighted epsilon cycles.

EricLebigot - 26 Mar 2013 - 10:02

(I'm not fully familiar with FST, so please let me know if I'm not clear.)

Transducer T can produce many epsilons (in about half of the 200k arcs) before reaching a final state (and T^-1 therefore can consume many epsilons). This produces a huge number of possible outputs, and I was hoping that OpenFST could somehow "magically" calculate T∘T^-1 despite this. This epsilon production cannot be removed, because it is one of the main purposes of transducer T to remove parts of the input strings.

The weights are in the log semiring. There are cycles that produce epsilon (in T), with a weight of LogWeight(0.69…) (corresponding to 0.5 in real space).

With these precisions, what do you think is the best way to speed up the composition? reducing the number of arcs (I can do this by reducing the number of recognized "letters")?

PaulDixon - 01 Apr 2013 - 07:10

Do have an example transducer you can share online?

EricLebigot - 01 Apr 2013 - 23:56

@PaulDixon: I put a simplified version at http://www.normalesup.org/~lebigot/special/transducer.pdf. The real version has many more arcs (20k arcs where there are ~40 between two states in the simplified version, 1k arcs where there are ~4 arcs). Note: I will have a look at the disambiguation paper that you mentioned yesterday: maybe determinization will speed up the calculations?

DoganCan - 02 Apr 2013 - 06:09

Hi Eric,

The problem is that transducer T can produce infinitely many consecutive epsilons (when it is not restricted by input A) and similarly transducer T^-1 can consume infinitely many consecutive epsilons (when it is not restricted by output B). Naturally, composition does not terminate. You will have to avoid "all" epsilon loops on the output side of T if you want to compute ToT^-1, i.e. limiting the number of output-epsilon arcs won't help. Unfortunately there is no such magic hidden in openfst

Cheers, Dogan

EricLebigot - 02 Apr 2013 - 09:32

@DoganCan: Thank you for your input: very interesting! I can certainly remove infinite epsilon-generating loops (thanks to the structure of the input strings). However, a strange thing is that T∘T^-1 does terminate, with OpenFST (when I restrict the number of arcs, like in the simplified version—the change is only quantitative, not qualitative). Does this mean that the composition obtained by OpenFST is likely incorrect? I am a little confused.

EricLebigot - 02 Apr 2013 - 09:46

PS: I put the transducer from the PDF above in files transducer.fst, transducer.ssym (state symbols) and transducer.sym (labels) at http://www.normalesup.org/~lebigot/special/, if this can help. OpenFST happily calculates T∘T^-1 after arc-sorting the outputs of T.

DoganCan - 03 Apr 2013 - 04:19

Hi Eric,

It looks like my initial assessment was incorrect. I thought composition would somehow force enumeration of all possible matches between the epsilon cycles on the output side of T and the epsilon cycles on the input of T^-1. Instead, what happens is composition simply creates paths which first generate possibly infinitely many epsilons and then consume possibly infinitely many epsilons. The composition output for the small transducer is correct as far as I can tell.

I think the blow-up in composition is caused by the self-loops on states A_CAT_V_KEPT and B_CAT_V_KEPT. More explicitly, since T maps each input symbol on these arcs to "Wild" and T^-1 maps "Wild" back to these symbols, composition output has a mapping between each input symbol pair. If the big transducer is following the same pattern, there will be around 10K x 10K = 100M self loops in the output of composition for each of these two states.

Cheers, Dogan

EricLebigot - 03 Apr 2013 - 05:31

Hi Dogan,

Thank you for following up! Your remarks are very useful. The structure of the input data is such that the Wild labels can be specialized, so that the number of possible matches is reduced (PCP_123:Wild is now PCP_123:PCP_Wild, etc., so that only similar categories match through Wild). The result is that calculating kernel elements is much faster, thank you!

Maybe will the calculation of T∘T^-1 require less memory, too…

Best, EOL

Double delete bug in fst far reader

DoganCan - 05 Mar 2013 - 04:14

Hi,

I came across a double delete bug in the implementation of FstFarReader class. ReadFst method deletes the current fst before reading a new one, however if there is no more fst to read then the current fst pointer is left dangling. That leads to a segfault when the class destructor is called. You might trigger the error with

farequal fst1 fst2


I fixed the problem by moving the pointer deletion after the check for whether there is more fst to read.

This error is not triggered by other far utilities reading from an fst archive (farprintstrings, farinfo and farextract) since these tools, unlike farequal, do not delete the far reader pointer after they are done with it. It might be a good idea to fix that as well.

Cheers, Dogan

Get the Arcs whose nextstate is equal to a given q

JuanMiguelCejuela - 04 Mar 2013 - 08:31

Is it possible to efficiently get all the arcs which have as nextstate a given state q?

DoganCan - 05 Mar 2013 - 04:50

I don't think there is builtin functionality for efficient retrieval of incoming arcs. Your best option is to implement a new Fst class similar to VectorFst which has this functionality. You will first need to modify the definition of VectorState and add a new vector of arcs for incoming arcs. Then you will need to modify the methods that has to do with arcs such as AddArc, RemoveArcs, DeleteArcs, etc. to handle the incoming arcs as well as outgoing arcs. Finally you will need to define a new arc iterator for visiting incoming arcs.

Cheers, Dogan

AlexZatv - 05 Mar 2013 - 07:10

May be, you can revert the Fst? If you will revert the fst but keep the state id's the same, you can simply get revert(F).out_arcs(q).

Testing an FST for epsilon cycles

KennethRBeesley - 24 Jan 2013 - 00:41

If I have an FST, how can I best test to see if it has input epsilon cycles?

KennethRBeesley - 22 Feb 2013 - 12:16

Here's my own best effort, which seems to be working. Comments welcome. C++ is definitely not my native language.

bool isInputEpsilonCycleFree(StdVectorFst * fstp) {

set<StateId> stateSet ;   // reuse this set of "visited states"
// if the machine has no cycles at all, then it's epsilon-cycle-free
if (fstp->Properties(kAcyclic, true))
return true ;

// loop through the states, looking for input epsilon cycles
for (StateIterator<StdVectorFst> siter(*fstp) ;
!siter.Done() ;
siter.Next()) {
stateSet.clear() ;
if (findInputEpsilonCycle(fstp, siter.Value(), stateSet))
return false ;
}
// no input epsilon cycles found
return true ;
}

bool findInputEpsilonCycle(StdVectorFst *fstp, StateId s, set<StateId> &stateSet) {
// try to find an input epsilon cycle from state s back
// to itself, or any other input epsilon cycle found during the search

// stateSet is the set of states previously visited during the search from s
if (stateSet.find(s) != stateSet.end())
// found a loop
return true ;
// remember state s as "visited"
stateSet.insert(s) ;
for (ArcIterator<StdVectorFst> aiter(*fstp, s) ;
!aiter.Done() ;
aiter.Next()) {
StdArc arc = aiter.Value() ;
// deal only with arcs with epsilon on the input side
if (arc.ilabel != 0)     // 0 (zero) is wired in as epsilon in OpenFst
continue ;
// here the ilabel is 0 (epsilon)
// recursively look for an input epsilon loop from arc.nextstate
if (findInputEpsilonCycle(fstp, arc.nextstate, stateSet))
return true ;
}
return false ;
}


PaulDixon - 23 Feb 2013 - 05:48

Using the OpenFst visitor to find a topological order on the epsilon input subgraph should work.

template<class Arc>
bool HasInputEpsilonCycle(const Fst<Arc>& fst) {
vector<typename Arc::StateId> order;
bool acyclic = false;
TopOrderVisitor<Arc> top_order_visitor(&order, &acyclic);
DfsVisit(fst, &top_order_visitor, InputEpsilonArcFilter<Arc>());
return !acyclic;
}


KennethRBeesley - 21 Mar 2013 - 15:30

Paul, Many thanks. Very elegant and seems to work perfectly.

Example of k-closed semiring that is non-idempotent

KonstantinSelivanov - 10 Jan 2013 - 04:30

Hi,

could you give an example k-close semiring that is not idempotent.

Thanks!

CyrilAllauzen - 18 Jan 2013 - 20:46

Hi, The k -tropical semiring defined by (Mohri, 2002) is (k-1) -closed and not idempotent when k ≥ 2 .