danbikel.parser
Class InterpolatedKnesserNeyModel

java.lang.Object
  danbikel.parser.Model
      danbikel.parser.InterpolatedKnesserNeyModel

All Implemented Interfaces:

Serializable

public class InterpolatedKnesserNeyModel
extends Model

Implements a model that uses interpolated Knesser-Ney smoothing.

See Also:

Serialized Form

Field Summary

protected double

optimalDiscountEstimate

Fields inherited from class danbikel.parser.Model

backOffMap, cache, cacheAccesses, cacheHits, canonicalEvents, counts, createHistBackOffMap, deleteCountsWhenPrecomputingProbs, dontAddNewParams, doPruning, globalDoPruning, histBackOffMap, historiesToPrune, lambdaFudge, lambdaFudgeTerm, lambdaPenalty, logOneMinusLambdaPenalty, numLevels, precomputedLambdas, precomputedNPBProbCalls, precomputedNPBProbHits, precomputedProbCalls, precomputedProbHits, precomputedProbs, precomputeProbs, printPrunedEvents, printUnprunedEvents, pruningThreshold, saveBackOffMap, saveHistBackOffMap, saveSmoothingParams, shortStructureClassName, smoothingParams, smoothingParamsFile, structure, structureClassName, topLevelCache, transitionsToPrune, useCache, useSmoothingParams, verbose, warnSmoothingHasHistoryNotInTraining

Constructor Summary

InterpolatedKnesserNeyModel(ProbabilityStructure structure)
Constructs a Model instance that uses interpolated Knesser-Ney smoothing instead of the default smoothing method when estimating probabilities.

Method Summary

void	deriveCounts(CountsTable trainerCounts, Filter filter, double threshold, FlexibleMap canonical, boolean deriveOtherModelCounts) Derives all counts from the specified counts table, using the probability structure specified in the constructor.
protected double	estimateProb(ProbabilityStructure probStructure, TrainerEvent event) Returns the smoothed probability estimate of a transition contained in the specified TrainerEvent object.
protected void	precomputeProbs(MapToPrimitive.Entry transEntry, double[] lambdas, double[] estimates, Transition[] transitions, Event[] histories, int lastLevel) Precomputes the probabilities and smoothing values for the Transition object contained as a key within the specified map entry, where the value is the count of the transition.
protected void	precomputeProbs(TrainerEvent event, Transition[] transitions, Event[] histories) Deprecated. This method is called by Model.precomputeProbs(CountsTable,Filter), which is also deprecated.
protected void	storePrecomputedProbs(double[] lambdas, double[] estimates, Transition[] transitions, Event[] histories, int lastLevel) Stores the specified smoothing values (lambdas) and smoothed probability estimates in the Model.precomputedProbs and Model.smoothingParams map arrays.

Methods inherited from class danbikel.parser.Model

beQuiet, beVerbose, canonicalize, canonicalize, canonicalizeEvent, cleanup, computeHistoriesAndTransitionsToPrune, deriveCounts, deriveDiversityCounts, deriveHistories, estimateLogProb, estimateLogProbUsingPrecomputed, estimateLogProbUsingPrecomputed, estimateProb, estimateProbOld, getCacheStats, getCanonical, getModel, getProbStructure, getTransitions, initializeSmoothingParams, numModels, precomputeProbs, precomputeProbs, pruneHistoriesAndTransitions, pruneHistoriesAndTransitionsOld, readSmoothingParams, readSmoothingParams, savePrecomputeData, setCanonicalEvents, share, writeSmoothingParams, writeSmoothingParams

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

optimalDiscountEstimate

protected double optimalDiscountEstimate

Constructor Detail

InterpolatedKnesserNeyModel

public InterpolatedKnesserNeyModel(ProbabilityStructure structure)

Constructs a Model instance that uses interpolated Knesser-Ney smoothing instead of the default smoothing method when estimating probabilities.

Parameters:

structure - the probability structure for which this model will estimate probabilities

Method Detail

deriveCounts

public void deriveCounts(CountsTable trainerCounts,
                         Filter filter,
                         double threshold,
                         FlexibleMap canonical,
                         boolean deriveOtherModelCounts)

Description copied from class: Model

Derives all counts from the specified counts table, using the probability structure specified in the constructor.

Overrides:

deriveCounts in class Model

Parameters:

trainerCounts - a map from TrainerEvent objects to their counts (as doubles) from which to derive counts

filter - used to filter out TrainerEvent objects whose derived counts should not be derived for this model

threshold - a (currently unused) count cut-off threshold

canonical - a reflexive map used to canonicalize objects created when deriving counts

deriveOtherModelCounts - an unused parameter, as this class does not contain other, internal Model instances

estimateProb

protected double estimateProb(ProbabilityStructure probStructure,
                              TrainerEvent event)

Returns the smoothed probability estimate of a transition contained in the specified TrainerEvent object. The smoothing method employed will be the interpolated version of Knesser-Ney.

Overrides:

estimateProb in class Model

Parameters:

probStructure - a ProbabilityStructure object that is either Model.structure or a copy of it, used for temporary storage during the computation performed by this method

event - the TrainerEvent containing a transition from a history to a future whose smoothed probability is to be computed

Returns:

the smoothed probability estimate of a transition contained in the specified TrainerEvent object

precomputeProbs

protected void precomputeProbs(MapToPrimitive.Entry transEntry,
                               double[] lambdas,
                               double[] estimates,
                               Transition[] transitions,
                               Event[] histories,
                               int lastLevel)

Description copied from class: Model

Precomputes the probabilities and smoothing values for the Transition object contained as a key within the specified map entry, where the value is the count of the transition.

Overrides:

precomputeProbs in class Model

Parameters:

transEntry - a map entry mapping a Transition object to its count (a double)

lambdas - an array in which to store the smoothing value for each of the back-off levels

estimates - an array in which to store the maximum-likelihood estimate at each of the back-off levels

transitions - an array in which to store the Transition instance for each of the back-off levels

histories - an array in which to store the history, an Event instance, for each of the back-off levels

lastLevel - the last back-off level (the value equal to Model.numLevels - 1)

See Also:

Model.precomputeProbs()

storePrecomputedProbs

protected void storePrecomputedProbs(double[] lambdas,
                                     double[] estimates,
                                     Transition[] transitions,
                                     Event[] histories,
                                     int lastLevel)

Description copied from class: Model

Stores the specified smoothing values (lambdas) and smoothed probability estimates in the Model.precomputedProbs and Model.smoothingParams map arrays.

Overrides:

storePrecomputedProbs in class Model

Parameters:

lambdas - an array containing the smoothing value for each of the back-off levels

estimates - an array containing the maximum-likelihood estimate at each of the back-off levels

transitions - an array containing the Transition instance for each of the back-off levels

histories - an array in which to store the history, an Event instance, for each of the back-off levels

lastLevel - the last back-off level (the value equal to Model.numLevels - 1)

See Also:

Model.precomputeProbs()

precomputeProbs

protected void precomputeProbs(TrainerEvent event,
                               Transition[] transitions,
                               Event[] histories)

Deprecated. This method is called by Model.precomputeProbs(CountsTable,Filter), which is also deprecated.

Precomputes probabilities for the specified event, using the specified arrays as temporary storage during this method invocation.

Overrides:

precomputeProbs in class Model

Parameters:

event - the TrainerEvent object from which probabilities are to be precomputed

transitions - temporary storage to be used during an invocation of this method

histories - temporary storage to be used during an invocation of this method