|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||
java.lang.Objectorg.opennms.netmgt.rrd.QueuingRrdStrategy
public class QueuingRrdStrategy
Provides queuing implementation of RrdStrategy. In order to provide a more scalable collector. We created a queuing RrdStrategy that enabled the system to amortize the high cost of opening an round robin database across multiple updates. This RrdStrategy implementation enqueues the create and update operations on a per file basis and maintains a set of threads that process enqueued work file by file. If the I/O system can keep up with the collection threads while performing only a single update per file then eventually all the data is processed and the threads sleep until there is more work to do. If the I/O system is initially slower than than the collection threads then work will enqueue here and the write threads will get behind. As this happens each file will eventually have more than a single update enqueued and therefore the number of updates pushed thru the system will increase because more then one will be output per 'open' Eventually, the I/O system and the collection system will balance out. When this happens all data will be collected but will not be output to the rrd files until the next time the file is processed by the write threads. As another performance improving strategy. The queue distinguishes between files with significant vs insignificant updates. Files with only insignificant updates are put at the lowest priority and are only written when the highest priority updates have been written This implementation delegates all the actual writing to another RrdStrategy implementation. System properties effecting the operation: org.opennms.rrd.queuing.writethreads: (default 2) The number of rrd write threads that process the queue org.opennms.rrd.queuing.queueCreates: (default false) indicates whether rrd file creates should be queued or processed synchronously org.opennms.rrd.queuing.maxInsigUpdateSeconds: (default 0) the number of seconds over which all files with significant updates only should be promoted onto the significant less. This is to ensure they don't stay unprocessed forever. Zero means not promotion. org.opennms.rrd.queuing.modulus: (default 10000) the number of updates the get enqueued between statistics output org.opennms.rrd.queuing.category: (default "OpenNMS.Queued") the log category to place the statistics output in TODO: Promote files when ZeroUpdate operations can't be merged. This may be a collection miss which we want to push thru. It should also help with memory. TODO: Set an upper bound on enqueued operations TODO: Provide an event that will write data for a particular file... Say right before we try to graph it.
| Nested Class Summary | |
|---|---|
class |
QueuingRrdStrategy.CreateOperation
This class represents an operation to create an rrd file |
class |
QueuingRrdStrategy.UpdateOperation
Represents an update to a rrd file. |
class |
QueuingRrdStrategy.ZeroUpdateOperation
Represents an update whose value is 0. |
| Constructor Summary | |
|---|---|
QueuingRrdStrategy(RrdStrategy<Object,Object> delegate)
Constructor for QueuingRrdStrategy. |
|
| Method Summary | |
|---|---|
void |
addOperation(org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation op)
Add an operation to the queue. |
void |
closeFile(String rrd)
closeFile |
org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation |
createDefinition(String creator,
String directory,
String rrdName,
int step,
List<RrdDataSource> dataSources,
List<String> rraList)
Create a round robin database definition from the supplied parameters. |
org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation |
createDefinition(String creator,
String directory,
String dsName,
int step,
String dsType,
int dsHeartbeat,
String dsMin,
String dsMax,
List<String> rraList)
createDefinition |
void |
createFile(org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation op)
createFile |
InputStream |
createGraph(String command,
File workDir)
Creates an InputStream representing the bytes of a graph created from round robin data. |
RrdGraphDetails |
createGraphReturnDetails(String command,
File workDir)
Creates an RrdGraphDetails object representing the graph created from round robin data. |
void |
ensureThreadsStarted()
Ensure that we have threads started to process the queue. |
Double |
fetchLastValue(String rrdFile,
String ds,
int interval)
Fetches the last value from the round robin database with the given name. |
Double |
fetchLastValue(String rrdFile,
String ds,
String consolidationFunction,
int interval)
Fetches the last value from the round robin database with the given name. |
Double |
fetchLastValueInRange(String rrdFile,
String ds,
int interval,
int range)
Fetches the last value from the round robin database with the given name within a time range. |
String |
getCategory()
getCategory |
Properties |
getConfigurationProperties()
getConfigurationProperties |
long |
getCreatesCompleted()
getCreatesCompleted |
String |
getDefaultFileExtension()
getDefaultFileExtension |
RrdStrategy<Object,Object> |
getDelegate()
getDelegate |
long |
getDequeuedItems()
getDequeuedItems |
long |
getDequeuedOperations()
getDequeuedOperations |
long |
getEnqueuedOperations()
getEnqueuedOperations |
long |
getErrors()
getErrors |
int |
getGraphLeftOffset()
getGraphLeftOffset |
int |
getGraphRightOffset()
getGraphRightOffset |
int |
getGraphTopOffsetWithText()
getGraphTopOffsetWithText |
long |
getInSigHighWaterMark()
getInSigHighWaterMark |
String |
getLapTime()
getLapTime |
long |
getMaxInsigUpdateSeconds()
getMaxInsigUpdateSeconds |
long |
getModulus()
getModulus |
LinkedList<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation> |
getNext()
Get the operations for the next file that should be worked on. |
long |
getPromotionCount()
getPromotionCount |
long |
getQueueHighWaterMark()
getQueueHighWaterMark |
long |
getSigHighWaterMark()
getSigHighWaterMark |
long |
getSignificantOpsCompleted()
getSignificantOpsCompleted |
long |
getSignificantOpsDequeued()
getSignificantOpsDequeued |
long |
getSignificantOpsEnqueued()
getSignificantOpsEnqueued |
long |
getStartTime()
getStartTime |
String |
getStats()
Print queue statistics. |
long |
getTotalOperationsPending()
getTotalOperationsPending |
long |
getUpdatesCompleted()
getUpdatesCompleted |
long |
getWriteThreadExitDelay()
getWriteThreadExitDelay |
int |
getWriteThreads()
getWriteThreads |
long |
getWriteThreadSleepTime()
getWriteThreadSleepTime |
void |
logStats()
logStats |
org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation |
makeCreateOperation(String fileName,
Object rrdDef)
makeCreateOperation |
org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation |
makeUpdateOperation(String fileName,
String owner,
String update)
makeUpdateOperation |
String |
openFile(String fileName)
Opens the round robin database with the supplied name. |
boolean |
prioritizeSignificantUpdates()
prioritizeSignificantUpdates |
void |
promoteEnqueuedFiles(Collection<String> rrdFiles)
In the event that this is a queuing implementation of the RrdStrategy. |
boolean |
queueCreates()
queueCreates |
void |
run()
run |
void |
setCategory(String category)
setCategory |
void |
setConfigurationProperties(Properties configurationParameters)
setConfigurationProperties |
void |
setCreatesCompleted(long createsCompleted)
setCreatesCompleted |
void |
setDequeuedItems(long dequeuedItems)
setDequeuedItems |
void |
setDequeuedOperations(long dequeuedOperations)
setDequeuedOperations |
void |
setEnqueuedOperations(long enqueuedOperations)
setEnqueuedOperations |
void |
setErrors(long errors)
setErrors |
void |
setInSigHighWaterMark(long inSigHighWaterMark)
setInSigHighWaterMark |
void |
setMaxInsigUpdateSeconds(long maxInsigUpdateSeconds)
setMaxInsigUpdateSeconds |
void |
setModulus(long modulus)
setModulus |
void |
setPrioritizeSignificantUpdates(boolean prioritizeSignificantUpdates)
setPrioritizeSignificantUpdates |
void |
setPromotionCount(long promotionCount)
setPromotionCount |
void |
setQueueCreates(boolean queueCreates)
setQueueCreates |
void |
setQueueHighWaterMark(long queueHighWaterMark)
setQueueHighWaterMark |
void |
setSigHighWaterMark(long sigHighWaterMark)
setSigHighWaterMark |
void |
setSignificantOpsCompleted(long significantOpsCompleted)
setSignificantOpsCompleted |
void |
setSignificantOpsDequeued(long significantOpsDequeued)
setSignificantOpsDequeued |
void |
setSignificantOpsEnqueued(long significantOpsEnqueued)
setSignificantOpsEnqueued |
void |
setStartTime(long updateStart)
setStartTime |
void |
setTotalOperationsPending(long totalOperationsPending)
setTotalOperationsPending |
void |
setUpdatesCompleted(long updatesCompleted)
setUpdatesCompleted |
void |
setWriteThreadExitDelay(long writeThreadExitDelay)
setWriteThreadExitDelay |
void |
setWriteThreads(int writeThreads)
setWriteThreads |
void |
setWriteThreadSleepTime(long writeThreadSleepTime)
setWriteThreadSleepTime |
void |
updateFile(String rrdFile,
String owner,
String data)
Updates the supplied round robin database with the given timestamp:value point |
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Constructor Detail |
|---|
public QueuingRrdStrategy(RrdStrategy<Object,Object> delegate)
Constructor for QueuingRrdStrategy.
delegate - a RrdStrategy object.| Method Detail |
|---|
public Properties getConfigurationProperties()
getConfigurationProperties
Properties object.public void setConfigurationProperties(Properties configurationParameters)
setConfigurationProperties
setConfigurationProperties in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>configurationParameters - a Properties object.public int getWriteThreads()
getWriteThreads
public void setWriteThreads(int writeThreads)
setWriteThreads
writeThreads - a int.public boolean queueCreates()
queueCreates
public void setQueueCreates(boolean queueCreates)
setQueueCreates
queueCreates - a boolean.public boolean prioritizeSignificantUpdates()
prioritizeSignificantUpdates
public void setPrioritizeSignificantUpdates(boolean prioritizeSignificantUpdates)
setPrioritizeSignificantUpdates
prioritizeSignificantUpdates - a boolean.public long getInSigHighWaterMark()
getInSigHighWaterMark
public void setInSigHighWaterMark(long inSigHighWaterMark)
setInSigHighWaterMark
inSigHighWaterMark - a long.public long getSigHighWaterMark()
getSigHighWaterMark
public void setSigHighWaterMark(long sigHighWaterMark)
setSigHighWaterMark
sigHighWaterMark - a long.public long getQueueHighWaterMark()
getQueueHighWaterMark
public void setQueueHighWaterMark(long queueHighWaterMark)
setQueueHighWaterMark
queueHighWaterMark - a long.public long getModulus()
getModulus
public void setModulus(long modulus)
setModulus
modulus - a long.public String getCategory()
getCategory
String object.public void setCategory(String category)
setCategory
category - a String object.public long getMaxInsigUpdateSeconds()
getMaxInsigUpdateSeconds
public void setMaxInsigUpdateSeconds(long maxInsigUpdateSeconds)
setMaxInsigUpdateSeconds
maxInsigUpdateSeconds - a long.public long getWriteThreadSleepTime()
getWriteThreadSleepTime
public void setWriteThreadSleepTime(long writeThreadSleepTime)
setWriteThreadSleepTime
writeThreadSleepTime - a long.public long getWriteThreadExitDelay()
getWriteThreadExitDelay
public void setWriteThreadExitDelay(long writeThreadExitDelay)
setWriteThreadExitDelay
writeThreadExitDelay - a long.
public org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation makeCreateOperation(String fileName,
Object rrdDef)
makeCreateOperation
fileName - a String object.rrdDef - a Object object.
QueuingRrdStrategy.Operation object.
public org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation makeUpdateOperation(String fileName,
String owner,
String update)
makeUpdateOperation
fileName - a String object.owner - a String object.update - a String object.
QueuingRrdStrategy.Operation object.public void addOperation(org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation op)
op - a QueuingRrdStrategy.Operation object.public void ensureThreadsStarted()
public LinkedList<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation> getNext()
public void promoteEnqueuedFiles(Collection<String> rrdFiles)
promoteEnqueuedFiles in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>rrdFiles - a Collection object.public RrdStrategy<Object,Object> getDelegate()
getDelegate
RrdStrategy object.
public void closeFile(String rrd)
throws Exception
closeFile
closeFile in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>rrd - a String object.
Exception - if any.
public org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation createDefinition(String creator,
String directory,
String dsName,
int step,
String dsType,
int dsHeartbeat,
String dsMin,
String dsMax,
List<String> rraList)
throws Exception
createDefinition
creator - a String object.directory - a String object.dsName - a String object.step - a int.dsType - a String object.dsHeartbeat - a int.dsMin - a String object.dsMax - a String object.rraList - a List object.
QueuingRrdStrategy.Operation object.
Exception - if any.
public org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation createDefinition(String creator,
String directory,
String rrdName,
int step,
List<RrdDataSource> dataSources,
List<String> rraList)
throws Exception
createDefinition in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>creator - -
A string representing who is creating this file for use in log
msgsdirectory - -
The directory to create the file inrrdName - -
The name to use for the round robin databasestep - -
the step for the databasedataSources - -
the data sources to use for round robin databaserraList - -
a List of the round robin archives to create in the database
Exception - If an error occurs while creating the definition
public void createFile(org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation op)
throws Exception
createFile
createFile in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>op - a QueuingRrdStrategy.Operation object.
Exception - if any.
public String openFile(String fileName)
throws Exception
openFile in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>fileName - the name of the associated rrd file
Exception - if an error occurs opening the file
public void updateFile(String rrdFile,
String owner,
String data)
throws Exception
updateFile in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>rrdFile - an rrd object created using openFileowner - the owner of the rrddata - a string of the form Exception - if an error occurs updating the file
public Double fetchLastValue(String rrdFile,
String ds,
int interval)
throws NumberFormatException,
RrdException
fetchLastValue in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>rrdFile - a name the represents a round robin databaseds - a name the represents a data source to be usedinterval - a step interval of the round robin database
NumberFormatException - if any.
RrdException - if any.
public Double fetchLastValue(String rrdFile,
String ds,
String consolidationFunction,
int interval)
throws NumberFormatException,
RrdException
fetchLastValue in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>rrdFile - a name the represents a round robin databaseds - a name the represents a data source to be usedconsolidationFunction - a String object.interval - a step interval of the round robin database
NumberFormatException - if any.
RrdException - if any.
public Double fetchLastValueInRange(String rrdFile,
String ds,
int interval,
int range)
throws NumberFormatException,
RrdException
fetchLastValueInRange in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>rrdFile - a name the represents a round robin databaseds - a name the represents a data source to be usedinterval - a step interval of the round robin databaserange - an acceptable range for which the last value will be returned
NumberFormatException - if any.
RrdException - if any.
public InputStream createGraph(String command,
File workDir)
throws IOException,
RrdException
createGraph in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>command - the command needed to create the graphworkDir - the directory that all referenced files are relative to
IOException - if an IOError occurs
RrdException - if an RRD error occurspublic void run()
run
run in interface Runnablepublic String getStats()
getStats in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>String object.public void logStats()
logStats
public String getLapTime()
getLapTime
String object.public int getGraphLeftOffset()
getGraphLeftOffset
getGraphLeftOffset in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>public int getGraphRightOffset()
getGraphRightOffset
getGraphRightOffset in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>public int getGraphTopOffsetWithText()
getGraphTopOffsetWithText
getGraphTopOffsetWithText in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>public String getDefaultFileExtension()
getDefaultFileExtension
getDefaultFileExtension in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>String object.
public RrdGraphDetails createGraphReturnDetails(String command,
File workDir)
throws IOException,
RrdException
createGraphReturnDetails in interface RrdStrategy<org.opennms.netmgt.rrd.QueuingRrdStrategy.Operation,String>command - the command needed to create the graphworkDir - the directory that all referenced files are relative to
IOException - if an IOError occurs
RrdException - if an RRD error occurspublic long getTotalOperationsPending()
getTotalOperationsPending
public void setTotalOperationsPending(long totalOperationsPending)
setTotalOperationsPending
totalOperationsPending - a long.public long getCreatesCompleted()
getCreatesCompleted
public void setCreatesCompleted(long createsCompleted)
setCreatesCompleted
createsCompleted - a long.public long getUpdatesCompleted()
getUpdatesCompleted
public void setUpdatesCompleted(long updatesCompleted)
setUpdatesCompleted
updatesCompleted - a long.public long getErrors()
getErrors
public void setErrors(long errors)
setErrors
errors - a long.public long getPromotionCount()
getPromotionCount
public void setPromotionCount(long promotionCount)
setPromotionCount
promotionCount - a long.public long getSignificantOpsEnqueued()
getSignificantOpsEnqueued
public void setSignificantOpsEnqueued(long significantOpsEnqueued)
setSignificantOpsEnqueued
significantOpsEnqueued - a long.public long getSignificantOpsDequeued()
getSignificantOpsDequeued
public void setSignificantOpsDequeued(long significantOpsDequeued)
setSignificantOpsDequeued
significantOpsDequeued - a long.public long getEnqueuedOperations()
getEnqueuedOperations
public void setEnqueuedOperations(long enqueuedOperations)
setEnqueuedOperations
enqueuedOperations - a long.public long getDequeuedOperations()
getDequeuedOperations
public void setDequeuedOperations(long dequeuedOperations)
setDequeuedOperations
dequeuedOperations - a long.public long getDequeuedItems()
getDequeuedItems
public void setDequeuedItems(long dequeuedItems)
setDequeuedItems
dequeuedItems - a long.public long getSignificantOpsCompleted()
getSignificantOpsCompleted
public void setSignificantOpsCompleted(long significantOpsCompleted)
setSignificantOpsCompleted
significantOpsCompleted - a long.public long getStartTime()
getStartTime
public void setStartTime(long updateStart)
setStartTime
updateStart - a long.
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||