Hadoop plugin for HQ

 

Hadoop简介

Hadoop 是一个能够对大量数据进行分布式处理的软件框架。但是 Hadoop 是以一种可靠、高效、可伸缩的方式进行处理的。Hadoop 是可靠的,因为它假设计算元素和存储会失败,因此它维护多个工作数据副本,确保能够针对失败的节点重新分布处理。Hadoop 是高效的,因为它以并行的方式工作,通过并行处理加快处理速度。Hadoop 还是可伸缩的,能够处理 PB 级数据。此外,Hadoop 依赖于社区服务器,因此它的成本比较低,任何人都可以使用。

Hadoop是项目的总称,起源于作者儿子的一只玩具大象的名字。主要是由HDFS、MapReduce和Hbase组成。

HDFS是Google File System(GFS)的开源实现。

MapReduce是Google MapReduce的开源实现。

HBase是Google BigTable的开源实现 

开源的Hyperic HQ软件可以对Hadoop DataNode server,Name Node server,Job Tracker等 实现监控。

Hadoop数据节点服务器

日志追踪支持: Yes

配置追踪支持: Yes

定制属性支持:

Name Description
VmVersion Vm Version
VmVendor Vm Vendor
VmName Vm Name

支持的控制操作: None

支持的度量参数:

 

Name Alias Units Category Default On Default Interval
Process Virtual Memory Size ProcessVirtualMemorySize B UTILIZATION false 5 min
Process Resident Memory Size ProcessResidentMemorySize B UTILIZATION true 5 min
Process Page Faults ProcessPageFaults none UTILIZATION false 10 min
Process Page Faults per Minute ProcessPageFaults1m none UTILIZATION false 10 min
Process Cpu System Time ProcessCpuSystemTime ms UTILIZATION false 10 min
Process Cpu System Time per Minute ProcessCpuSystemTime1m ms UTILIZATION false 10 min
Process Cpu User Time ProcessCpuUserTime ms UTILIZATION false 10 min
Process Cpu User Time per Minute ProcessCpuUserTime1m ms UTILIZATION false 10 min
Process Cpu Total Time ProcessCpuTotalTime ms UTILIZATION false 10 min
Process Cpu Total Time per Minute ProcessCpuTotalTime1m ms UTILIZATION false 10 min
Process Cpu Usage ProcessCpuUsage percentage UTILIZATION true 5 min
Process Start Time ProcessStartTime epoch-millis AVAILABILITY false 5 min
Process Open File Descriptors ProcessOpenFileDescriptors none UTILIZATION false 5 min
Process Threads ProcessThreads none UTILIZATION false 5 min
Availability Availability percentage AVAILABILITY true 5 min
Heap Memory Free HeapMemoryFree B UTILIZATION true 5 min
Thread Count ThreadCount none UTILIZATION true 5 min
Total HotSpot Compilation Time TotalHotSpotCompilationTime ms PERFORMANCE true 5 min

 

配置帮助:

一般日志和配置追踪属性

启用配置追踪 - 检查启用配置追踪

配置文件 - 逗号分隔的配置文件跟踪列表。相对文件是指相对于$ {安装路径}

启用日志追踪 - 检查启用日志追踪

跟踪事件日志级别 - 仅跟踪时间级别等于或大于这个级别.顺序是: [Error, Warn, Info, Debug]

映射:

FATAL,ERROR -> Error

WARN -> Warn

INFO -> Info

DEBUG -> Debug

登录模式匹配 - 包括匹配给定的正则表达式的消息。给定的模式可以是日志消息或正则表达式的子串。请参见:java.util.regex.Pattern。

登录模式排除 - 排除匹配给定的正则表达式的消息。

日志文件 - 逗号分隔的日志文件跟踪列表。相对文件是指相对于$ {安装路径}。

Hadoop数据节点FS数据集服务

日志跟踪技术支持:No

配置跟踪支持:No

支持自定义属性:No

支持的控制动作:None

支持的指标:

Name Alias Units Category Default On Default Interval
Availability Availability percentage AVAILABILITY true 10 min
Remaining Remaining B UTILIZATION true 5 min
Capacity Capacity B UTILIZATION true 5 min
DfsUsed DfsUsed B UTILIZATION true 5 min

Hadoop数据节点RPC服务

日志追踪支持: No

配置追踪支持: No

定制属性支持: None

支持的控制操作: None

支持的参数:

Name Alias Units Category Default On Default Interval
Availability Availability percentage AVAILABILITY true 10 min
Num Open Connections NumOpenConnections none UTILIZATION true 5 min
Call Queue Len CallQueueLen none UTILIZATION false 5 min
Rpc Ops Avg Processing Time RpcOpsAvgProcessingTime ms UTILIZATION false 5 min
Rpc Ops Avg Processing Time Max RpcOpsAvgProcessingTimeMax ms UTILIZATION false 5 min
Rpc Ops Avg Processing Time Min RpcOpsAvgProcessingTimeMin ms UTILIZATION false 5 min
Rpc Ops Avg Queue Time RpcOpsAvgQueueTime ms UTILIZATION true 5 min
Rpc Ops Avg Queue Time Max RpcOpsAvgQueueTimeMax ms UTILIZATION false 5 min
Rpc Ops Avg Queue Time Min RpcOpsAvgQueueTimeMin ms UTILIZATION false 5 min
Rpc Ops Number RpcOpsNumber none UTILIZATION false 5 min

Hadoop数据节点统计服务

日志追踪支持: No

配置追踪支持: No

定制属性支持: None

支持的控制操作: None

支持的参数:

Name Alias Units Category Default On Default Interval
Availability Availability percentage AVAILABILITY true 10 min
Bytes Read BytesRead B UTILIZATION false 10 min
Bytes Read per Minute BytesRead1m B UTILIZATION true 10 min
Bytes Written BytesWritten B UTILIZATION false 10 min
Bytes Written per Minute BytesWritten1m B UTILIZATION true 10 min
Blocks Read BlocksRead none UTILIZATION false 10 min
Blocks Read per Minute BlocksRead1m none UTILIZATION false 10 min
Blocks Removed BlocksRemoved none UTILIZATION false 10 min
Blocks Removed per Minute BlocksRemoved1m none UTILIZATION false 10 min
Blocks Replicated BlocksReplicated none UTILIZATION false 10 min
Blocks Replicated per Minute BlocksReplicated1m none UTILIZATION false 10 min
Blocks Written BlocksWritten none UTILIZATION false 10 min
Blocks Written per Minute BlocksWritten1m none UTILIZATION false 10 min
Block Verification Failures BlockVerificationFailures none UTILIZATION false 10 min
Block Verification Failures per Minute BlockVerificationFailures1m none UTILIZATION false 10 min
Blocks Verified BlocksVerified none UTILIZATION false 10 min
Blocks Verified per Minute BlocksVerified1m none UTILIZATION false 10 min
Reads From Local Client ReadsFromLocalClient none UTILIZATION false 10 min
Reads From Local Client per Minute ReadsFromLocalClient1m none UTILIZATION false 10 min
Reads From Remote Client ReadsFromRemoteClient none UTILIZATION false 10 min
Reads From Remote Client per Minute ReadsFromRemoteClient1m none UTILIZATION false 10 min
Writes From Local Client WritesFromLocalClient none UTILIZATION false 10 min
Writes From Local Client per Minute WritesFromLocalClient1m none UTILIZATION false 10 min
Writes From Remote Client WritesFromRemoteClient none UTILIZATION false 10 min
Writes From Remote Client per Minute WritesFromRemoteClient1m none UTILIZATION false 10 min
Read Block Op Average Time ReadBlockOpAverageTime none UTILIZATION false 5 min
Read Block Op Max Time ReadBlockOpMaxTime none UTILIZATION false 5 min
Read Block Op Min Time ReadBlockOpMinTime none UTILIZATION false 5 min
Read Block Op Num ReadBlockOpNum none UTILIZATION false 5 min
Read Metadata Op Average Time ReadMetadataOpAverageTime none UTILIZATION false 5 min
Read Metadata Op Max Time ReadMetadataOpMaxTime none UTILIZATION false 5 min
Read Metadata Op Min Time ReadMetadataOpMinTime none UTILIZATION false 5 min
Read Metadata Op Num ReadMetadataOpNum none UTILIZATION false 5 min
Block Checksum Op Average Time BlockChecksumOpAverageTime none UTILIZATION false 5 min
Block Checksum Op Max Time BlockChecksumOpMaxTime none UTILIZATION false 5 min
Block Checksum Op Min Time BlockChecksumOpMinTime none UTILIZATION false 5 min
Block Checksum Op Num BlockChecksumOpNum none UTILIZATION false 5 min
Replace Block Op Average Time ReplaceBlockOpAverageTime none UTILIZATION false 5 min
Replace Block Op Max Time ReplaceBlockOpMaxTime none UTILIZATION false 5 min
Replace Block Op Min Time ReplaceBlockOpMinTime none UTILIZATION false 5 min
Replace Block Op Num ReplaceBlockOpNum none UTILIZATION false 5 min
Write Block Op Average Time WriteBlockOpAverageTime none UTILIZATION false 5 min
Write Block Op Max Time WriteBlockOpMaxTime none UTILIZATION false 5 min
Write Block Op Min Time WriteBlockOpMinTime none UTILIZATION false 5 min
Write Block Op Num WriteBlockOpNum none UTILIZATION false 5 min
Copy Block Op Average Time CopyBlockOpAverageTime none UTILIZATION false 5 min
Copy Block Op Max Time CopyBlockOpMaxTime none UTILIZATION false 5 min
Copy Block Op Min Time CopyBlockOpMinTime none UTILIZATION false 5 min
Copy Block Op Num CopyBlockOpNum none UTILIZATION false 5 min
Block Reports Average Time BlockReportsAverageTime none UTILIZATION false 5 min
Block Reports Max Time BlockReportsMaxTime none UTILIZATION false 5 min
Block Reports Min Time BlockReportsMinTime none UTILIZATION false 5 min
Block Reports Num BlockReportsNum none UTILIZATION false 5 min
Heartbeats Average Time HeartbeatsAverageTime none UTILIZATION false 5 min
Heartbeats Max Time HeartbeatsMaxTime none UTILIZATION false 5 min
Heartbeats Min Time HeartbeatsMinTime none UTILIZATION false 5 min
Heartbeats Num HeartbeatsNum none UTILIZATION false 5 min

Hadoop Job Tracker服务器

日志追踪支持: Yes

配置追踪支持: Yes

定制属性支持:

Name Description
VmVersion Vm Version
VmVendor Vm Vendor
VmName Vm Name

支持的控制操作: None

支持的参数:

Name Alias Units Category Default On Default Interval
Process Virtual Memory Size ProcessVirtualMemorySize B UTILIZATION false 5 min
Process Resident Memory Size ProcessResidentMemorySize B UTILIZATION true 5 min
Process Page Faults ProcessPageFaults none UTILIZATION false 10 min
Process Page Faults per Minute ProcessPageFaults1m none UTILIZATION false 10 min
Process Cpu System Time ProcessCpuSystemTime ms UTILIZATION false 10 min
Process Cpu System Time per Minute ProcessCpuSystemTime1m ms UTILIZATION false 10 min
Process Cpu User Time ProcessCpuUserTime ms UTILIZATION false 10 min
Process Cpu User Time per Minute ProcessCpuUserTime1m ms UTILIZATION false 10 min
Process Cpu Total Time ProcessCpuTotalTime ms UTILIZATION false 10 min
Process Cpu Total Time per Minute ProcessCpuTotalTime1m ms UTILIZATION false 10 min
Process Cpu Usage ProcessCpuUsage percentage UTILIZATION true 5 min
Process Start Time ProcessStartTime epoch-millis AVAILABILITY false 5 min
Process Open File Descriptors ProcessOpenFileDescriptors none UTILIZATION false 5 min
Process Threads ProcessThreads none UTILIZATION false 5 min
Availability Availability percentage AVAILABILITY true 5 min
Heap Memory Free HeapMemoryFree B UTILIZATION true 5 min
Thread Count ThreadCount none UTILIZATION true 5 min
Total HotSpot Compilation Time TotalHotSpotCompilationTime ms PERFORMANCE true 5 min

一般日志和配置追踪属性

启用配置追踪 - 检查启用配置追踪

配置文件 - 逗号分隔的配置文件跟踪列表。相对文件是指相对于$ {安装路径}

启用日志追踪 - 检查启用日志追踪

跟踪事件日志级别 - 仅跟踪时间级别等于或大于这个级别.顺序是: [Error, Warn, Info, Debug]

映射:

FATAL,ERROR -> Error

WARN -> Warn

INFO -> Info

DEBUG -> Debug

登录模式匹配 - 包括匹配给定的正则表达式的消息。给定的模式可以是日志消息或正则表达式的子串。请参见:java.util.regex.Pattern.

登录模式排除 - 排除匹配给定的正则表达式的消息。

日志文件 - 逗号分隔的日志文件跟踪列表。相对文件是指相对于$ {安装路径}。

Hadoop Job Tracker RPC服务

日志追踪支持: Yes

配置追踪支持: Yes

定制属性支持:None

支持的控制操作: None

支持的参数:

Name Alias Units Category Default On Default Interval
Availability Availability percentage AVAILABILITY true 10 min
Num Open Connections NumOpenConnections none UTILIZATION true 5 min
Call Queue Len CallQueueLen none UTILIZATION false 5 min
Rpc Ops Avg Processing Time RpcOpsAvgProcessingTime ms UTILIZATION false 5 min
Rpc Ops Avg Processing Time Max RpcOpsAvgProcessingTimeMax ms UTILIZATION false 5 min
Rpc Ops Avg Processing Time Min RpcOpsAvgProcessingTimeMin ms UTILIZATION false 5 min
Rpc Ops Avg Queue Time RpcOpsAvgQueueTime ms UTILIZATION true 5 min
Rpc Ops Avg Queue Time Max RpcOpsAvgQueueTimeMax ms UTILIZATION false 5 min
Rpc Ops Avg Queue Time Min RpcOpsAvgQueueTimeMin ms UTILIZATION false 5 min
Rpc Ops Number RpcOpsNumber none UTILIZATION false 5 min

Hadoop Name Node服务器

日志追踪支持: Yes

配置追踪支持: Yes

定制属性支持:

支持的控制操作: None

支持的参数:

Name Alias Units Category Default On Default Interval
Process Virtual Memory Size ProcessVirtualMemorySize B UTILIZATION false 5 min
Process Resident Memory Size ProcessResidentMemorySize B UTILIZATION true 5 min
Process Page Faults ProcessPageFaults none UTILIZATION false 10 min
Process Page Faults per Minute ProcessPageFaults1m none UTILIZATION false 10 min
Process Cpu System Time ProcessCpuSystemTime ms UTILIZATION false 10 min
Process Cpu System Time per Minute ProcessCpuSystemTime1m ms UTILIZATION false 10 min
Process Cpu User Time ProcessCpuUserTime ms UTILIZATION false 10 min
Process Cpu User Time per Minute ProcessCpuUserTime1m ms UTILIZATION false 10 min
Process Cpu Total Time ProcessCpuTotalTime ms UTILIZATION false 10 min
Process Cpu Total Time per Minute ProcessCpuTotalTime1m ms UTILIZATION false 10 min
Process Cpu Usage ProcessCpuUsage percentage UTILIZATION true 5 min
Process Start Time ProcessStartTime epoch-millis AVAILABILITY false 5 min
Process Open File Descriptors ProcessOpenFileDescriptors none UTILIZATION false 5 min
Process Threads ProcessThreads none UTILIZATION false 5 min
Availability Availability percentage AVAILABILITY true 5 min
Heap Memory Free HeapMemoryFree B UTILIZATION true 5 min
Thread Count ThreadCount none UTILIZATION true 5 min
Total HotSpot Compilation Time TotalHotSpotCompilationTime ms PERFORMANCE true 5 min

一般日志和配置追踪属性

启用配置追踪 - 检查启用配置追踪

配置文件 - 逗号分隔的配置文件跟踪列表。相对文件是指相对于$ {安装路径}

启用日志追踪 - 检查启用日志追踪

跟踪事件日志级别 - 仅跟踪时间级别等于或大于这个级别.顺序是: [Error, Warn, Info, Debug]

映射:

FATAL,ERROR -> Error

WARN -> Warn

INFO -> Info

DEBUG -> Debug

登录模式匹配 - 包括匹配给定的正则表达式的消息。给定的模式可以是日志消息或正则表达式的子串。请参见:java.util.regex.Pattern.

登录模式排除 - 排除匹配给定的正则表达式的消息。

日志文件 - 逗号分隔的日志文件跟踪列表。相对文件是指相对于$ {安装路径}。

Hadoop Name Node FS Namesystem服务

日志追踪支持: No

配置追踪支持: No

定制属性支持: None

支持的控制操作: None

支持的参数:

Name Alias Units Category Default On Default Interval
Availability Availability percentage AVAILABILITY true 10 min
Capacity Total CapacityTotal B UTILIZATION false 5 min
Capacity Used CapacityUsed B UTILIZATION false 5 min
Capacity Remaining CapacityRemaining B UTILIZATION true 5 min
Total Load TotalLoad none UTILIZATION true 5 min
Blocks Total BlocksTotal none UTILIZATION false 5 min
Files Total FilesTotal none UTILIZATION false 5 min
Pending Replication Blocks PendingReplicationBlocks none UTILIZATION true 5 min
Under Replicated Blocks UnderReplicatedBlocks none UTILIZATION false 5 min
Scheduled Replication Blocks ScheduledReplicationBlocks none UTILIZATION false 5 min

Hadoop Name Node RP服务

日志追踪支持: No

配置追踪支持: No

定制属性支持: None

支持的控制操作: None

支持的参数:

Name Alias Units Category Default On Default Interval
Availability Availability percentage AVAILABILITY true 10 min
Num Open Connections NumOpenConnections none UTILIZATION true 5 min
Call Queue Len CallQueueLen none UTILIZATION false 5 min
Rpc Ops Avg Processing Time RpcOpsAvgProcessingTime ms UTILIZATION false 5 min
Rpc Ops Avg Processing Time Max RpcOpsAvgProcessingTimeMax ms UTILIZATION false 5 min
Rpc Ops Avg Processing Time Min RpcOpsAvgProcessingTimeMin ms UTILIZATION false 5 min
Rpc Ops Avg Queue Time RpcOpsAvgQueueTime ms UTILIZATION true 5 min
Rpc Ops Avg Queue Time Max RpcOpsAvgQueueTimeMax ms UTILIZATION false 5 min
Rpc Ops Avg Queue Time Min RpcOpsAvgQueueTimeMin ms UTILIZATION false 5 min
Rpc Ops Number RpcOpsNumber none UTILIZATION false 5 min

Hadoop命名节点统计服务

日志追踪支持: No

配置追踪支持: No

定制属性支持: None

支持的控制操作: None

支持的参数:

Name Alias Units Category Default On Default Interval
Availability Availability percentage AVAILABILITY true 10 min
Block Report Average Time BlockReportAverageTime ms UTILIZATION true 5 min
Block Report Max Time BlockReportMaxTime ms UTILIZATION false 5 min
Block Report Min Time BlockReportMinTime ms UTILIZATION false 5 min
Block Report Num BlockReportNum none UTILIZATION false 10 min
Block Report Num per Minute BlockReportNum1m none UTILIZATION false 10 min
Journal Transaction Average Time JournalTransactionAverageTime ms UTILIZATION true 5 min
Journal Transaction Num JournalTransactionNum none UTILIZATION false 10 min
Journal Transaction Num per Minute JournalTransactionNum1m none UTILIZATION false 10 min
Journal Transaction Max Time JournalTransactionMaxTime ms UTILIZATION false 5 min
Journal Transaction Min Time JournalTransactionMinTime ms UTILIZATION false 5 min
Journal Sync Average Time JournalSyncAverageTime ms UTILIZATION false 5 min
Journal Sync Max Time JournalSyncMaxTime ms UTILIZATION false 5 min
Journal Sync Min Time JournalSyncMinTime ms UTILIZATION false 5 min
Journal Sync Num JournalSyncNum none UTILIZATION false 10 min
Journal Sync Num per Minute JournalSyncNum1m none UTILIZATION false 10 min
Safemode Time SafemodeTime ms UTILIZATION false 5 min
FS Image Load Time FSImageLoadTime ms UTILIZATION false 5 min
Num Files Created NumFilesCreated none UTILIZATION false 10 min
Num Files Created per Minute NumFilesCreated1m none UTILIZATION false 10 min
Num Files Listed NumFilesListed none UTILIZATION false 10 min
Num Files Listed per Minute NumFilesListed1m none UTILIZATION false 10 min
Num Get Listing Ops NumGetListingOps none UTILIZATION false 10 min
Num Get Listing Ops per Minute NumGetListingOps1m none UTILIZATION false 10 min
Num Create File Ops NumCreateFileOps none UTILIZATION false 10 min
Num Create File Ops per Minute NumCreateFileOps1m none UTILIZATION false 10 min
Num Delete File Ops NumDeleteFileOps none UTILIZATION false 10 min
Num Delete File Ops per Minute NumDeleteFileOps1m none UTILIZATION false 10 min
Num Add Block Ops NumAddBlockOps none UTILIZATION false 10 min
Num Add Block Ops per Minute NumAddBlockOps1m none UTILIZATION false 10 min
Num Get Block Locations NumGetBlockLocations none UTILIZATION false 10 min
Num Get Block Locations per Minute NumGetBlockLocations1m none UTILIZATION false 10 min
Num Files Renamed NumFilesRenamed none UTILIZATION false 10 min
Num Files Renamed per Minute NumFilesRenamed1m none UTILIZATION false 10 min