大数据之elasticsearch集群搭建与基本使用-渗透人员入门

wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.1.tar.gz
tar zxvf elasticsearch-1.5.1.tar.gz
cd elasticsearch-1.5.1

vim bin/elasticsearch
修改
JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64（替换为实际的JAVA安装目录）

vim bin/plugin
修改
JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64（替换为实际的JAVA安装目录）

vim config/elasticsearch.yml
做如下修改
cluster.name: elasticsearch-tanjiti  配置es的集群名称，默认是elasticsearch，es会自动发现在同一网段下的es，如果在同一网段下有多个集群，就可以用这个属性来区分不同的集群
node.name: "tanjiti No.00" 节点名，默认随机指定一个name列表中名字，该列表在es的jar包中config文件夹里name.txt文件中，其中有很多作者添加的有趣名字。
index.number_of_shards: 3 设置默认索引分片个数，默认为5片。
path.logs: /home/elasticsearch-1.5.1/logs 设置日志文件的存储路径，默认是es根目录下的logs文件夹
path.plugins: /home/elasticsearch-1.5.1/plugins设置插件的存放路径，默认是es根目录下的plugins文件夹
#bootstrap.mlockall: true  设置为true来锁住内存。因为当jvm开始swapping时es的效率会降低，所以要保证它不swap，因为发现开启这个选项会莫名的错误，所以选择了关闭，依靠把ES_MIN_MEM和ES_MAX_MEM两个环境变量设置成同一个足够大的值来保证机器有足够的内存分配给es。
transport.tcp.port: 9310  设置节点间交互的tcp端口，默认是9300。
http.port: 8765 设置对外服务的http端口，默认为9200
discovery.zen.ping.timeout: 30s 设置集群中自动发现其它节点时ping连接超时时间，默认为3秒，对于比较差的网络环境可以高点的值来防止自动发现时出错。
discovery.zen.ping.multicast.enabled: false 设置是否打开多播发现节点，默认是false,开启单播模式
discovery.zen.ping.unicast.hosts: ["xxx.xxx.xxx.xxx", "xxx.xxx.xxx.xxx", "xxx.xxx.xxx.xxx"]  设置集群中master节点的初始列表，可以通过这些节点来自动发现新加入集群的节点

vim bin/elasticsearch.in.sh
修改
if [ "x$ES_MIN_MEM" = "x" ]; then

    ES_MIN_MEM=256m #调整为机器内存的一半

fi

if [ "x$ES_MAX_MEM" = "x" ]; then

    ES_MAX_MEM=1g#调整为机器内存的一半 

fi

 bin/plugin -i elasticsearch/marvel/latest

wget https://download.elasticsearch.org/elasticsearch/marvel/marvel-latest.zip
bin/plugin -i marvel -u file:/home/elasticsearch-1.5.1/marvel-latest.zip 

mv elasticsearch-servicewrapper-master/service/ bin/

vim bin/service/elasticsearch.conf 
按需作如下修改
set.default.ES_HOME=/home/elasticsearch-1.5.1 #替换为实际的elasticsearch路径
wrapper.java.command=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java #替换为实际的java二进制文件路径

bin/plugin -i royrusso/elasticsearch-HQ -u file:/home/elasticsearch-1.5.1/royrusso-elasticsearch-HQ-603ae9e.zip 

bin/service/elasticsearch  start|stop|console|install|remove

start 在后台运行elasticsearch
stop 停止elasticsearch
console 在前台运行elasticsearch
install elasticsearch自启动
remove elasticsearch取消自启动

wget http://www.elasticsearch.org/guide/en/kibana/3.0/snippets/shakespeare.json
curl -XPUT http://localhost:8765/_bulk --data-binary @shakespeare.json

more shakespeare.json 察看存储内容
{"index":{"_index":"shakespeare","_type":"act","_id":0}}
{"line_id":1,"play_name":"Henry IV","speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"}
{"index":{"_index":"shakespeare","_type":"scene","_id":1}}

http localhost:8765/shakespeare/

{
    "shakespeare": {
        "mappings": {
            "act": {
                "properties": {
                    "line_id": {
                        "type": "long"
                    }, 
                    "line_number": {
                        "type": "string"
                    }, 
                    "play_name": {
                        "type": "string"
                    }, 
                    "speaker": {
                        "type": "string"
                    }, 
                    "speech_number": {
                        "type": "long"
                    }, 
                    "text_entry": {
                        "type": "string"
                    }
                }
            }, 
            "line": {
                "properties": {
                    "line_id": {
                        "type": "long"
                    }, 
                    "line_number": {
                        "type": "string"
                    }, 
                    "play_name": {
                        "type": "string"
                    }, 
                    "speaker": {
                        "type": "string"
                    }, 
                    "speech_number": {
                        "type": "long"
                    }, 
                    "text_entry": {
                        "type": "string"
                    }
                }
            }, 
            "scene": {
                "properties": {
                    "line_id": {
                        "type": "long"
                    }, 
                    "line_number": {
                        "type": "string"
                    }, 
                    "play_name": {
                        "type": "string"
                    }, 
                    "speaker": {
                        "type": "string"
                    }, 
                    "speech_number": {
                        "type": "long"
                    }, 
                    "text_entry": {
                        "type": "string"
                    }
                }
            }
        }, 
        "settings": {
            "index": {
                "creation_date": "1429691321987", 
                "number_of_replicas": "1", 
                "number_of_shards": "5", 
                "uuid": "rrCmsKKcSDyLSpLFVnQnbg", 
                "version": {
                    "created": "1040299"
                }
            }
        }
    }
}

http localhost:8108/shakespeare/line/2

{
    "_id": "2", 
    "_index": "shakespeare", 
    "_source": {
        "line_id": 3, 
        "line_number": "", 
        "play_name": "Henry IV", 
        "speaker": "", 
        "speech_number": "", 
        "text_entry": "Enter KING HENRY, LORD JOHN OF LANCASTER, the EARL of WESTMORELAND, SIR WALTER BLUNT, and others"
    }, 
    "_type": "line", 
    "_version": 1, 
    "found": true
}

http localhost:8108/shakespeare/line/2/_source

{
    "line_id": 3, 
    "line_number": "", 
    "play_name": "Henry IV", 
    "speaker": "", 
    "speech_number": "", 
    "text_entry": "Enter KING HENRY, LORD JOHN OF LANCASTER, the EARL of WESTMORELAND, SIR WALTER BLUNT, and others"
}

http localhost:8108/shakespeare/_search?q=play_name:"Romeo and Juliet"

{
    "_shards": {
        "failed": 0, 
        "successful": 5, 
        "total": 5
    }, 
    "hits": {
        "hits": [
            {
                "_id": "86748", 
                "_index": "shakespeare", 
                "_score": 3.3792284, 
                "_source": {
                    "line_id": 86749, 
                    "line_number": "", 
                    "play_name": "Romeo and Juliet", 
                    "speaker": "JULIET", 
                    "speech_number": 19, 
                    "text_entry": "Exeunt"
                }, 
                "_type": "line"
            }, 

支持AND，OR，NOT
查询语句"apache AND lucene"的意思是匹配含apache且含lucene的文档。
查询表达式"apache OR lucene"能够匹配包含“apache”的文档，也能匹配包含"lucene"的文档，还能匹配同时包含这两个Term的文档。
查询表达式“lucene NOT elasticsearch”就只能匹配包含lucene但是不含elasticsearch的文档

支持+, -符号
例如：希望搜索到包含关键词lucene,但是不含关键词elasticsearch的文档，可以用如下的查询表达式："+lucene -elasticsearch"。

支持指定字段名进行搜索（类似RDBS按列名搜索）
例如：查询title域中包含关键词elasticsearch的文档，查询表达式如下：title:elasticsearch

支持通配符
 ? (匹配单个字符）
* (匹配多个字符）
注意默认的通配符不能是关键词的首字母

支持~整数符号
一个~符号，后面紧跟一个整数，~后面的整数表示短语中可接收的最大的词编辑距离(短语中替换一个词，添加一个词，删除一个词)
"writer~2"能够搜索到含writer和writers的文档。
title:"mastering elasticsearch"~2能够搜匹配title域中含"mastering elasticsearch"的文档与包含"mastering book elasticsearch"的文档

支持^符号进行加权boost设置
一个^符号后面接一个浮点数表示权重。如果权重小于1，就会降低关键词的重要程度。同理，如果权重大于1就会增加关键词的重要程度。默认的加权值为1

支持区间搜索
price:[10.00 TO 15.00查询price域的值在10.00到15.00之间的所有文档。
price:[10.00 TO 15.00}查询price域中价格在10.00(10.00要能够被搜索到)到15.00(15.00不能被搜索到)之间的文档

特殊字符需转义
+, -, &&, || , ! , (,) , { } , [ ] , ^, " , ~, *, ?, : , \, /

curl -XPOST 'http://localhost:8108/shakespeare/line/_search?pretty' -d '
{
"query":{ "match_all": {} },
"sort": {"line_id": {"order": "desc" }},
"size": 1,
"from": 10
}'

use shakespeare;
select * 
from line
order by line_id desc
limit 10,1

curl -XPOST 'http://localhost:8108/shakespeare/line/_search?pretty' -d ' 
{
"query":{ 
"bool":{
"must":[
{"match_phrase": {"text_entry":"question"}},
{"match_phrase": {"text_entry":"not to be"}}
]
}
}
}'

  "took" : 253,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 4.0433946,
    "hits" : [ {
      "_index" : "shakespeare",
      "_type" : "line",
      "_id" : "34229",
      "_score" : 4.0433946,
      "_source":{"line_id":34230,"play_name":"Hamlet","speech_number":19,"line_number":"3.1.64","speaker":"HAMLET","text_entry":"To be, or not to be: that is the question:"}
    }, {
      "_index" : "shakespeare",
      "_type" : "line",
      "_id" : "1397",
      "_score" : 4.0004296,
      "_source":{"line_id":1398,"play_name":"Henry IV","speech_number":152,"line_number":"2.4.392","speaker":"FALSTAFF","text_entry":"blackberries? a question not to be asked. Shall"}
    } ]
  }
}

use shakespeare;
select * 
from line
where text_entry like "%question%" and text_entry like "%not to be%"