antinex-core 1.3.13

精确性+培训+Jupyter笔记本中的交叉验证

https://github.com/jay-johnson/antinex-core/blob/master/docker/notebooks/AntiNex-Protecting-Django.ipynb

在Jupyter笔记本中使用预先训练好的深层神经网络

https://github.com/jay-johnson/antinex-core/blob/master/docker/notebooks/AntiNex-Using-Pre-Trained-Deep-Neural-Networks-For-Defense.ipynb

概述

核心是一个芹菜工人池，用于处理深层神经网络的训练和预测请求，以近乎实时地使用keras和tensorflow检测网络开发（nex）。在内部，每个工人管理一个缓冲区，缓冲区由初始培训请求中的label标识。一旦经过训练，只要对预测请求使用相同的label名称，就可以使用模型进行快速预测测试。还可以使用具有相同label的训练api重新训练模型。虽然最初的重点是网络开发，但存储库还包括模拟股票数据，用于演示如何运行一个工人池，以便使用许多经过预训练的深层神经网络快速预测回归数据（如股票价格）。

此存储库是一个独立的培训和预测工作池，它与antinex rest api分离：

https://github.com/jay-johnson/train-ai-with-django-swagger-jwt

Antinex堆栈状态

antinex核心工作进程是antinex堆栈的一部分：

Component	Build	Docs Link	Docs Build
REST API		Docs
Core Worker		Docs
Network Pipeline		Docs
AI Utils		Docs
Client		Docs

安装

pip安装antinex核心

可用于生成图像

如果你想生成图片，请在ubuntu上安装python3-tk。

sudo apt-get install python3-tk

码头工人

使用jupyter启动用于浏览的容器：

# if you do not have docker compose installed, you can try installing it with:
# pip install docker-compose
cd docker
./start-stack.sh

使用django deep神经网络分析打开jupyter笔记本

默认密码是：admin

http://localhost:8888/notebooks/AntiNex-Protecting-Django.ipynb

查看笔记本演示文稿幻灯片

1. 在笔记本中使用Alt + r。
2. 使用非垂直的scolling url：http://localhost:8889/Slides-AntiNex-Protecting-Django.slides.html
3. 使用非垂直的scolling url：http://localhost:8890/Slides-AntiNex-Using-Pre-Trained-Deep-Neural-Networks-For-Defense.slides.html

运行

启动Core之前，请确保Redis正在运行并可访问：

redis-cli
127.0.0.1:6379>

运行redis并在python 3运行时安装antinex core pip后，使用此命令启动核心：

./run-antinex-core.sh

或芹菜：

celery worker -A antinex_core.antinex_worker -l DEBUG

发布预测请求

要使用新的自动定标器规范化数据集进行训练和预测，其预测精度为99.8%，可以使用带有AntiNex datasets的宽两层深层神经网络检测攻击，请执行以下步骤。

克隆

请确保将数据集repo克隆到预先配置的位置：

mkdir -p -m 777 /opt/antinex
git clone https://github.com/jay-johnson/antinex-datasets.git /opt/antinex/antinex-datasets

Django-训练和预测

./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-django-antinex-simple.json

烧瓶-训练和预测

./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-flask-antinex-simple.json

反应和重复-训练和预测

./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-react-redux-antinex-simple.json

Vue-训练和预测

./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-vue-antinex-simple.json

弹簧-训练和预测

./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-spring-antinex-simple.json

准确度和预测报告

几分钟后，最终报告将打印出来，如下所示：

2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30178 - label_value=1.0 predicted=1 label=attack
2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30179 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30180 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30181 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30182 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30183 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30184 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30185 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30186 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30187 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30188 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30189 - label_value=1.0 predicted=1 label=attack
2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30190 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30191 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30192 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30193 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30194 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30195 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30196 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30197 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30198 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30199 - label_value=-1.0 predicted=-1 label=not_attack
2018-03-11 23:35:00,947 - antinex-prc - INFO - Full-Django-AntiNex-Simple-Scaler-DNN made predictions=30200 found=30200 accuracy=99.84685430463577
2018-03-11 23:35:00,947 - antinex-prc - INFO - Full-Django-AntiNex-Simple-Scaler-DNN - saving model=full-django-antinex-simple-scaler-dnn

如果没有在本地克隆数据集，则可以使用repo中包含的最小化数据集：

./antinex_core/scripts/publish_predict_request.py -f training/scaler-django-antinex-simple.json

发布列车请求

./antinex_core/scripts/publish_train_request.py

发布回归预测请求

./antinex_core/scripts/publish_regression_predict.py

JSON API

antinex核心管理一个工作池，这些工作池订阅了两个队列中的进程任务（webapp.train.requests和webapp.predict.requests）。任务定义为JSON字典，必须具有以下结构：

{
    "label": "Django-AntiNex-Simple-Scaler-DNN",
    "dataset": "./tests/datasets/classification/cleaned_attack_scans.csv",
    "apply_scaler": true,
    "ml_type": "classification",
    "predict_feature": "label_value",
    "features_to_process": [
        "eth_type",
        "idx",
        "ip_ihl",
        "ip_len",
        "ip_tos",
        "ip_version",
        "tcp_dport",
        "tcp_fields_options.MSS",
        "tcp_fields_options.Timestamp",
        "tcp_fields_options.WScale",
        "tcp_seq",
        "tcp_sport"
    ],
    "ignore_features": [
    ],
    "sort_values": [
    ],
    "seed": 42,
    "test_size": 0.2,
    "batch_size": 32,
    "epochs": 10,
    "num_splits": 2,
    "loss": "binary_crossentropy",
    "optimizer": "adam",
    "metrics": [
        "accuracy"
    ],
    "histories": [
        "val_loss",
        "val_acc",
        "loss",
        "acc"
    ],
    "model_desc": {
        "layers": [
            {
                "num_neurons": 250,
                "init": "uniform",
                "activation": "relu"
            },
            {
                "num_neurons": 1,
                "init": "uniform",
                "activation": "sigmoid"
            }
        ]
    },
    "label_rules": {
        "labels": [
            "not_attack",
            "not_attack",
            "attack"
        ],
        "label_values": [
            -1,
            0,
            1
        ]
    },
    "version": 1
}

还支持回归预测任务，下面是一个包含模拟股价的数据集的示例：

{
    "label": "Scaler-Close-Regression",
    "dataset": "./tests/datasets/regression/stock.csv",
    "apply_scaler": true,
    "ml_type": "regression",
    "predict_feature": "close",
    "features_to_process": [
        "high",
        "low",
        "open",
        "volume"
    ],
    "ignore_features": [
    ],
    "sort_values": [
    ],
    "seed": 7,
    "test_size": 0.2,
    "batch_size": 32,
    "epochs": 50,
    "num_splits": 2,
    "loss": "mse",
    "optimizer": "adam",
    "metrics": [
        "accuracy"
    ],
    "model_desc": {
        "layers": [
            {
                "activation": "relu",
                "init": "uniform",
                "num_neurons": 200
            },
            {
                "activation": null,
                "init": "uniform",
                "num_neurons": 1
            }
        ]
    }
}

splunk环境变量

此存储库使用Spylunking记录器，该记录器支持将日志发布到经过身份验证的hec rest api上。您可以将这些环境变量设置为发布到splunk:

export SPLUNK_ADDRESS="<splunk address host:port>"
export SPLUNK_API_ADDRESS="<splunk api address host:port>"
export SPLUNK_USER="<splunk username for login>"
export SPLUNK_PASSWORD="<splunk password for login>"
export SPLUNK_TOKEN="<Optional - username and password will login or you can use a pre-existing splunk token>"
export SPLUNK_INDEX="<splunk index>"
export SPLUNK_QUEUE_SIZE="<num msgs allowed in queue - 0=infinite>"
export SPLUNK_RETRY_COUNT="<attempts per log to retry publishing>"
export SPLUNK_RETRY_BACKOFF="<cooldown in seconds per failed POST>"
export SPLUNK_SLEEP_INTERVAL="<sleep in seconds per batch>"
export SPLUNK_SOURCE="<splunk source>"
export SPLUNK_SOURCETYPE="<splunk sourcetype>"
export SPLUNK_TIMEOUT="<timeout in seconds>"
export SPLUNK_DEBUG="<1 enable debug|0 off - very verbose logging in the Splunk Publishers>"

开发

virtualenv -p python3 ~/.venvs/antinexcore && source ~/.venvs/antinexcore/bin/activate && pip install -e .

测试

全部运行

python setup.py test

运行测试用例

python -m unittest tests.test_train.TestTrain.test_train_antinex_simple_success_retrain

起毛

薄片8。

PycodeStyle。

许可证

apache 2.0-有关详细信息，请参阅LICENSE