Tutorial¶
This tutorial demonstrates how you can: 1. convert a pretrained Keras model to SoftNeuro's DNN format 2. tune the converted model for speed optimization. 3. use the converted model for iimage classification 4. encrypt a DNN file
It is assumed that SoftNeuro Python package and the CLI tool for the given platform have been properly installed.
Keras model conversion¶
Convert the Keras model to DNN format Using the import-tensorflow command of the CLI tool.
In this tutorial, we convert the VGG16 model of keras.applications.
(softneuro)$ softneuro import-tensorflow --keras vgg16 vgg16.dnn
Tuning and Profiling for the target platform¶
During this process, the model file will be tuned to achieve optimal performance
for the target platform.
Since the tool with calculating and comparing processing times for calculations
under various settings, it is recommended that you close all non-essential applications
during this task.
When tuning is completed, the tuned model file vgg16_tuned.dnn will be saved
to the current directory.
(softneuro)$ softneuro tune vgg16.dnn vgg16_tuned.dnn
Profiling...100.0% [00:50]
[preprocessor]
# NAME SCHEMA ATIME TIME DESC PARAMS
0 input (source) * cpu
1 premute (permute) * cpu 136 cpu/naive
2 madd (madd) * cpu 18 cpu/avx {"ops_in_task":16384}
3 output (sink) * cpu
4 ? ()
154
[main]
# NAME SCHEMA ATIME TIME DESC PARAMS
0 input_1 (source) * cpu
1 block1_conv1 (conv2) * cpu 777 cpu/owc64_avx {"cache":8192,"task_ops":65536}
2 block1_conv2 (conv2) * cpu 5,680 cpu/wg2 {"tile_size":8}
3 block1_pool (max_pool2) * cpu 454 cpu/avx
4 block2_conv1 (conv2) * cpu 1,783 cpu/wg2 {"tile_size":8}
5 block2_conv2 (conv2) * cpu 2,930 cpu/wg2 {"tile_size":8}
6 block2_pool (max_pool2) * cpu 70 cpu/avx
7 block3_conv1 (conv2) * cpu 1,294 cpu/wg2 {"tile_size":8}
8 block3_conv2 (conv2) * cpu 2,280 cpu/wg2 {"tile_size":8}
9 block3_conv3 (conv2) * cpu 2,280 cpu/wg2 {"tile_size":8}
10 block3_pool (max_pool2) * cpu 45 cpu/avx
11 block4_conv1 (conv2) * cpu 1,362 cpu/wg2 {"tile_size":4}
12 block4_conv2 (conv2) * cpu 2,616 cpu/wg2 {"tile_size":4}
13 block4_conv3 (conv2) * cpu 2,616 cpu/wg2 {"tile_size":4}
14 block4_pool (max_pool2) * cpu 31 cpu/avx
15 block5_conv1 (conv2) * cpu 1,127 cpu/wg2 {"tile_size":2}
16 block5_conv2 (conv2) * cpu 1,127 cpu/wg2 {"tile_size":2}
17 block5_conv3 (conv2) * cpu 1,127 cpu/wg2 {"tile_size":2}
18 block5_pool (max_pool2) * cpu 22 cpu/avx
19 flatten (reshape) * cpu 0 cpu
20 fc1 (dense) * cpu 11,669 cpu/m1x1l_avx
21 fc2 (dense) * cpu 1,716 cpu/m1x1l_avx
22 predictions (dense) * cpu 210 cpu/avx {"task_ops":32}
23 predictions_softmax (softmax) * cpu 12 cpu/naive
24 sink_0 (sink) * cpu
25 ? ()
41,229
SCHEMAS cpu
TOTAL 41,383
You can use attrcommand to check the status of tuning (see the TUNE column).
(softneuro)$ softneuro attr -a vgg16_tuned.dnn
NAME NET INPUT OUTPUT #OPS CIPHER COMPRESS TUNE #THR AFFMASKS
vgg16_tuned.dnn preprocessor 1x224x224x3 1x1000 30,953,174,639 cpu
main cpu/naive
If you want to save the profiling information, you manually initialize and execute the tuning process as follows.
(softneuro)$ softneuro init mobilenet.prof mobilenet.dnn
(softneuro)$ softneuro profile mobilenet.prof
(softneuro)$ softneuro tune mobilenet.dnn mobilenet_tuned.dnn --prof mobilenet.prof
Verifying the results of tuning¶
We perform inference on the sample image shovel.jpg to compare the performance
of models before and after tuning.
Inference with the model before tuning
(softneuro)$ softneuro run vgg16.dnn shovel.jpg --top 5
---------------------------------
Top 5 Labels
---------------------------------
# SCORE LABEL
1 0.9948 shovel
2 0.0017 hatchet
3 0.0012 swab
4 0.0011 broom
5 0.0007 hammer
---------------------------------
Statistics
---------------------------------
FUNCTION AVE(us) MIN(us) MAX(us) #RUN
Dnn_load() 326,679 326,679 326,679 1
Dnn_compile() 333,007 333,007 333,007 1
Dnn_forward() 199,992 199,992 199,992 1
Used memory: 1,140,817,920 Bytes
Inference with the model after tuning
(softneuro)$ softneuro run vgg16_tuned.dnn shovel.jpg --top 5
---------------------------------
Top 5 Labels
---------------------------------
# SCORE LABEL
1 0.9948 shovel
2 0.0016 hatchet
3 0.0012 swab
4 0.0011 broom
5 0.0007 hammer
---------------------------------
Statistics
---------------------------------
FUNCTION AVE(us) MIN(us) MAX(us) #RUN
Dnn_load() 325,598 325,598 325,598 1
Dnn_compile() 446,959 446,959 446,959 1
Dnn_forward() 68,648 68,648 68,648 1
Used memory: 890,712,064 Bytes
You can compare the performance using the Statistics sections of the command line outputs. Note the reduction in inference time (Dnn_forward) and memory consumption (Used memory).
For more stable statistics, you can use the -loop command option.
Please refer to run command options for
more details.
DNN file encryption¶
You can use the cipher command for encrypting a DNN file to secure its parameters.
After completion the file vgg16_tuned_enc.dnn will be saved in the current directory.
The password specified during encryption is required for inference. Further, the
view command cannot be used to see model information of an encrypted DNN file.
(softneuro)$ softneuro cipher --new-pass 123456 vgg16_tuned.dnn vgg16_tuned_enc.dnn
(softneuro)$ softneuro attr -a vgg16_tuned_enc.dnn
NAME NET INPUT OUTPUT #OPS CIPHER COMPRESS TUNE #THR AFFMASKS
vgg16_tuned_enc.dnn ? password (invalid)
(softneuro)$ softneuro view vgg16_tuned_enc.dnn
mor_dnn.c:908:Error: Invalid password for dnn.
(softneuro)$ softneuro run vgg16_tuned_enc.dnn -p 123456
---------------------------------
Statistics
---------------------------------
FUNCTION AVE(us) MIN(us) MAX(us) #RUN
Dnn_load() 419,614 419,614 419,614 1
Dnn_compile() 860,013 860,013 860,013 1
Dnn_forward() 203,562 203,562 203,562 1
Used memory: 1,138,716,672 Bytes