nwztools/database: misc improvements

* make gen_db.py work on Windows/Python 2

- use hashlib module instead of md5sum, also don't rely on / for file path
matching
- don't use 'file' for a variable name

* fix parse_nvp_header.sh for older kernels

pre-emmc kernel sources use a slightly different #define format; adjust
regexp to catch it.

* add nwz-x1000 series NVP layout (from icx1087_nvp.h)

some new tags have no description, alas the driver doesn't have
them :/

*  minor fixes to nvp/README

fixed typos/wording

Change-Id: I77d8c2704be2f2316e32aadcfd362df7102360d4

utils/nwztools/database/nvp/README

@@ -1,38 +1,39 @@
 The NVP map varies a lot from players to players, it is inconceivable to build
 it by hand. The approach taken is to extract it from the kernel of each player.
 Since Sony provides the kernel of all players, it is 'only' a matter of
-downloading all of them. A bit of back story on the NVP: it is non-volatile
-area of the flash that is divided in regions and then "sectors" (unrelated to
-hard drive sectors). Each "sector" stores the data of a "node". The ABI
+downloading all of them. A bit of background on the NVP: it is non-volatile 
+partition of the flash that is divided in regions and then "zones". 
+Each "zone" stores the data of a "node". The ABI
 between the NVP driver and the userspace is an index: the userspace gives the
-index of a node, and then drives looksup its table to see where it is and what
+index of a node, and then drives looks up its table to see where it is and what
 is its size. The index map changes over time so Sony introduces standard "names"
 for its entries, those are 3-letters acronym (for example "fup" or "bti" or "shp")
-that have a corresponding index. For some reason, the driver also contains a
-description of the nodes, in english (so "bti" stands for "boot image").
+that have a corresponding index. Sometimes the driver also contains a
+description of the nodes, in English (e.g. "bti" stands for "boot image").
 
 parse_nvp_header.sh
 ===================
 
-This script is given a file name, a kernel directory or a kernel tgz and it will
+This script takes a header filename, a kernel directory or a kernel tgz and will
 try to extract the mapping automatically. It produces a list of pairs
   <node>,<name>
 where <node> is the index of the node (that's the only thing that is usable on
-a running device) and <name> is the standard name of the node. Note that is
-some kind of acronym (like 'fup') and the description needs to be generated
-separatly (see other section).
+a running device) and <name> is the standard name of the node. The output should
+be written to <series name>.txt
+Note that <name> is an acronym (like 'fup') and the description needs to be generated
+separately (see nvprool section).
 
 parse_all_nvp_headers.sh
 ========================
 
-This scripts expects a directory to have the following structure:
+This script expects a directory to have the following structure:
   dir/
     nwz-a10/
       linux-kernel-*.tgz
     nwz-e460/
-      linxu-kernel-*.tgz
+      linux-kernel-*.tgz
     ...
-Each sudirectory must the series name (as used in ../series.txt) and the kernel
+Each sudirectory must be the series name (as used in ../series.txt) and the kernel
 must be a tgz (end in .tgz and not .tar.gz) of the form linux-kernel-*.tgz. Usually
 the variable bit will be the version but some kernels have unknown versions. It
 will then run parse_nvp_header.sh on each of them and store the result in a
@@ -43,24 +44,24 @@ NOTE: the kernel can be symlinks to other files
 nvptool
 =======
 
-The kernel headers do no contain the description of the nvp node names.
-This one can be extract from the icx_nvp[_emmc].ko driver on target using complicated
-elf parsing done by nvptool. Technically nvptoo can find much more information
-like the node -> humanname mapping as well and the actual sector on the disk but
+The kernel headers do not contain descriptions of the nvp node names.
+They can be extracted from the icx_nvp[_emmc].ko driver on target using complicated
+elf parsing done by nvptool. Technically nvptool can discover much more information
+like the node -> human name mapping as well and the actual zone in the flash but
 since we can already extract it easily from the headers, we only extract description
 names from it.
 
 parse_all_nvp_nodes.sh
 ======================
 
-This scripts expects a directory to have the following structure:
+This script expects a directory to have the following structure:
   dir/
     nwz-a10/
       rootfs.tgz
     nwz-e460/
       rootfs.tgz
     ...
-Each sudirectory must the series name (as used in ../series.txt) and the rootfs
+Each sudirectory must be the series name (as used in ../series.txt) and the rootfs
 must be a tar. It will then extract the relevant icx_nvp driver from it and run
 nvptool on it to produce a file called nodes-<series name>.txt
 

utils/nwztools/database/nvp/nwz-x1000.txt

@@ -0,0 +1,82 @@
+app,0
+bti,1
+hdi,2
+cng,3
+ser,4
+dg0,5
+dg1,6
+dcc,7
+mdl,8
+fup,9
+bok,10
+shp,11
+dba,12
+dbv,13
+tr0,14
+tr1,15
+mid,16
+tst,17
+gty,18
+fui,19
+lbi,20
+dor,21
+edw,22
+ubp,23
+syi,24
+var,25
+pcd,26
+dbs,27
+rnd,28
+ufn,29
+sdp,30
+ncp,31
+kas,32
+pnc,33
+rtc,34
+bpr,35
+e00,36
+e01,37
+e02,38
+e03,39
+e04,40
+e05,41
+e06,42
+e07,43
+e08,44
+e09,45
+e10,46
+e11,47
+e12,48
+e13,49
+e14,50
+e15,51
+e16,52
+e17,53
+e18,54
+e19,55
+e20,56
+e21,57
+e22,58
+e23,59
+e24,60
+e25,61
+e26,62
+e27,63
+e28,64
+e29,65
+e30,66
+e31,67
+clv,68
+slp,69
+ipt,70
+ep0,71
+ep1,72
+ep2,73
+ep3,74
+pts,75
+skt,76
+mac,77
+apd,78
+blf,79
+hld,80
+skd,81

utils/nwztools/database/nvp/parse_nvp_header.sh

@@ -66,13 +66,16 @@ else
     >&2 echo "Analyzing $FILE"
 fi
 
+# old format: #define ICX1087_NVP_NODE_APP "/dev/icx1087_nvp/0"
+# new format: #define ICX_NVP_NODE_APP ICX_NVP_NODE_BASE "0"
+
 cat "$FILE" | awk ' \
 BEGIN { \
-    expr = "#define[[:space:]]+ICX_NVP_NODE_([[:alnum:]]+)[[:space:]]+ICX_NVP_NODE_BASE[[:space:]]*\"([[:digit:]]+)\""; \
+    expr = "#define[[:space:]]+ICX[[:digit:]]*_NVP_NODE_([[:alnum:]]+)[[:space:]]+(ICX_NVP_NODE_BASE[[:space:]]*\"|\"/dev.*_nvp/)([[:digit:]]+)\"";
 } \
 { \
     if($0 ~ expr) \
     { \
-        print(tolower(gensub(expr, "\\1,\\2", "g", $0)));
+        print(tolower(gensub(expr, "\\1,\\3", "g", $0)));
     } \
 }'

utils/nwztools/database/nvp/parse_nvp_nodes.sh

@@ -4,6 +4,9 @@
 # parse_nodes.sh /path/to/rootfs/dir output_file
 # parse_nodes.sh /path/to/rootfs.tgz output_file
 #
+if [ -z "$NVP_LOG" ]; then
+    NVP_LOG=/dev/null
+fi
 if [ "$#" -lt 2 ]; then
     >&2 echo "usage: parse_header.sh /path/to/icx_nvp.ko|/path/to/rootfs/dir|/path/to/rootfs.tgz output_file"
     exit 1
@@ -56,4 +59,4 @@ else
     >&2 echo "Analyzing $FILE"
 fi
 
-./nvptool -x "$FILE" -o "$OUTPUT" >/dev/null
+./nvptool -x "$FILE" -o "$OUTPUT" >"$NVP_LOG"