Merge pull request #17020 from dbudniko:dbudniko/serialization_backend

G-API Serialization routines

* Serialization backend in tests, initial version

* S11N/00: A Great Rename

- "Serialization" is too long and too error-prone to type,
  so now it is renamed to "s11n" everywhere;
- Same applies to "SRLZ";
- Tests also renamed to start with 'S11N.*' (easier to run);
- Also updated copyright years in new files to 2020.

* S11N/01: Some basic interface segregation

- Moved some details (low-level functions) out of serialization.hpp;
- Introduced I::IStream and I::OStream interfaces;
- Implemented those via the existing [De]SerializationStream classes;
- Moved all operators to use interfaces instead of classes;
- Moved the htonl/ntohl handling out of operators (to the classes).

The implementation didn't change much, it is a subject to the further
refactoring

* S11N/02: Basic operator reorg, basic tests, vector support

- Reorganized operators on atomic types to follow >>/<< model
  (put them closer in the code for the respective types);
- Introduce more operators for basic (scalar) types;
- Drop all vector s11n overloads -- replace with a generic
  (template-based) one;
- Introduced a new test suite where low-level s11n functionality
  is tested (for the basic types).

* S11N/03: Operators reorganization

- Sorted the Opaque types enum by complexity;
- Reorganized the existing operators for basic types, also ordered by
  complexity;
- Organized operators in three groups (Basics, OpenCV, G-API);
- Added a generic serialization for variant<>;
- Reimplemented some of the existing operators (for OpenCV and G-API
  data structures);
- Introduced new operators for cv::gimpl data types. These operators
  (and so, the data structures) are not yet used in the graph
  dump/reconstruction routine, it will be done as a next step.

* S11N/04: The Great Clean-up

- Drop the duplicates of GModel data structures from the
  serialization, serialize the GModel data structures themselve
  instead (hand-written code replaced with operators).
- Also removed usuned code for printing, etc.

* S11N/05: Internal API Clean-up

- Minimize the serialization API to just Streams and Operators;
- Refactor and fix the graph serialization (deconstruction and
  reconstruction) routines, fix data addressing problems there;
- Move the serialization.[ch]pp files to the core G-API library

* S11N/06: Top-level API introduction

- !!!This is likely the most invasive commit in the series!!!
- Introduced a top-level API to serialize and deserialize a GComputation
- Extended the compiler to support both forms of a GComputation:
  an expession based and a deserialized one. This has led to changes in
  the cv::GComputation::Priv and in its dependent components (even the
  transformation tests);
- Had to extend the kernel API (GKernel) with extra information on
  operations (mainly `outMeta`) which was only available for expression
  based graphs. Now the `outMeta` can be taken from kernels too (and for
  the deserialized graphs it is the only way);
- Revisited the internal serialization API, had to expose previously
  hidden entities (like `GSerialized`);
- Extended the serialized graph info with new details (object counter,
  protocol). Added unordered_map generic serialization for that;
- Reworked the very first pipeline test to be "proper"; GREEN now, the rest
  is to be reworked in the next iteration.

* S11N/07: Tests reworked

- Moved the sample pipeline tests w/serialization to
  test the public API (`cv::gapi::serialize`, then
  followed by `cv::gapi::deserialize<>`). All GREEN.
- As a consequence, dropped the "Serialization" test
  backend as no longer necessary.

* S11N/08: Final touches

- Exposed the C++ native data types at Streams level;
- Switched the ByteMemoryIn/OutStreams to store data in `char`
  internally (2x less memory for sample pipelines);
- Fixed and refactored Mat dumping to the stream;
- Renamed S11N pipeline tests to their new meaning.

* linux build fix

* fix RcDesc and int uint warnings

* more Linux build fix

* white space and virtual android error fix (attempt)

* more warnings to be fixed

* android warnings fix attempt

* one more attempt for android build fix

* android warnings one more fix

* return back override

* avoid size_t

* static deserialize

* and how do you like this, elon? anonymous namespace  to fix android warning.

* static inline

* trying to fix standalone build

* mat dims fix

* fix mat r/w for standalone

Co-authored-by: Dmitry Matveev <dmitry.matveev@intel.com>

modules/gapi/src/compiler/gcompiler.cpp

@@ -111,24 +111,6 @@ namespace
         return result;
     }
 
-    // Creates ADE graph from input/output proto args
-    std::unique_ptr<ade::Graph> makeGraph(const cv::GProtoArgs &ins, const cv::GProtoArgs &outs) {
-        std::unique_ptr<ade::Graph> pG(new ade::Graph);
-        ade::Graph& g = *pG;
-
-        cv::gimpl::GModel::Graph gm(g);
-        cv::gimpl::GModel::init(gm);
-        cv::gimpl::GModelBuilder builder(g);
-        auto proto_slots = builder.put(ins, outs);
-
-        // Store Computation's protocol in metadata
-        cv::gimpl::Protocol p;
-        std::tie(p.inputs, p.outputs, p.in_nhs, p.out_nhs) = proto_slots;
-        gm.metadata().set(p);
-
-        return pG;
-    }
-
     using adeGraphs = std::vector<std::unique_ptr<ade::Graph>>;
 
     // Creates ADE graphs (patterns and substitutes) from pkg's transformations
@@ -146,14 +128,10 @@ namespace
                                       ade::util::toRange(patterns),
                                       ade::util::toRange(substitutes))) {
             const auto& t = std::get<0>(it);
-            auto& p = std::get<1>(it);
-            auto& s = std::get<2>(it);
-
-            auto pattern_comp = t.pattern();
-            p = makeGraph(pattern_comp.priv().m_ins, pattern_comp.priv().m_outs);
-
-            auto substitute_comp = t.substitute();
-            s = makeGraph(substitute_comp.priv().m_ins, substitute_comp.priv().m_outs);
+            auto&       p = std::get<1>(it);
+            auto&       s = std::get<2>(it);
+            p = cv::gimpl::GCompiler::makeGraph(t.pattern().priv());
+            s = cv::gimpl::GCompiler::makeGraph(t.substitute().priv());
         }
     }
 
@@ -311,11 +289,19 @@ cv::gimpl::GCompiler::GCompiler(const cv::GComputation &c,
 
 void cv::gimpl::GCompiler::validateInputMeta()
 {
-    if (m_metas.size() != m_c.priv().m_ins.size())
+    // FIXME: implement testing/accessor methods at the Priv's API level?
+    if (!util::holds_alternative<GComputation::Priv::Expr>(m_c.priv().m_shape))
+    {
+        GAPI_LOG_WARNING(NULL, "Metadata validation is not implemented yet for"
+                               " deserialized graphs!");
+        return;
+    }
+    const auto &c_expr = util::get<cv::GComputation::Priv::Expr>(m_c.priv().m_shape);
+    if (m_metas.size() != c_expr.m_ins.size())
     {
         util::throw_error(std::logic_error
                     ("COMPILE: GComputation interface / metadata mismatch! "
-                     "(expected " + std::to_string(m_c.priv().m_ins.size()) + ", "
+                     "(expected " + std::to_string(c_expr.m_ins.size()) + ", "
                      "got " + std::to_string(m_metas.size()) + " meta arguments)"));
     }
 
@@ -343,7 +329,7 @@ void cv::gimpl::GCompiler::validateInputMeta()
         return false; // should never happen
     };
 
-    for (const auto &meta_arg_idx : ade::util::indexed(ade::util::zip(m_metas, m_c.priv().m_ins)))
+    for (const auto &meta_arg_idx : ade::util::indexed(ade::util::zip(m_metas, c_expr.m_ins)))
     {
         const auto &meta  = std::get<0>(ade::util::value(meta_arg_idx));
         const auto &proto = std::get<1>(ade::util::value(meta_arg_idx));
@@ -362,7 +348,15 @@ void cv::gimpl::GCompiler::validateInputMeta()
 
 void cv::gimpl::GCompiler::validateOutProtoArgs()
 {
-    for (const auto &out_pos : ade::util::indexed(m_c.priv().m_outs))
+    // FIXME: implement testing/accessor methods at the Priv's API level?
+    if (!util::holds_alternative<GComputation::Priv::Expr>(m_c.priv().m_shape))
+    {
+        GAPI_LOG_WARNING(NULL, "Output parameter validation is not implemented yet for"
+                               " deserialized graphs!");
+        return;
+    }
+    const auto &c_expr = util::get<cv::GComputation::Priv::Expr>(m_c.priv().m_shape);
+    for (const auto &out_pos : ade::util::indexed(c_expr.m_outs))
     {
         const auto &node = proto::origin_of(ade::util::value(out_pos)).node;
         if (node.shape() != cv::GNode::NodeShape::CALL)
@@ -383,7 +377,7 @@ cv::gimpl::GCompiler::GPtr cv::gimpl::GCompiler::generateGraph()
         validateInputMeta();
     }
     validateOutProtoArgs();
-    auto g = makeGraph(m_c.priv().m_ins, m_c.priv().m_outs);
+    auto g = makeGraph(m_c.priv());
     if (!m_metas.empty())
     {
         GModel::Graph(*g).metadata().set(OriginalInputMeta{m_metas});
@@ -512,3 +506,27 @@ void cv::gimpl::GCompiler::runMetaPasses(ade::Graph &g, const cv::GMetaArgs &met
     }
     engine.runPasses(g);
 }
+
+// Creates ADE graph from input/output proto args OR from its
+// deserialized form
+cv::gimpl::GCompiler::GPtr cv::gimpl::GCompiler::makeGraph(const cv::GComputation::Priv &priv) {
+    std::unique_ptr<ade::Graph> pG(new ade::Graph);
+    ade::Graph& g = *pG;
+
+    if (cv::util::holds_alternative<cv::GComputation::Priv::Expr>(priv.m_shape)) {
+        auto c_expr = cv::util::get<cv::GComputation::Priv::Expr>(priv.m_shape);
+        cv::gimpl::GModel::Graph gm(g);
+        cv::gimpl::GModel::init(gm);
+        cv::gimpl::GModelBuilder builder(g);
+        auto proto_slots = builder.put(c_expr.m_ins, c_expr.m_outs);
+
+        // Store Computation's protocol in metadata
+        cv::gimpl::Protocol p;
+        std::tie(p.inputs, p.outputs, p.in_nhs, p.out_nhs) = proto_slots;
+        gm.metadata().set(p);
+    } else if (cv::util::holds_alternative<cv::GComputation::Priv::Dump>(priv.m_shape)) {
+        auto c_dump = cv::util::get<cv::GComputation::Priv::Dump>(priv.m_shape);
+        cv::gimpl::s11n::reconstruct(c_dump, g);
+    }
+    return pG;
+}

modules/gapi/src/compiler/gcompiler.hpp

@@ -58,6 +58,8 @@ public:
     GCompiled   produceCompiled(GPtr &&pg);    // Produce GCompiled from processed GModel
     GStreamingCompiled  produceStreamingCompiled(GPtr &&pg); // Produce GStreamingCompiled from processed GMbodel
     static void runMetaPasses(ade::Graph &g, const cv::GMetaArgs &metas);
+
+    static GPtr makeGraph(const cv::GComputation::Priv &);
 };
 
 }}

modules/gapi/src/compiler/gmodel.hpp

@@ -73,7 +73,7 @@ struct Data
     HostCtor ctor;  // T-specific helper to deal with unknown types in our code
     // FIXME: Why rc+shape+meta is not represented as RcDesc here?
 
-    enum class Storage
+    enum class Storage: int
     {
         INTERNAL,   // data object is not listed in GComputation protocol
         INPUT,      // data object is listed in GComputation protocol as Input
@@ -138,7 +138,9 @@ class DataObjectCounter
 public:
     static const char* name() { return "DataObjectCounter"; }
     int GetNewId(GShape shape) { return m_next_data_id[shape]++; }
-private:
+
+    // NB: private!!! but used in the serialization
+    // couldn't get the `friend` stuff working correctly -- DM
     std::unordered_map<cv::GShape, int> m_next_data_id;
 };
 
@@ -166,6 +168,19 @@ struct Streaming
     static const char *name() { return "StreamingFlag"; }
 };
 
+
+// This is a graph-global flag indicating this graph is compiled
+// after the deserialization. Some bits of information may be
+// unavailable (mainly callbacks) so let sensitive passes obtain
+// the required information in their special way.
+//
+// FIXME: Probably a better design can be suggested.
+struct Deserialized
+{
+    static const char *name() { return "DeserializedFlag"; }
+};
+
+
 // Backend-specific inference parameters for a neural network.
 // Since these parameters are set on compilation stage (not
 // on a construction stage), these parameters are bound lately
@@ -213,6 +228,7 @@ namespace GModel
         , ActiveBackends
         , CustomMetaFunction
         , Streaming
+        , Deserialized
         >;
 
     // FIXME: How to define it based on GModel???
@@ -234,6 +250,7 @@ namespace GModel
         , ActiveBackends
         , CustomMetaFunction
         , Streaming
+        , Deserialized
         >;
 
     // FIXME:

modules/gapi/src/compiler/passes/kernels.cpp

@@ -150,6 +150,17 @@ void cv::gimpl::passes::resolveKernels(ade::passes::PassContext   &ctx,
             selected_backend.priv().unpackKernel(ctx.graph, nh, selected_impl);
             op.backend = selected_backend;
             active_backends.insert(selected_backend);
+
+            if (gr.metadata().contains<Deserialized>())
+            {
+                // Trick: in this case, the op.k.outMeta is by default
+                // missing. Take it from the resolved kernel
+                GAPI_Assert(op.k.outMeta == nullptr);
+                const_cast<cv::GKernel::M&>(op.k.outMeta) = selected_impl.outMeta;
+            } else {
+                // Sanity check: the metadata funciton must be present
+                GAPI_Assert(op.k.outMeta != nullptr);
+            }
         }
     }
     gr.metadata().set(ActiveBackends{active_backends});

modules/gapi/src/compiler/passes/transformations.cpp

@@ -77,7 +77,8 @@ bool tryToSubstitute(ade::Graph& main,
 
     // 2. build substitute graph inside the main graph
     cv::gimpl::GModelBuilder builder(main);
-    const auto& proto_slots = builder.put(substitute.priv().m_ins, substitute.priv().m_outs);
+    auto expr = cv::util::get<cv::GComputation::Priv::Expr>(substitute.priv().m_shape);
+    const auto& proto_slots = builder.put(expr.m_ins, expr.m_outs);
     Protocol substituteP;
     std::tie(substituteP.inputs, substituteP.outputs, substituteP.in_nhs, substituteP.out_nhs) =
         proto_slots;