Bug 29811 - Better categorize harmless unknown array size changes

Let's compile the code snippet:

$ echo "unsigned int is_basic_table[];" | gcc -g -c -o test-v0.o -x c -

Let's see what abidw sees from it:

$ abidw test-v0.o | cat -n

     1	<abi-corpus version='2.1' path='test-v0.o' architecture='elf-amd-x86_64'>
     2	  <elf-variable-symbols>
     3	    <elf-symbol name='is_basic_table' size='4' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
     4	  </elf-variable-symbols>
     5	  <abi-instr address-size='64' path='&lt;stdin&gt;' comp-dir-path='/home/dodji/git/libabigail/PR29811/prtests' language='LANG_C11'>
     6	    <type-decl name='unsigned int' size-in-bits='32' id='type-id-1'/>
     7	    <array-type-def dimensions='1' type-id='type-id-1' size-in-bits='infinite' id='type-id-2'>
     8	      <subrange length='infinite' id='type-id-3'/>
     9	    </array-type-def>
    10	    <var-decl name='is_basic_table' type-id='type-id-2' mangled-name='is_basic_table' visibility='default' filepath='/home/dodji/git/libabigail/PR29811/prtests/&lt;stdin&gt;' line='1' column='1' elf-symbol-id='is_basic_table'/>
    11	  </abi-instr>
    12	</abi-corpus>

See how the at line 7, the array type of ID 'type-id-2' has an unknown
size.  This is the type of the 'is_basic_table' variable defined at
line 10.  Note however that the symbol size of the is_basic_table
symbol is 4 bytes (32 bits).

Now, let's compile a similar code where the is_basic_table variable is
now initialized:

$ echo "unsigned int is_basic_table[] = {0};" | gcc -g -c -o test-v1.o -x c -

$ $ abidw test-v1.o | cat -n

     1	<abi-corpus version='2.1' path='test-v1.o' architecture='elf-amd-x86_64'>
     2	  <elf-variable-symbols>
     3	    <elf-symbol name='is_basic_table' size='4' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
     4	  </elf-variable-symbols>
     5	  <abi-instr address-size='64' path='&lt;stdin&gt;' comp-dir-path='/home/dodji/git/libabigail/PR29811/prtests' language='LANG_C11'>
     6	    <type-decl name='unsigned int' size-in-bits='32' id='type-id-1'/>
     7	    <array-type-def dimensions='1' type-id='type-id-1' size-in-bits='32' id='type-id-2'>
     8	      <subrange length='1' type-id='type-id-3' id='type-id-4'/>
     9	    </array-type-def>
    10	    <type-decl name='unsigned long int' size-in-bits='64' id='type-id-3'/>
    11	    <var-decl name='is_basic_table' type-id='type-id-2' mangled-name='is_basic_table' visibility='default' filepath='/home/dodji/git/libabigail/PR29811/prtests/&lt;stdin&gt;' line='1' column='1' elf-symbol-id='is_basic_table'/>
    12	  </abi-instr>
    13	</abi-corpus>

Now, see like at line 7, the array type is now of 4 bytes (32 bits).
Note that the size of is_basic_table is still 32 bits.

Normally, abidiff-ing test-v0 and test-v1 should tell us that the two
versions of the is_basic_table variable are compatible because
fundamentally the structure and the size of the ELF symbol
is_basic_table hasn't changed, even if in the first case, it's an
array of unknown size.  It's ELF symbol size was already 32 bits.

Here is what abidiff says:

$ abidiff test-v0.o test-v1.o
Functions changes summary: 0 Removed, 0 Changed, 0 Added function
Variables changes summary: 0 Removed, 1 Changed, 0 Added variable

1 Changed variable:

  [C] 'unsigned int is_basic_table[]' was changed to 'unsigned int is_basic_table[1]' at <stdin>:1:1:
    type of variable changed:
      type name changed from 'unsigned int[]' to 'unsigned int[1]'
      array type size changed from infinity to 32
      array type subrange 1 changed length from infinity to 1

$

This is because the comparison engine doesn't recognize we are looking
at a type change that is harmless because the ELF size hasn't changed
and because this is an array of one dimension so fundamentally, the
"meaning" of the type of the array hasn't fundamentally changed for
ABI-related purposes.

This patch teaches the diff node categorizer to recognise that we are
in a case where the (one dimension) array of unknown size actually is
the type of an array which symbol size is 4 bytes.  In the second
case, the one dimension array has a size of 4 bytes, just as its ELF
symbol size.  The diff node categorizer then categorizes the diff node
into the existing category BENIGN_INFINITE_ARRAY_CHANGE_CATEGORY,
which is a harmless diff node category.  Everything then falls into
place to filter the change out.  Also, the patch adapts the diff
reporter to better describe this type of harmless array variable type
changes.

The output then becomes:

$ abidiff test-v0.o test-v1.o
Functions changes summary: 0 Removed, 0 Changed, 0 Added function
Variables changes summary: 0 Removed, 0 Changed (1 filtered out), 0 Added variable

$

The change is filtered out.

To have details about the change that has been filtered out, one has
to use "--harmless" option:

$ abidiff --harmless test-v0.o test-v1.o
Functions changes summary: 0 Removed, 0 Changed, 0 Added function
Variables changes summary: 0 Removed, 1 Changed, 0 Added variable

1 Changed variable:

  [C] 'unsigned int is_basic_table[]' was changed to 'unsigned int is_basic_table[1]' at <stdin>:1:1:
    size of variable symbol ( 32 (in bits)) hasn't changed
    but it does have a harmless type change
    type of variable changed:
      type name changed from 'unsigned int[]' to 'unsigned int[1]'
      array type size changed from 'unknown' to 32
      array type subrange 1 changed length from 'unknown' to 1

$

	* include/abg-comp-filter.h
	(is_var_1_dim_unknown_size_array_change): Declare new function.
	* src/abg-comp-filter.cc (is_var_1_dim_unknown_size_array_change):
	Define new function.
	(has_benign_array_of_unknown_size_change): Rename
	has_benign_infinite_array_change into this.  Make this call the
	new is_var_1_dim_unknown_size_array_change.
	(categorize_harmless_diff_node): Adjust the call to
	has_benign_infinite_array_change into the new
	has_benign_array_of_unknown_size_change.
	* include/abg-ir.h (var_equals_modulo_types): Declare new
	function.  Make it friend of class decl_base.
	* src/abg-default-reporter.cc (default_reporter::report): In the
	overload for var_diff, call the new
	maybe_report_diff_for_variable.
	* src/abg-ir.cc (var_equals_modulo_types): Factorize this out of
	the equals() function for var_decl.
	(equals): In the overload for var_decl, call the new
	var_equals_modulo_types.
	* src/abg-reporter-priv.h (maybe_report_diff_for_variable):
	Declare new function.
	* src/abg-reporter-priv.cc (maybe_report_diff_for_variable):
	Define new function.
	* tests/data/test-diff-filter/test-PR29811-0-report-0.txt: Add
	new reference test output.
	* tests/data/test-diff-filter/test-PR29811-0-report-1.txt:
	Likewise.
	* tests/data/test-diff-filter/test-PR29811-0-v{0,1}.o: Add new
	binary test inputs.
	* tests/data/test-diff-filter/test-PR29811-0-v{0,1}.c: Add source
	code of the binary test inputs.
	* tests/data/Makefile.am: Add new test input files above to source
	distribution.
	* tests/test-diff-filter.cc (in_out_specs): Add new tests to
	harness.

Signed-off-by: Dodji Seketeli <dodji@redhat.com>

include/abg-comp-filter.h

@@ -91,6 +91,13 @@ has_anonymous_data_member_change(const diff_sptr &d);
 bool
 has_data_member_replaced_by_anon_dm(const diff* diff);
 
+bool
+is_var_1_dim_unknown_size_array_change(const diff*);
+
+bool
+is_var_1_dim_unknown_size_array_change(const var_decl_sptr& var1,
+				       const var_decl_sptr& var2);
+
 struct filter_base;
 /// Convenience typedef for a shared pointer to filter_base
 typedef shared_ptr<filter_base> filter_base_sptr;

include/abg-ir.h

@@ -1694,6 +1694,9 @@ public:
   friend bool
   equals(const var_decl&, const var_decl&, change_kind*);
 
+  friend bool
+  var_equals_modulo_types(const var_decl&, const var_decl&, change_kind*);
+
   friend bool
   maybe_compare_as_member_decls(const decl_base& l,
 				const decl_base& r,
@@ -2874,6 +2877,9 @@ public:
 bool
 equals(const var_decl&, const var_decl&, change_kind*);
 
+bool
+var_equals_modulo_types(const var_decl&, const var_decl&, change_kind*);
+
 bool
 equals_modulo_cv_qualifier(const array_type_def*, const array_type_def*);
 

src/abg-comp-filter.cc

@@ -633,6 +633,85 @@ has_data_member_replaced_by_anon_dm(const diff* diff)
   return !c->data_members_replaced_by_adms().empty();
 }
 
+/// Test if we are looking at two variables which types are both one
+/// dimension array, with one of them being of unknow size and the two
+/// variables having the same symbol size.
+///
+/// This can happen in the case of these two declarations, for instance:
+///
+///     unsigned int array[];
+///
+/// and:
+///
+///     unsigned int array[] ={0};
+///
+/// In both cases, the size of the ELF symbol of the variable 'array'
+/// is 32 bits, but, at least in the first case
+bool
+is_var_1_dim_unknown_size_array_change(const var_decl_sptr& var1,
+				       const var_decl_sptr& var2)
+{
+  type_base_sptr /*first type*/ft =
+    peel_qualified_or_typedef_type(var1->get_type());
+  type_base_sptr /*second type*/st =
+    peel_qualified_or_typedef_type(var2->get_type());
+
+  array_type_def_sptr /*first array type*/fat = is_array_type(ft);
+  array_type_def_sptr /*second array type*/sat = is_array_type(st);
+
+  // The types of the variables must be arrays.
+  if (!fat || !sat)
+    return false;
+
+  // The arrays must have one dimension and at least one of them must
+  // be of unknown size.
+  if (fat->get_subranges().size() != 1
+      || sat->get_subranges().size() != 1
+      || (!fat->is_infinite() && !sat->is_infinite()))
+    return false;
+
+  // The variables must be equal modulo their type.
+  if (!var_equals_modulo_types(*var1, *var2, nullptr))
+    return false;
+
+  // The symbols of the variables must be defined and of the same
+  // non-zero size.
+  if (!var1->get_symbol()
+      || !var2->get_symbol()
+      || var1->get_symbol()->get_size() != var2->get_symbol()->get_size())
+    return false;
+
+  return true;
+}
+
+/// Test if we are looking at a diff that carries a change of
+/// variables which types are both one dimension array, with one of
+/// them being of unknow size and the two variables having the same
+/// symbol size.
+///
+/// This can happen in the case of these two declarations, for instance:
+///
+///     unsigned int array[];
+///
+/// and:
+///
+///     unsigned int array[] ={0};
+///
+/// In both cases, the size of the ELF symbol of the variable 'array'
+/// is 32 bits, but, at least in the first case
+bool
+is_var_1_dim_unknown_size_array_change(const diff* diff)
+{
+  const var_diff* d = is_var_diff(diff);
+
+  if (!d)
+    return false;
+
+  var_decl_sptr f = d->first_var(), s = d->second_var();
+
+  return is_var_1_dim_unknown_size_array_change(f, s);
+}
+
 /// Test if a class_diff node has static members added or removed.
 ///
 /// @param diff the diff node to consider.
@@ -1652,30 +1731,9 @@ has_void_ptr_to_ptr_change(const diff* dif)
 ///
 /// @return true iff @p dif contains the benign array type size change.
 static bool
-has_benign_infinite_array_change(const diff* dif)
+has_benign_array_of_unknown_size_change(const diff* dif)
 {
-  if (const var_diff* var_dif = is_var_diff(dif))
-    {
-      if (!var_dif->first_var()->get_symbol()
-	  || var_dif->second_var()->get_symbol())
-	return false;
-
-      if (var_dif->first_var()->get_symbol()->get_size()
-	  != var_dif->second_var()->get_symbol()->get_size())
-	return false;
-
-      const diff *d = var_dif->type_diff().get();
-      if (!d)
-	return false;
-      d = peel_qualified_diff(d);
-      if (const array_diff *a = is_array_diff(d))
-	{
-	  array_type_def_sptr f = a->first_array(), s = a->second_array();
-	  if (f->is_infinite() != s->is_infinite())
-	    return true;
-	}
-    }
-  return false;
+  return is_var_1_dim_unknown_size_array_change(dif);
 }
 
 /// Test if a union diff node does have changes that don't impact its
@@ -1769,7 +1827,7 @@ categorize_harmless_diff_node(diff *d, bool pre)
       if (has_void_ptr_to_ptr_change(d))
 	category |= VOID_PTR_TO_PTR_CHANGE_CATEGORY;
 
-      if (has_benign_infinite_array_change(d))
+      if (has_benign_array_of_unknown_size_change(d))
 	category |= BENIGN_INFINITE_ARRAY_CHANGE_CATEGORY;
 
       if (category)

src/abg-default-reporter.cc

@@ -1658,6 +1658,8 @@ default_reporter::report(const var_diff& d, ostream& out,
 
   maybe_report_diff_for_member(first, second, d.context(), out, indent);
 
+  maybe_report_diff_for_variable(first, second, d.context(), out, indent);
+
   if (diff_sptr dif = d.type_diff())
     {
       if (dif->to_be_reported())

src/abg-ir.cc

@@ -19052,10 +19052,12 @@ var_decl::set_scope(scope_decl* scope)
     get_context_rel()->set_scope(scope);
 }
 
-/// Compares two instances of @ref var_decl.
+/// Compares two instances of @ref var_decl without taking their type
+/// into account.
 ///
-/// If the two intances are different, set a bitfield to give some
-/// insight about the kind of differences there are.
+/// If the two intances are different modulo their type, set a
+/// bitfield to give some insight about the kind of differences there
+/// are.
 ///
 /// @param l the first artifact of the comparison.
 ///
@@ -19072,27 +19074,10 @@ var_decl::set_scope(scope_decl* scope)
 ///
 /// @return true if @p l equals @p r, false otherwise.
 bool
-equals(const var_decl& l, const var_decl& r, change_kind* k)
+var_equals_modulo_types(const var_decl& l, const var_decl& r, change_kind* k)
 {
   bool result = true;
 
-  // First test types of variables.  This should be fast because in
-  // the general case, most types should be canonicalized.
-  if (*l.get_naked_type() != *r.get_naked_type())
-    {
-      result = false;
-      if (k)
-	{
-	  if (!types_have_similar_structure(l.get_naked_type(),
-					    r.get_naked_type()))
-	    *k |= (LOCAL_TYPE_CHANGE_KIND);
-	  else
-	    *k |= SUBTYPE_CHANGE_KIND;
-	}
-      else
-	ABG_RETURN_FALSE;
-    }
-
   // If there are underlying elf symbols for these variables,
   // compare them.  And then compare the other parts.
   const elf_symbol_sptr &s0 = l.get_symbol(), &s1 = r.get_symbol();
@@ -19164,6 +19149,52 @@ equals(const var_decl& l, const var_decl& r, change_kind* k)
   ABG_RETURN(result);
 }
 
+/// Compares two instances of @ref var_decl.
+///
+/// If the two intances are different, set a bitfield to give some
+/// insight about the kind of differences there are.
+///
+/// @param l the first artifact of the comparison.
+///
+/// @param r the second artifact of the comparison.
+///
+/// @param k a pointer to a bitfield that gives information about the
+/// kind of changes there are between @p l and @p r.  This one is set
+/// iff @p k is non-null and the function returns false.
+///
+/// Please note that setting k to a non-null value does have a
+/// negative performance impact because even if @p l and @p r are not
+/// equal, the function keeps up the comparison in order to determine
+/// the different kinds of ways in which they are different.
+///
+/// @return true if @p l equals @p r, false otherwise.
+bool
+equals(const var_decl& l, const var_decl& r, change_kind* k)
+{
+  bool result = true;
+
+  // First test types of variables.  This should be fast because in
+  // the general case, most types should be canonicalized.
+  if (*l.get_naked_type() != *r.get_naked_type())
+    {
+      result = false;
+      if (k)
+	{
+	  if (!types_have_similar_structure(l.get_naked_type(),
+					    r.get_naked_type()))
+	    *k |= (LOCAL_TYPE_CHANGE_KIND);
+	  else
+	    *k |= SUBTYPE_CHANGE_KIND;
+	}
+      else
+	ABG_RETURN_FALSE;
+    }
+
+  result &= var_equals_modulo_types(l, r, k);
+
+  ABG_RETURN(result);
+}
+
 /// Comparison operator of @ref var_decl.
 ///
 /// @param o the instance of @ref var_decl to compare against.

src/abg-reporter-priv.cc

@@ -1070,6 +1070,49 @@ maybe_report_diff_for_member(const decl_base_sptr&	decl1,
   return reported;
 }
 
+/// Report the differences between two generic variables.
+///
+/// @param decl1 the first version of the variable.
+///
+/// @param decl2 the second version of the variable.
+///
+/// @param ctxt the context of the diff.
+///
+/// @param out the output stream to emit the change report to.
+///
+/// @param indent the indentation prefix to emit.
+///
+/// @return true if any text has been emitted to the output stream.
+bool
+maybe_report_diff_for_variable(const decl_base_sptr&	decl1,
+			       const decl_base_sptr&	decl2,
+			       const diff_context_sptr& ctxt,
+			       ostream&		out,
+			       const string&		indent)
+{
+  bool reported = false;
+
+  var_decl_sptr var1 = is_var_decl(decl1);
+  var_decl_sptr var2 = is_var_decl(decl2);
+
+  if (!var1 || !var2)
+    return reported;
+
+  if (filtering::is_var_1_dim_unknown_size_array_change(var1, var2))
+    {
+      uint64_t var_size_in_bits = var1->get_symbol()->get_size() * 8;
+
+      out << indent;
+      show_offset_or_size("size of variable symbol (",
+			  var_size_in_bits, *ctxt, out);
+      out << ") hasn't changed\n"
+	  << indent << "but it does have a harmless type change\n";
+      reported = true;
+    }
+
+  return reported;
+}
+
 /// Report the difference between two ELF symbols, if there is any.
 ///
 /// @param symbol1 the first symbol to consider.

src/abg-reporter-priv.h

@@ -206,6 +206,13 @@ maybe_report_diff_for_member(const decl_base_sptr&	decl1,
 			     ostream&			out,
 			     const string&		indent);
 
+bool
+maybe_report_diff_for_variable(const decl_base_sptr&	decl1,
+			       const decl_base_sptr&	decl2,
+			       const diff_context_sptr& ctxt,
+			       ostream&		out,
+			       const string&		indent);
+
 void
 maybe_report_diff_for_symbol(const elf_symbol_sptr&	symbol1,
 			     const elf_symbol_sptr&	symbol2,

tests/data/Makefile.am

@@ -1135,6 +1135,12 @@ test-diff-filter/test-PR29811-unknown-size-array-dwarf-ctf-CTF.o \
 test-diff-filter/test-PR29811-unknown-size-array-dwarf-ctf-DWARF.o \
 test-diff-filter/test-PR29811-unknown-size-array-dwarf-ctf-report.txt \
 test-diff-filter/test-PR29811-unknown-size-array-dwarf-ctf.c \
+test-diff-filter/test-PR29811-0-report-0.txt \
+test-diff-filter/test-PR29811-0-report-1.txt \
+test-diff-filter/test-PR29811-0-v0.o \
+test-diff-filter/test-PR29811-0-v1.o \
+test-diff-filter/test-PR29811-0-v0.c \
+test-diff-filter/test-PR29811-0-v1.c \
 \
 test-diff-suppr/test0-type-suppr-v0.cc	\
 test-diff-suppr/test0-type-suppr-v1.cc	\

tests/data/test-diff-filter/test-PR29811-0-report-0.txt

@@ -0,0 +1,3 @@
+Functions changes summary: 0 Removed, 0 Changed, 0 Added function
+Variables changes summary: 0 Removed, 0 Changed (1 filtered out), 0 Added variable
+

tests/data/test-diff-filter/test-PR29811-0-report-1.txt

@@ -0,0 +1,13 @@
+Functions changes summary: 0 Removed, 0 Changed, 0 Added function
+Variables changes summary: 0 Removed, 1 Changed, 0 Added variable
+
+1 Changed variable:
+
+  [C] 'unsigned int is_basic_table[]' was changed to 'unsigned int is_basic_table[1]' at test-PR29811-0-v1.c:1:1:
+    size of variable symbol ( 32 (in bits)) hasn't changed
+    but it does have a harmless type change
+    type of variable changed:
+      type name changed from 'unsigned int[]' to 'unsigned int[1]'
+      array type size changed from 'unknown' to 32
+      array type subrange 1 changed length from 'unknown' to 1
+

tests/data/test-diff-filter/test-PR29811-0-v0.c

@@ -0,0 +1 @@
+unsigned int is_basic_table[];

tests/data/test-diff-filter/test-PR29811-0-v1.c

@@ -0,0 +1 @@
+unsigned int is_basic_table[] = {0};

tests/test-diff-filter.cc

@@ -822,6 +822,20 @@ InOutSpec in_out_specs[] =
    "data/test-diff-filter/test-PR29387-report.txt",
    "output/test-diff-filter/test-PR29387-report.txt",
   },
+  {
+    "data/test-diff-filter/test-PR29811-0-v0.o",
+    "data/test-diff-filter/test-PR29811-0-v1.o",
+    "--no-default-suppression",
+    "data/test-diff-filter/test-PR29811-0-report-0.txt",
+    "output/test-diff-filter/test-PR29811-0-report-0.txt",
+  },
+  {
+    "data/test-diff-filter/test-PR29811-0-v0.o",
+    "data/test-diff-filter/test-PR29811-0-v1.o",
+    "--harmless --no-default-suppression",
+    "data/test-diff-filter/test-PR29811-0-report-1.txt",
+    "output/test-diff-filter/test-PR29811-0-report-1.txt",
+  },
 #ifdef WITH_CTF
   {
    "data/test-diff-filter/test-PR29811-unknown-size-array-dwarf-ctf-DWARF.o",